A Framework for Personalized Competitive Programming Training

T. Di Mascio, L. Laura and M. Temperini, "A Framework for Personalized Competitive Programming Training," 2018 17th International Conference on Information Technology Based Higher Education and Training (ITHET), Olhao, Portugal, 2018, pp. 1-8, doi: 10.1109/ ITHET.2018.8424620. A Framework for Personalized Competitive Programming Training Tania Di Mascio Luigi Laura, Marco Temperini Dept. of Information Engineering, Dept. of Computer, Control, and Computer Science and Mathematics, Management Engineering, University of L’Aquila, Italy. Sapienza University of Rome, Italy. [email protected] {laura,marte}@diag.uniroma1.it Abstract—Programming Contests are a special branch in the The evaluation should give a real time feedback to the general area of training and education programming support and student, so it consists in the execution and testing of the cover an important role in the area of computer science. Rather submitted program (hence, we are applying dynamic testing than the direct provision of concepts and exercises related to programming, the main concern is on the algorithms and data in our framework). structures managed while composing the solution to a problem, While the training activity could be made by a fixed set (or and the quality of the solution program itself. sequence) of problems, all of them to be met by the student in Training via web for programming contests is mainly com- order to reach the training aims (one-size-fits-all aaproach), we prised of the activity of producing solutions to problems offered are seeking for a personalized approach, in which the selection by the online system (online training platforms), whereas a solution is a program, submitted by the student, and it has to of the next problem to meet is done by the student, basing be evaluated, in order to assess the programming performance, on her will and preferences, and possibly on the suggestions and give suggestions about the quality of the solution, so to (recommendations) given by the system. In practice, we aim make the experience an informative (learning) one. To the best of at defining a framework in which a recommendation is the our knowledge, supporting a personalized approach and possibly suggestion that a certain problem would be more beneficial of recommendations given by the system, is a challenge. In this paper we tackle the problem of supporting students in others, for the student skills development, and it is defined their online training toward the participation in a programming by the system basing on the specification of the available contest, aiming at defining a framework that can allow recom- problems, and the student model. Moreover, in line with the mendations, under the form of suggestions, to the learner, about more interesting online training systems (see e.g., HackerRank the next programming problem to undertake, and that can foster - HR - https://www.hackerrank.com/) motivating students to motivation in students by means of a lightweight, badge-based, gamified approach. solve new and more complex problems, we are also seeking a framework in which recommendations will be “experienced” I. I NTRODUCTION in a gamified approach. As observed by Dagienė [1] and Garcia-Mateos and Citing Ricci et al. [9], we say that “Recommender Systems Fernandez-Aleman [2], the programming contests, a com- (RSs) are software tools and techniques providing suggestions petition in which the contestant are faced with a set of for items to be of use to a user”. Since 80’s RSs has programming problems to be solved in a limited amount of been realized in different field in which users can make time, are very important and effective in the process of learning a decision. From 80’s to nowadays, RSs researchers and computer programming and, more generally, computer science developers moved from simply designing and implementing [3], [4], [5], [6]. new recommendation methods to fine-tuning these methods Training for programming contests is mainly comprised of and finding solutions to utilize and manipulate additional the activity of producing solutions to problems offered by the information about the user in recommendation processes. It is online system (online training platforms), whereas a solution is out of the scope of this paper analyzing RS and their methods – a program, submitted by the student, and it has to be evaluated, we suggest reading one of the most cited survey [10] in which in order to assess the programming performance, and give authors clearly and exhaustive describe RSs’ algorithms – we suggestions about the quality of the solution, so to make the here only should suggest to read papers about one of the most experience an informative (learning) one. important and recently challenge of RSs: the diversification, In these contests, however, rather than the direct provision becoming crucial to increase the user satisfaction and the over- of concepts and exercises related to programming, the main fitting issues (see e.g., [11] or [12]), since the diversification concern is on the algorithms and data structures managed not only try to solve the cited issues, but also require a lot of while composing the solution (program) to a problem, and human-oriented involvement then other RS related aspects. the quality of the program itself, such as the correctness of the Gamification approach is actually used to stimulate users solution, the efficiency of the process, the time spent executing and promote behavior changes in a broad range of domains, it [7], [8]. such as training, education, online communities etc. (see e.g., [13] or [14]). Formally, it is defined as the use of game design II. P ROGRAMMING CONTESTS : TASKS , SKILLS , AND elements in non-game contexts [15]. Despite its popularity on PLATFORMS the Web, gamification research is still an emergent field and In this section we briefly recall some aspects of a program- much remains to be done. Several reviews exist about this issue ming contest, providing the necessary background, comprising (see e.g., [14], or [16]), however, making a complete review information about the required skills and the online training about gamification is actually out of the scope of this paper. platforms currently available. On the other hand, it is important to notice that the practice of adding game design to non-game contexts has gained A. Programming Contests a relatively large amount of attention, since, as mentioned A programming contest is a competition in which the above, gamification seems to increase the user engagement, contestant are faced with a set of programming tasks, also the commitment to goal, and the overall betterment of various called problems, to be solved in a limited amount of time. In behavioral outcomes[17]. In fact, gamification offers an affec- Fig. 1 we report, as an example, the text of the task Fractal tive experience (that is an artificial emotional intelligence – Graph, that has been used in an online contest of the 2018 see e.g. the seminal paper [18]) that, by nature, improves the edition of the International Olympiad in Informatics in Team users motivation to use a product, system, or service [15]. (IOIT https://ioi.team). Each task has, usually, a general descriptive section in While, on the one hand, our framework is in line with a which the problem is presented in a colloquial and informal recent work of Tondello et al. [19], in which authors stress way; depending on the competition, technical terms might be the necessity to “identifying different personality traits or avoided, in order not to discourage a casual reader. After the preferences for personalization” to improve the quality of the informal description there is one or more sections in which gamification approach and its efficacy in terms of results, are described: on the other hand, our contribution is not focused on a • the input and output specifications, i.e. a precise and non theoretical proposal aiming at building a recommender systems for personalized gamification (reported in in the cited paper) ambiguous description of the format of input and output; • time and/or space constraints that the program must obey, but rather on supporting students in their online training for the participation in (algorithmic) programming contests such such as being able to process each input instance in less as the International Olympiads in Informatics (IOI) and the than one second or using less than 1MB of RAM; • additional constraint about the problem: for example, ACM International Collegiate Programming Contest (ICPC). if the task involves finding a particular graph structure With this goal in mind, the framework we propose aims to: inside a given graph, it might guarantee that in each input instance there is such a structure. • specify the training content of a programming problem, It is very important to mention that a single task can as the capabilities (skills) that can be considered as owned be broken into different subtasks of increasing complexity: by the student once (s)he provides the system with a basic techniques might be enough to solve, within the given successful solution; time and/or space limits, some of the subtasks whilst the • the requirements of a problem, in term of skills that most difficult ones might require very specific algorithmic should be owned by a student before undertaking the techniques and data structure. problem’s solution; Among the programming contest, we mention: • maintain a representation of the programming skills of • The International Olympiads in Informatics (IOI), that the student (student model) are an annual programming competition for secondary • compare a problem’s specification with the a student school students patronized by UNESCO. Participants are model, in order to decide whether such problem could usually the winners of national competitions. The 2017 be fruitfully undertaken by the student, or it is already edition of IOI was held in Iran, with 326 participants from covered by the student model (and so it is not crucial 84 countries. The competition is divided in two days; in to meet it), or it is out of reach for the time being each day the contestant are faced with three tasks to be (its requirements make it unlikely that undertaking the solved in at most five ours. http://www.ioinformatics.org/ problem could benefit - rather than frustrating - the • The ACM International Collegiate Programming Contest student. (ICPC) is a multitier, team-based, programming com- petition operating under the auspices of ACM. This Although or work is designed for a specific type of training year 49,935 contestants from 3,098 universities in 111 on programming, it has a general validity, in our opinion, as countries competed in regional competitions at over 530 far as the competence on algorithms (to be selected and imple- sites worldwide1 . https://icpc.baylor.edu/ • The very recent International Olympiads in Informat- mented) and data structures (usable for such implementations) are concerned. So in the paper we are also comparing our ics in Team (IOIT), that started in 2017, that are a logical framework with the trends in systems for the automatic 1 Data from this year ACM ICPC Factsheet, available at https://icpc.baylor. assessment of programs in educational settings. edu/worldfinals/pdf/Factsheet.pdf.  team competition, like ACM ICPC, differently from IOI that provide essential material about algorithms, data struc- (individual competition). Currently there are only four tures, and heuristics needed in programming contests. nations involved: Italy, Romania, Russia, and Sweden. Thus, a single task might be solved by several techniques, https://ioi.team/ not all able to score the full points available. Also, note that • Google Code Jam, that is based on multiple online usually solving a problem requires to adapt, often in a not rounds that concludes in the World Finals. https:// trivial way, the proper technique to the setting provided by code.google.com/codejam/. Google has also two other the task. coding competitions: Kickstart (https://code.google.com/ codejam/kickstart/) for university students looking to de- C. Online training platforms velop their coding skills and potentially pursue a Google Besides the already mentioned platforms that provides on- career, and Hash Code (https://hashcode.withgoogle. line programming contests, and usually have the archive of com/) for students in Africa, Europe, and the Middle East the previous years’ tasks, there are several online training looking to work as a team on real Google problems. platforms; we mention the well known UVa Online Judge4 , • Facebook Hacker Cup, that is an annual worldwide Sphere Online Judge5 (SPOJ) and Open Kattis6 . programming competition where hackers compete against All these platforms are, roughly speaking, huge repositories each other for fame, fortune, glory and a shot at the cov- of tasks and it is possible to submit solutions and be part of eted Hacker Cup. https://www.facebook.com/hackercup/ a global ranking based on the overall number of tasks solved. In general, students participating to programming contests To help learners, in the above platform, there is no other tool have at disposal several programming contest platforms, in- than an online forum, where they can discuss with other users. cluding Codeforces, USACO, COCI, TopCoder, Codechef, and The most interesting exception is a common tool, available (for Hackerearth, that run contests with different periodicity. some tasks) in almost all the mentioned platforms: uDebug The programming languages used in the competitions vary (https://www.udebug.com/). uDebug is a platform where, for considerably: for example, IOI and IOIT accept only C,C++, each available task, you can insert an input instance and and Pascal; ICPC adds to the list also Python (both 2 and 3), get back the correct output instance. This way, you can see Java, and Kotlin. Google Code Jam accepts any programming properties of the computed solutions without looking at the language and any editor freely available for everyone, i.e. no solutions’ code. uDebug currently hosts problems from eight commercial tools. different training platforms, that provide a link to uDebug in all the tasks for which is available. B. Algorithmic skills and techniques Another platform, Light Online Judge (LOJ - http://www. The programming contests mentioned in the previous sec- lightoj.com/) provides also a list of categorized problems that tion focus on a large set of algorithmic skills, that includes can be useful for beginners. • Basic Data Structures An interesting platform is the recent HackerRank (HR - • Complete Search, Divide & Conquer, Greedy techniques https://www.hackerrank.com/), that is both a training platform • Dynamic Programming (basic ideas) and a job placement website: the idea is that companies can • Dynamic Programming (advanced topics) evaluate developer on the basis of their actual skills. HR uses • Basic Graph algorithms (e.g.: connected components, a complex ELO based rank system, where a user rank in a distance between two nodes) contest is estimated and compared to the rank obtained in • Advanced Graph algorithms (e.g.: network flows, match- the contest: if the actual rank in the contest is better than ing) the expected rank the rating will increase, otherwise it will • Mathematical problems decrease. Also, they provide a basic badge system with an • Computational Geometry increasing number of stars inside each badge, and a medal • String Processing system for competitions. Most notably, they also offer few • Advanced topics (currently four) basic tutorials. As observed in the previous section, each task might be III. OII: A F RAMEWORK FOR D EVELOPING A LGORITHMIC divided into subtasks of increasing complexity: for example, if C OMPETENCIES a problem asks, given an array of integers, to compute the sum of the first i integers in the array, to solve the first subtasks it A. Automated analysis of programs, and badges might be enough to iterate the array, whilst the most complex A system supporting program assessment, for educational subtasks might need the use of a dedicated data structure such purposes, can encompass a wide range of tasks [22], from the as the Fenwick Tree [20]. Some programming contests, such static analysis (with allowance for testing slightly statically as IOI, have a syllabus3 , whilst other simply refer to general incorrect programs), through a comprehensive dynamic testing algorithmic skills. There are some books, such as the ones of (providing timely and ”correcting” feedback), to an analysis Skiena [7], Halim [8], and the recent one of Laaksonen [21], 4 https://uva.onlinejudge.org/ 3 The 5 http://www.spoj.com/ IOI syllabus is available at the url http://people.ksp.sk/∼misof/ ioi-syllabus/ 6 http://www.spoj.com/  O I ( ) OIS2018 – Round 2 ; + + S Online, November 28th, 2017 fractal • EN Fractal graphs (fractal) After Edoardo accidentally found Giorgio working on fractals (amazing mathematical shapes with recursive definitions), he After Edoardo accidentally found Giorgio working on fractals decided to do something similar... but in a somewhat more computer science fashion. (amazing After days ofmathematical shapeshe excruciating research, with recursive finally definitions), discovered the fractal he graphs GN ! The first member of this family of graphs,2 decided to do something similar... but in a somewhat more G0 , is very simple: a single node without any edges. After that, the graphs quickly grow in complexity as N increases. computer More science precisely, fashion. each fractal graph GN for N > 0 is obtained from its predecessor GN −1 by adding: • A triangle T for every node v in GN −1 , so that one of the nodes of T is v and the other nodes and edges are new; After days of excruciating research, he finally discovered the • A segment S in the middle of every edge e in GN −1 , that is, e is1 split in half into two edges e1 and e2 joined by node fractal graphs GN ! The first member of this family of graphs, v, and S starts from v (thus adding a further node and edge). G0 , is very simple: a single node without any edges. After The first seven fractal graphs G0 , . . . , G6 are the following: that,Edoardo Help the graphs show quickly his fractalgrow in complexity supremacy, by countingas the increases. N number of nodes and edges in GN ! More precisely, each fractal graph GN for N > 0 is obtained from its predecessor GN ≠1 by adding: Input file The first and only line contains the only integer N . • A triangle T for every node v in GN ≠1 , so that one of Figure 1: A detail of the Mandelbrot set, file nodes of T is v and the other nodes and edges are the most glorious among all fractals. Outputthe new; You need to write a single line with two integers: the number of nodes and edges in GN modulo 32 749. • A segment S in the middle of every edge e in GN ≠1 , that is, e is split in half into two edges e1 and e2 joined by node v, and S starts from v (thus adding a further node and edge). 0 The first seven fractal graphs G0 , . . . , G6 are the following: 1 2 3 0 1 2 3 4 5 6 Fig. 1. An example of a problem from a programming contest; this task is taken from an online contest of the 2018 edition of the International Olympiad 4 in Informatics in Team (IOIT). 5 6 Help Edoardo show his fractal supremacy, by counting the number of nodes and edges in GN ! 1 A graph is a mathematical object consisting of a set V of nodes (unlabelled points) and a set E of edges (undirected links between couple of points), so that E ™ V ◊ V . of the program structure (so to uncover solutions that get the this area) is discussed in [39], under the form of analysis of expected results just through workarounds and cheating). information trails, learner’s behaviour and accomplishments. Research on the computer based analysis, and automated assessment, of programs in education is active since long The model proposed in the following subsection is comprised time [23]. From it several applications stemmed, dedicated of an approach to dynamic analysis of programming tasks, to learning of introductory programming concepts, and, more flanked by a mechanism of student modeling. In the frame- recently, to the development of computational thinking [24], work, the student modeling supports the personalized creation [25]. Usually the programming task submitted by the student of learning paths by the students, and features a minimal, yet is evaluated by either Static Analysis, or Dynamic Analysis, significant, aspect of gamification, where badges are used to whereas combinations of the two approaches are less frequent track competence accumulated by the learners. (an occurrence is in [22]). Static Analysis provides an assess- B. The OII framework ment in relation to the use of syntax in the program, and to the resulting static semantic. Compiler generated errors can The model is expected to support the overall system [33], be analysed and commented for the student [24], [26]. In [27] where a program is assessed against its structural similarity with a • programming problems (tasks) are provided; reference one provided by the teacher. Dynamic Analysis, on • learners submit a solution, comprised of the specification the other hand, proceeds by actually running the program, of the solution made according to the model, and the on a selection of test cases accurately pre-defined, and takes actual program; into consideration the success in such tests, together with • the feedback on the solution specification can help devel- the running time [28]. As seen earlier in this paper, systems oping the learner’s capability to select solution strategies supporting competitive programming use dynamic testing [1], • the learner is modeled (student model) by the set of [29]. Among approaches in this line we may cite Kattis [30], problems she met, by the quality of the related solutions, that is used for the ACM-ICPC contest. Different interesting and by the skill acquirements obtained during the solving evolutions of this approach, where the tests can be (or have activity, as represented by a set of badges; to be) produced by the students, whose analysis is beyond the • the learner can built an own path of practice, by select- scope of this paper, are proposed in [31], [32]. ing her own sequence of problems, possibly accepting The main task of OII is to support the non novice students suggestions coming from the system (a problem can be with training for the programming contests, and to help the suggested by the system, to the individual learner, basing instructors that will manage such contests, So, in this section on her current student model, and on the target stated for we provide a model for the system described in [33], where the overall learner’s learning activity in the system. a solution to a problem is characterized based on its chosen • the quality of a solution is determined by algorithmic strategy (pointing out what algorithm is elected, – analysis of how the chosen algorithm fits with the and what data structures are used for its implementation): problem, and how the selected data structure does fit this enhances the usual approach to static analysis, where the with the algorithm implementation) effort is on unveiling syntactic/semantics errors and, possibly, – dynamic testing of the program provided with the misconceptions in novice programmers. solution. Gamification, and Game based learning (GBL), are provided In order to support student modeling, in a lightweight- with a good deal of research activity in Technology Enhanced gamification environment, we use skill badges. A skill badge Learning [34], [35], [36]. About these research topics we have (badge henceforth) is the visual representation of a compe- provided a short discussion in the introduction section; here tency (a quantum of knowledge, possibly associated to the we just recall some aspects related to capability to perform a task). This definition of a badge 1) the adaptivity of the learning path, where the learning is consistent with the concept of competency usually met path can be possibly determined by the learner herself, in competence based student modeling ([40]). In the badge 2) and to the management of the student model, supporting definition is implicit a level of proficiency for the skill. the above mentioned adaptivity. A badge, b, definition is comprised of So, it is interesting to mention the concept of adaptive game, 1) a name (univocal identifier), which basically means to allow for game applications in which 2) an image, imageresource the gaming interface and/or the game mechanics can depend 3) a textual description of the competency, on player’s gamer type (Explorers, Achievers, Winners) [37]. 4) a possible reference to another badge (from which the b Personalization of learning game activity is discussed in is dependent or derived), subadge , and [36], where students with limited access to teaching staff are 5) a value in [0,1] of minimal proficiency for the compe- proposed games according to their student model (here the tency, needed to own the badge, bval . model is determined by previous learner’s choices, and game b = (name, imageresource , [text], subadge , bval ) reputation among the users). Student modeling by learning style and player (gaming) style is described in [38], while the A problem (P ) is defined as a family of approaches to the assessment of the gaming performance (another hot topic in problem. A problem approach is comprised of  • a pair < A, DS >, where A is an algorithm suitable to be Accordingly, the student model of a learner, s, denoted as used for the solution, and DS is the data structure usable SM (s), is defined as the couple of sets representing products for the implementation of such algorithm; and competence of the learner, i.e. the solutions she submitted • the information about the reputation (Rep()) of such and the badges she collected on the system: an approach among those that have been used for the SM (s) =< P ADS(s), B(s) > problem P. This is a real value in [0,1] (0 lowest, 1 highest) stating how good is such a choice, according to The above specification framework can be used in contests, the history of the contests. It’s computation is based on in order to gather information and develop a database of how frequently the approach was successfully employed problems specifications, solutions, and student models. for the problem, and on the reputation of the solvers that On the other hand, it can also be used, in a social, web- adopted it in their contests. based, environment, to support training, for both instructors • the set of badges that can be acquired by submitting a and prospective students participating in contests. In this case, solution to the problem (BA) the learner builds a learning path of problems, and related • the set of badges that the learner has to possess, as submitted solutions, with the aim to cover learning aims of prerequisites to being able to submit a solution to P (BR). the course. The Training Target of the course (T T ) is defined as the couple of sets representing the problems to be met as a P = {(< AP P P P i , DSi >, Rep(< Ai , DSi >, P ), BRiP , BAP priority, and the competence to be gained, just mirroring the i )} whereas definition of the student model: • i is an index identifying the problem approach (with i ∈ T T =< T P , T B > [1, . . . , nP ] and nP the number of distinct approaches known for the problem P), and where P • the sets of badges are computed by a badge function • T is a set of problems in the system, that are mandatory b() as, resp., BAP i = b(P, i, acquire) and BRiP = to be met by the students in the course (which does not b(P, i, require). exclude the possibility to select also other problems, in In a contest, the managing instructors (guru) provide evalu- addition); B • T is a set of badges, required to be possessed by the ations about the solving learners (this is, provisionally in OII, the means to compute learners’ reputation), and elect some of students, by the end of the course activity. them as Exemplary Peers (EPs). EPs are used to update the A student s is said to satisfy the T T iff list of approaches for the problems they met. 1) for each P ∈ T P there is a solution Sol = A solution, Sol, submitted, by a student s, for a problem (s, P, conteXt, < A, DS >) ∈ solutions(s), such that P , in a given contest (conteXt) is formally modeled by the < Sol, Eval(Sol) >∈ P ADS(s), and approach (< A, DS >) declared at submission time: 2) for each skill badge b ∈ T B , it is b ∈ B(s). Sol = (s, P, conteXt, < A, DS >) During a course, the learner s builds her own path of problems. Supposing that s selected the problem (when IOI2 is used for instructors training, conteXt is not a contest, and is assigned as training (meaning off contest). For P = {(< AP P P P i , DSi >, Rep(< Ai , DSi >, P ), the set of all the solutions submitted by a student s we use BRiP , BAP i )} the notation solutions(s). and provided the solution in the contest g Given a submitted solution, Sol, its evaluation, Eval(Sol), Sol0 = (s, P, g, < A0 , DS 0 >) is a real value in [0,1], computed basing on • the Rep(< A, DS >, P ) value of the approach selected then, after the automated evaluation of the solution 0 in the solution: The correspondence between the spec- • Eval(Sol is produced. ification of Sol and the actual implementation of the • Feedback is provided about the solution approach < solution, is care of the guru(s) of a contest (or training A0 , DS 0 >. This feedback regards the appropriateness of course). the Sol0 approach, and in based on the Rep() function; in • The grade obtained by the implementation during dy- addition, comments can be added, automatically as well, namic testing (level, L). telling the history of adoption of such approach, and the We also denote as P ADS(s) the set of all the solutions reputation of the solvers that selected it. submitted by the student s, and evaluated, associated with their • The SM (s) is updated, according to the quality of the respective evaluation: solution: – an element < Sol0 , Eval(Sol0 ) > is added to P ADS(s) = {< Sol, Eval(Sol) > | Sol ∈ solutions(s)} P ADS(s); Moreover, we denote by B(s) the set of badges possessed by – Given k such that < A0 , DS 0 >=< AP P k , DSk >, the student s, on account of the solutions submitted by s and for each badge b = (name, image, [text], su, bval ), 0 evaluated. with b ∈ BAP k , and Eval(Sol ) ≥ bval  b is added to B(s). target. This feature cannot currently par with the functionalities By repeating the above selection/evaluation/model update of a full fledged recommender system, and yet constitutes process for a sequence of problems, the learner builds the a very effective, and novel to the extent of our knowledge, learning path of problems aiming at total coverage of the T T . ground for future work. At each selection stage, the system may provide the learner R EFERENCES with a suggestion, about the next best problem(s) to meet, in view of TT coverage. [1] V. Dagienė, “Sustaining informatics education by contests,” in Teaching Fundamentals Concepts of Informatics. Springer, 2010, pp. 1–12. Such suggestion is produced based on the student model [2] G. Garcia-Mateos and J. L. Fernandez-Aleman, “Make learning fun with SM (s) =< P ADS(s), B(s) >, and has the shape of a ranked programming contests,” in Transactions on Edutainment II. Springer, list of problems (or rather problem approaches) to undertake, 2009, pp. 246–257. [3] G. Audrito, G. B. Demo, and E. Giovannetti, “The role of contests in in order to increase SM (s) towards the fulfillment of T T . changing informatics education: A local view.” Olympiads in Informat- In this initial formulation of our framework, an approach to ics, vol. 6, 2012. problem P , [4] O. Astrachan, “Non-competitive programming contest problems as the basis for just-in-time teaching,” in Frontiers in Education, 2004. FIE App = (< A, DS >, Rep(< A, DS >, P ), BR, BA) 2004. 34th Annual, Oct 2004, pp. T3H/20–T3H/24 Vol. 1. [5] M. Blumenstein, S. Green, S. Fogelman, A. Nguyen, and V. Muthukku- marasamy, “Performance analysis of game: a generic automated marking 1) gets into the ranked list for the learner s, if f environment,” Computers and Education, vol. 50, pp. 1203—-1216, a) the approach is not yet in the SM(s): there is no 2008. element < Sol, Eval(Sol) >∈ P ADS(s) such [6] T. Wang, P. Su, X.and Ma, Y. Wang, and K. Wang, “Ability-training- oriented automated assessment in introductory programming course,” that Sol = (s, P, conteXt, < A, DS >); Computers and Education, vol. 56, pp. 220—-226, 20011. b) the requirements for the approach are in possession [7] S. S. Skiena and M. A. Revilla, Programming challenges: The program- of the learner: BR ⊆ B(s), ming contest training manual. Springer Science & Business Media, 2003. c) and at least some of the badges in BA are not yet [8] S. Halim and F. Halim, Competitive Programming, Third Edition. Lulu. in the SM(s): BA ∩ B(s) 6= ∅, com, 2013. [9] F. Ricci, L. Rokach, and B. Shapira, Introduction to Recommender 2) has a rank proportional to the reputation value of the Systems, 2011, pp. 1—-29. approach, and to the number of badges that can be added [10] J. Bobadilla, F. Ortega, A. Hernando, and A. Gutiérrez, “Recommender in the SM(s) through the approach: systems survey,” Knowledge-Based Systems, vol. 46, pp. 109–132, 2013. [11] B. Bradley and B. Smyth, “Improving recommendation diversity,” in Rank(App) = Proc. of 12th Nat. Conf. Artif. Intell. Cogn. Sci (AICS01), 2001, pp. 75–84. α·Rep(< A, DS > +β ·Card(BA∩B(s)) [12] M. Kunaver and T. Pozrl, “Diversity in recommender systems – a survey,” Knowledge-Based Systems, vol. 123, pp. 154–162, 2017. where α and β are coefficients that express the policy [13] E. N. Lennart and S. Deterding, “The maturing of gamification research,” of the system in regard to Computers in Human Behavior, 2017. α the speed of fulfillment of the mandatory ap- [14] K. Seaborn and D. Fels, “Gamification in theory and action: A survey,” International Journal of Human-Computer Studies, vol. 74, pp. 14–31, proaches; 2014. β the preference for the learner being developing [15] S. Deterding, D. Dixon, R. Khaled, and L. Nacke, “From game design her competence on her own, as a priority, before elements to gamefulness: Defining gamification,” in Proc. of the 15th In- ternational Academic MindTrek Conference: Envisioning Future Media of caring to complete the mandatory tasks. Environments, 2011, pp. 9–15. [16] J. Witte, R. Westbrook, and M. Witte, “Gamification and training,” IV. C ONCLUSIONS AND F UTURE WORKS in Proc. of Global Conference on Education and Research (GLOCER 2017), W. B. J. and C. Cobanoglu, Eds., 2017. In this paper we presented a framework based on the dy- [17] A. Domı́nguez, J. Saenz-de Navarrete, L. De-Marcos, L. Fernández- namic evaluation of programming tasks in education, deemed Sanz, C. Pagés, and J. Martı́nez-Herráiz, “Gamifying learning experi- to support learning in computer programming, algorithms, ences: Practical implications and outcomes,” Computers and Education, vol. 63, pp. 380–392, 2013. and data structures, for students preparing to participate in [18] R. Picard, “Affective computing,” MIT Cambridge (MA), Tech. Rep., competitive programming contests, such as the International 1995. Olympiads in Informatics and the ACM International Colle- [19] G. Tondello, R. Orji, and E. Lennart, “Recommender systems for personalized gamification,” in Proc. of Fifty Shades of Personalization giate Programming Contest. - Workshop on Personalization in Serious and Persuasive Games and The main, innovative, traits of the framework are in the Gameful Interaction, 2017. definition of a student model paradigm, and in the use of [20] P. M. Fenwick, “A new data structure for cumulative frequency tables,” Softw. Pract. Exper., vol. 24, no. 3, pp. 327–336, Mar. 1994. [Online]. badges to support it. Available: http://dx.doi.org/10.1002/spe.4380240306 The framework is partially implemented in the system [21] A. Laaksonen, Guide to Competitive Programming. Springer, 2017. described in [33], used for the training of students for the [22] T. Wang, P. Su, X. Ma, Y. Wang, and K. Wang, “Ability-training-oriented automated assess-ment in introductory programming course.” Comp. and Italian Olympiads in Informatics. Ed., vol. 56, pp. 220–226, 2011. The framework features some basic recommending abilities, [23] J. Hollingsworth, “Automatic graders for programming classes.” Comm. under the form of suggestions, given to the individual student, ACM, vol. 3, 1960. [24] M. Hristova, A. Misra, M. Rutter, and R. Mercuri, “Identifying and about what next programming task would be the most fruitful correcting java program-ming errors for introductory computer science to undertake, in order for the learner to reach the training students.” in Proc. SIGCSE 2003, 2003. [25] V. Pieterse, “Automated assessment of programming assignments.” in L. Versari, “oii-web: an interactive online programming contest training Proc. CSERC 2013, 2013, pp. 200–203. system,” Olympiads in Informatics, vol. 10, pp. 195–205, 2016. [26] C. Watson, F. Li, and J. Godwin, “Bluefix: Using crowd-sourced [34] R. Van Eck, “Digital game-based learning: It’s not just the digital natives feedback to support programming students in error diagnosis and repair,” who are restless.” EDUCAUSE Review, vol. 41, pp. 16–30, 2006. in Proc. Int. Conf. on Web-based Learning, ICWL 2012, ser. LNCS, vol. [35] S. Deterding, R. Khaled, L. Nacke, and D. Dixon, “Gamification: toward 7558. Springer Verlag, 2012, pp. 228–239. a definition.” in Proc. CHI 2011 Gamification Workshop, 2011, pp. 12– [27] K. Naudé, J. Greyling, and D. Vogts, “Marking student programs using 15. graph similarity,” Comp. and Ed., vol. 54, pp. 545––561, 2010. [36] M. Metawaa and K. Berkling, “Personalizing game selection for mobile [28] M. Joy, N. Griffiths, and R. Boyatt, “The boss online submission learning.” in Proc. CSEDU 2016, 2016, pp. 306–311. and assessment system.” ACM J. Educational Resources in Computing, [37] B. Magerko, C. Heeter, J. Fitzgerald, and B. Medler, “Intelligent vol. 5, 2005. adaptation of digital game-based learning.” in Proc. ACM FuturePlay08, [29] S. Combéfis and J. Wautelet, “Programming trainings and informatics 2008, pp. 200–203. teaching through online contests.” Olympiads in Informatics, vol. 21, [38] R. Lindberg and T. Laine, “Detecting play and learning styles for 2014. adaptive educational games.” in Proc. CSEDU 2016, 2016, pp. 181– 189. [30] E. Enstrom, G. Kreitz, F. Niemela, P. Soderman, and V. Kann, “Five [39] C. Loh, “Designing online games assessment as information trails.” in years with kattis – us-ing an automated assessment system in teaching.” Gibson, D., Al-drich, C., Prensky, M. (eds.) Games and Simulations in in Proc. FIE 2011, 2011. Online Learning: Research and De-velopment Frameworks. Infosci, [31] S. Edwards and M. Perez-Quinones, “Web-cat: automatically grading 2007, pp. 323–348. programming assignments.” in Proc. ITiCSE 2008, 2008. [40] M. Ilahi-Amri, L. Cheniti-Belcadhi, and R. Braham, “A framework for [32] D. de Souza, J. Maldonado, and E. Barbosa, “Progtest: An environment competence based e-assessment.” Interaction Design and Architecture(s) for the sub-mission and evaluation of programming assignments.” in Journal (IxD&A), vol. 32, pp. 189–204, 2017. Proc. IEEE SEET 2011, 2011. [33] W. Di Luigi, G. Farina, L. Laura, U. Nanni, M. Temperini, and