A Review of Techniques in Automatic Programming Assessment for Practical Skill Test
Keywords:Cognitive Assessment, Assisted Assessment, Programming, Competency-Based,
AbstractComputer programming ability is a challenging competency that requires several cognitive skills and extensive practice. The increased number of students enrolled in computer and engineering courses and also the increased of failure and drop rate in programming subject is the motivational factor to this research. Due to the importance of this skill, this paper intends to study the landscape of current scenario in assisted assessment for hands-on practical programming focusing on competency-based assessment. The Bloom Taxonomy is used as a competency-based assessment platform. The review showed to-date that there are several automatic assessments for programming skills. However, there is no common grading being applied. Thus, further research is required to propose an automatic assessment that grades the student achievement based on learning taxonomy such as Bloom Cognitive Competency model.
Mohayidin, Mohd Ghazali (2008). "Implementation of Outcome-Based Education in Universiti Putra Malaysia: A Focus on Students' Learning Outcomes". International Education Studies. 1(4). Retrieved 23 October 2014.
G, R.V., & Jayaprakash (2009). Classification of Cognitive Difficulties of Students to Learn Computer Programming.
A. McGettrick program-ability, (2005). "Grand challenges in Computing: Education – A Summary", The Computer Journal Vol. 48
A. Robins et al., (2003). “Learning and Teaching Programming: A Review and Discussion”, Computer Science Education Journal, Vol. 13
Parson, D., Wood, K. & Haden, Patricia, (2015). What are we doing when we assess programming?. Proceedings of the 17th Australasian Computing Education Conference (ACE 2015), Sydney, Australia, 27 - 30 January 2015
Gronlund, N.E. (2000). How to write and use instructional objectives. Toronto: Prentice-Hall
Clark, D. (2005) Softskills and E-learning. London: Epic Performance Improvement Limited.
MacLeod, A. (2000). The importance of soft skills in the current Canadian labour market. Sectoral and Occupational Studies Division of Human Resources Development Canada, April.
Krathwohl, D. R., Bloom, B. S. & Masia, B. B. (1964). Taxonomy of Educational Objectives. [Handbook 2: Affective Domain]. London: Longman
Nghi Truong, Paul Roe, & Peter Bancroft. (2004). "Static Analysis of Students’ Java Programs". Paper read at 6th Australian Computing Education
T. Wang, X. Su, Y. Wang and P. Ma, (2007). ” Semantic SimilarityBased Grading of Student Programs,” Information and Software Technology, Vol. 49, No. 2, 2007, pp. 99-107. doi:10.1016/j.infsof.2006.03.001
N. Truong, P. Bancroft and P. Roe, (2002). “ELP-A Web Environment for Learning to Program,” Proceeding of the 19th Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education, Vol. 19, Auckland, 8-11 December 2002, pp. 661-670.
Khirulnizam Abd Rahman, Syarbaini Ahmad & Md Jan Nordin (2007). The Design of an Automated C Programming Assessment Using Pseudo-code Comparison Technique. National Conference on Software Engineering and Computer Systems 2007, organized by Universiti Malaysia Pahang, Pahang, Malaysia.
Stephens D., Bull, J. E. and Wade, W. (2011). Computer-assisted assessment: suggested guideline for an institutional strategy. Online Learning. Part V: Online Assessment. London: Sage
Aizyl Azlee (2016). Coding to be in school curricula next year, says MDEC CEO, MalayMail Online, Retrieved: 20 July 2017, http://www.themalaymailonline.com/malaysia/article/coding-to-be-inschool-curricula-next-year-says-mdec-ceo
McCracken, M., Almstrum, V., Diaz, D., Guzdial, M., Hagan, D., Kolikant, Y. B.-D., Laxer, C., Thomas, L., Utting, I., and Wilusz, T., (2001). A multi-national, multi-institutional study of assessment of programming skills of first-year CS students. SIGCSE Bulletin 33(4), 125–180
James H. Block., (1978), ‘The ‘C’ in CBE’, Educational Researcher, 7(5), pp. 13-16.
Simpson E.J. (1972). The Classification of Educational Objectives in the Psychomotor Domain. Washington, DC: Gryphon House.
Harrow, A. (1972) A Taxonomy of Psychomotor Domain: A Guide for Developing Behavioral Objectives. New York: David McKay.
Lister, R. and Leaney, J. (2003), Introductory programming, criterion-referencing, SIGCSE ‘03: Proceedings of the 34th SIGCSE technical symposium on Computer science education , 143-147, ACM Press
Oliver, D., Dobele, T., Greber, M. and Roberts, T. (2004), This course has a Bloom Rating of 3.9. in Proceedings of the sixth conference on Australasian computing education - Volume 30 , Dunedin, New Zealand, 227-231, Australian Computer Society Inc
Lister, R., Adams, E.S., Fitzgerald, S., Fone, W., Hamer, J., Lindholm, M., McCartney, R., Moström, J.E., Sanders, K., Seppälä, O., Simon, B. and Thomas, L. (2004) A multi-national study of reading and tracing skills in novice programmers. Inroads - The SIGCSE Bulletin, 36(4). 119-150.
Shneider, E. and Gladkikh, O. (2006) Designing questioning strategies for information technology courses. Mann, S. and Bridgeman, N. eds. The 19th Annual Conference of the National Advisory Committee on Computing Qualifications: Preparing for the Future — Capitalising on IT , Wellington, 243- 248, National Advisory Committee on Computing Qualifications.
Thompson, E., Luxton-Reilly, A., Whalley, J. L., Hu, M. and Robbins, P. (2008), Bloom's taxonomy for CS assessment. Proceedings of the Tenth Conference Australasian Computing Education (Wollongong, NSW, Australia, January 20-23, 2008), 155-161.
Ala-Mutka, K. M. (2005). A survey of automated assessment approaches for programming assignments. Computer science education, 15(2), 83-102.
N. Truong, P. Roe and P. Bancroft, (2004), “Static Analysis of Students’ Java Programs,” Proceedings of the 6th Conference on Australasian Computing Education, Vol. 30, p. 325.
K. A. Naude, J. H. Greyling and D. Vogts, (2010), “Marking Student Programs Using Graph Similarity,” Computers & Education, Vol. 54, No. 2, pp. 545-561
Shuhida, Shuhida, Hamilton, M., D’Souza, D. (2009). A Taxonomic Study of Novice Programming Summative Assessment, Eleventh Australasian Computing Education Conference (ACE2009), Wellington, New Zealand, January 2009
Alaoutinen, S, & Smolander, K. (2010). Student self-assessment in a programming course using bloom's revised taxonomy. In Proceedings of the fifteenth annual conference on Innovation and technology in computer science education (pp. 155-159). ACM, New York, NY, USA.
Romli, R., Sulaiman, S., Zamli, K. Z. (2010). Automatic Programming Assessment and Test Data Generation a Review on its Approaches. Information Technology (ITSim), 2010 International Symposium, pp. 1186-1192
Fatima Al Shamsi, Ashraf Elnagar (2012). An Intelligent Assessment Tool for Students’ Java Submissions in Introductory Programming Courses. Journal of Intelligent Learning Systems and Applications, 2012, 4, 59-69
Rahman, K. A., & Nordin, M. J. (2007). A review on the static analysis approach in the automated programming assessment systems.
Athanasios Tsintsifas. (2002). A Framework for the Computer Based Assessment of Diagram-Based Coursework. PhD thesis.
Michael Blumenstein, Steve Green, Ann Nguyen, and Vallipuram Muthukkumarasamy, (2004), GAME: A generic automated marking environment for programming assessment. In Proceedings of the International Conference on Information Technology: Coding and Computing (ITCC’04) Volume 2, pages 212–, Washington, DC, USA, 2004. IEEE Computer Society.
Alaoutinen, S, & Smolander, K. (2010). Student self-assessment in a programming course using bloom's revised taxonomy. In Proceedings of the fifteenth annual conference on Innovation and technology in computer science education (pp. 155-159). ACM, New York, NY, USA
Srikant, S., and Aggarwal, V. (2014). A system to grade computer programming skills using machine learning. In Proc. 20th ACM SIGKDD Intl. Conf. on Knowledge Discovery and Data Mining (Aug. 2014), 1887–1896
Roozbeh Matloobi, Michael Myer Blumenstein, and Steven Green. (2009), Extensions to generic automated marking environment: Game- 2+.
Cara MacNish (2007). Longitudinal Syntactic Analysis of Laboratory Submission for Examining Problem – Solving Behavior, Advanced Learning Technologies , 2007, ICALT 2007, Seventh IEEE International Conference.
Susan A. Mengel and Vinay Yerramilli. (1999). A case study of the static analysis of the quality of novice student programs. SIGCSE Bull., 31:78–82, March 1999
Verifysoft Technology GmbH. Verifysoft (2010), Halstead metrics. http://www.verifysoft.com/en_halstead_metrics.html, June 2010.
Deek, F.P., & McHugh, J. (2000), Problem-solving methodologies and the development of critical thinking skills. Journal of Computer Science Education, 14(1-2), 6-12.
Brun, Y., & Ernst, M. D. (2004, May). Finding latent code errors via machine learning over program executions. In Proceedings of the 26th International Conference on Software Engineering (pp. 480-490). IEEE Computer Society.
Deek F. P., McHugh J. A. (1998), "A survey and critical analysis of tools for learning programming". Computer Science Education, Vol.8, n°2, p. 130-178
Secretary’s Commission on Achieving Necessary Skills (SCANS). What work requires of schools A SCANS Report for America (2000). Washington DC: US Labor Department. http://wdr.doleta.gov/SCANS/whatwork/whatwork.pdf.
ASPIRINGMINDS (2017). AUTOMATA. Retrieved 23 July 2017 at http://www.aspiringminds.com/technology/automata
Brusilovsky, P. & Sosnovsky, S. (2005). Individualized Exercises for Self-Assessment of Programming Knowledge: An Evaluation of QuizPACK, Journal on Educational Resources in Computing (JERIC), 5(3), Article 6
Ministry Higher Education (2016). Malaysia Higher Education Blueprint 2015-2025, available at https://www.acu.ac.uk/events/perspectives/datin-siti-hamisahpresentation, 21 September 2017
Koyya, P., Lee, Y, & Yang, J. (2013). Feedback for Programming Assignments Using Software-Metrics and Reference Code, ISRN Software Engineering Volume 2013 (2013), Article ID 805963, 8 pages
Higgins, C. A., Gray, G., Symeonidis, P., Tsintsifas, A. (2005). Automated Assessment and Experiences of Teaching Programming. Journal on Educational Resources in Computing (JERIC), vol. 5, pp. 5.
Wang, T., Su, X., Ma, P., Wang, Y., Wang, K. (2011). Ability-trainingoriented Automated Assessment in Introductory Programming Course. Computer. Education, Elsevier, vol. 56, pp. 220-226
Jelemenská, K. Čičák, (2012). Improved Assignments Management in MOODLE Environment. INTED2012 Proceedings, pp. 1809-1817.
Queirós, R. A. P., Leal, J. P. (2012). PETCHA: A Programming Exercises Teaching Assistant. Proceedings of the 17th ACM Annual Conference on Innovation and Technology in Computer Science Education, pp. 192-197
Milena Vujoˇsevi´c-Janiˇci´c · Mladen Nikoli´c · Duˇsan Toˇsi´c · Viktor Kuncak (2012). Software Verification and Graph Similarity for Automated Evaluation of Students’ Assignments. Information and Software Technology Volume 55, Issue 6, June 2013, Pages 1004-1016
Rivers, K. & Kenneth R. Koedinger (2013). Automatic Generation of Programming Feedback: A Data-Driven Approach, AIED 2013 Workshops Proceedings Volume 9
Fonte, D., Daniela da Cruz, Alda Lopes Gancarski, && Henriques, P. R. (2013). A Flexible Dynamic System for Automatic Grading of Programming Exercise. 2nd Symposium on Languages, Application and Technologies (SLATE 13)
Anuj Shah. (2003), Web-CAT: A Web-based Center for Automated Testing. Technical report, Virginia Polytechnic Institute and State University, Blacksburg, VA, USA, 2003
Don Colton, Leslie Fife, and Andrew Thompson. (2006), A Web-based Automatic Program Grader. Information Systems Education Journal (ISEDJ), 4(114), November 2006.
Riku Saikkonen, Lauri Malmi, and Ari Korhonen. (2001), Fully automatic assessment of programming exercises. In Proceedings of the 6th annual conference on Innovation and technology in computer science education, ITiCSE ’01, pages 133–136, New York, USA, 2001. ACM
Daniela Fonte, Ismael Vilas Boas, Daniela da Cruz, Alda Lopes Gançarski, and Pedro Rangel Henriques. (2012), Program analysis and evaluation using quimera. In ICEIS’2012 — 14th International Conference on Enterprise Information Systems, pages 209–219. INSTICC, June 2012.
S D Benford, E K Burke, E Foxley, and C A Higgins. (1995), The Ceilidh system for the automatic grading of students on programming courses. In Proceedings of the 33rd annual on Southeast regional conference, ACM-SE 33, pages 176–182, New York, NY, USA, 1995. ACM.
José Paulo Leal and Fernando Silva. (2003), Mooshak: aWeb-based multi-site programming contest system. Software: Practice and Experience, 33(6):567–581, May 2003.
D.S. Morris. (2002), Automatically grading Java programming assignments via reflection, inheritance, and regular expressions. In Frontiers in Education, 2002. FIE 2002. 32nd Annual, volume 1, pages T3G–22, 2002.
Edward L. Jones. (2000), Grading student programs - a software testing approach. In Proceedings of the second annual CCSC on Computing in Small Colleges Northwestern conference, pages 185–192, USA, 2000. Consortium for Computing Sciences in Colleges.
Emma Enstr¨om, Gunnar Kreitz, Fredrik Niemela, Pehr Soderman, and Viggo Kann. (2011), Five Years with Kattis – Using an Automated Assessment System in Teaching. In Frontiers in Education Conference (FIE), 2011. Institute of Electrical and Electronics Engineers, Piscataway, NJ, T3J–1.
Caiza, J. C., & Ramiro, Á. (2013). Programming assignments automatic grading: review of tools and implementations
How to Cite
TRANSFER OF COPYRIGHT AGREEMENT
The manuscript is herewith submitted for publication in the Journal of Telecommunication, Electronic and Computer Engineering (JTEC). It has not been published before, and it is not under consideration for publication in any other journals. It contains no material that is scandalous, obscene, libelous or otherwise contrary to law. When the manuscript is accepted for publication, I, as the author, hereby agree to transfer to JTEC, all rights including those pertaining to electronic forms and transmissions, under existing copyright laws, except for the following, which the author(s) specifically retain(s):
- All proprietary right other than copyright, such as patent rights
- The right to make further copies of all or part of the published article for my use in classroom teaching
- The right to reuse all or part of this manuscript in a compilation of my own works or in a textbook of which I am the author; and
- The right to make copies of the published work for internal distribution within the institution that employs me
I agree that copies made under these circumstances will continue to carry the copyright notice that appears in the original published work. I agree to inform my co-authors, if any, of the above terms. I certify that I have obtained written permission for the use of text, tables, and/or illustrations from any copyrighted source(s), and I agree to supply such written permission(s) to JTEC upon request.