A Review of Techniques in Automatic Programming Assessment for Practical Skill Test

Authors

  • Adidah Lajis Universiti Kuala Lumpur Malaysian Institute of Information Technology
  • Shahidatul Arfah Baharudin Universiti Kuala Lumpur Malaysian Institute of Information Technology
  • Diyana Ab Kadir Universiti Kuala Lumpur Malaysian Institute of Information Technology
  • Nadilah Mohd Ralim Universiti Kuala Lumpur Malaysian Institute of Information Technology
  • Haidawati Mohd Nasir Universiti Kuala Lumpur Malaysian Institute of Information Technology
  • Normaziah Abdul Aziz Kulliyah of Information Technology International Islamic University Malaysia

Keywords:

Cognitive Assessment, Assisted Assessment, Programming, Competency-Based,

Abstract

Computer programming ability is a challenging competency that requires several cognitive skills and extensive practice. The increased number of students enrolled in computer and engineering courses and also the increased of failure and drop rate in programming subject is the motivational factor to this research. Due to the importance of this skill, this paper intends to study the landscape of current scenario in assisted assessment for hands-on practical programming focusing on competency-based assessment. The Bloom Taxonomy is used as a competency-based assessment platform. The review showed to-date that there are several automatic assessments for programming skills. However, there is no common grading being applied. Thus, further research is required to propose an automatic assessment that grades the student achievement based on learning taxonomy such as Bloom Cognitive Competency model.

References

Mohayidin, Mohd Ghazali (2008). "Implementation of Outcome-Based Education in Universiti Putra Malaysia: A Focus on Students' Learning Outcomes". International Education Studies. 1(4). Retrieved 23 October 2014.

G, R.V., & Jayaprakash (2009). Classification of Cognitive Difficulties of Students to Learn Computer Programming.

A. McGettrick program-ability, (2005). "Grand challenges in Computing: Education – A Summary", The Computer Journal Vol. 48

A. Robins et al., (2003). “Learning and Teaching Programming: A Review and Discussion”, Computer Science Education Journal, Vol. 13

Parson, D., Wood, K. & Haden, Patricia, (2015). What are we doing when we assess programming?. Proceedings of the 17th Australasian Computing Education Conference (ACE 2015), Sydney, Australia, 27 - 30 January 2015

Gronlund, N.E. (2000). How to write and use instructional objectives. Toronto: Prentice-Hall

Clark, D. (2005) Softskills and E-learning. London: Epic Performance Improvement Limited.

MacLeod, A. (2000). The importance of soft skills in the current Canadian labour market. Sectoral and Occupational Studies Division of Human Resources Development Canada, April.

Krathwohl, D. R., Bloom, B. S. & Masia, B. B. (1964). Taxonomy of Educational Objectives. [Handbook 2: Affective Domain]. London: Longman

Nghi Truong, Paul Roe, & Peter Bancroft. (2004). "Static Analysis of Students’ Java Programs". Paper read at 6th Australian Computing Education

T. Wang, X. Su, Y. Wang and P. Ma, (2007). ” Semantic SimilarityBased Grading of Student Programs,” Information and Software Technology, Vol. 49, No. 2, 2007, pp. 99-107. doi:10.1016/j.infsof.2006.03.001

N. Truong, P. Bancroft and P. Roe, (2002). “ELP-A Web Environment for Learning to Program,” Proceeding of the 19th Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education, Vol. 19, Auckland, 8-11 December 2002, pp. 661-670.

Khirulnizam Abd Rahman, Syarbaini Ahmad & Md Jan Nordin (2007). The Design of an Automated C Programming Assessment Using Pseudo-code Comparison Technique. National Conference on Software Engineering and Computer Systems 2007, organized by Universiti Malaysia Pahang, Pahang, Malaysia.

Stephens D., Bull, J. E. and Wade, W. (2011). Computer-assisted assessment: suggested guideline for an institutional strategy. Online Learning. Part V: Online Assessment. London: Sage

Aizyl Azlee (2016). Coding to be in school curricula next year, says MDEC CEO, MalayMail Online, Retrieved: 20 July 2017, http://www.themalaymailonline.com/malaysia/article/coding-to-be-inschool-curricula-next-year-says-mdec-ceo

McCracken, M., Almstrum, V., Diaz, D., Guzdial, M., Hagan, D., Kolikant, Y. B.-D., Laxer, C., Thomas, L., Utting, I., and Wilusz, T., (2001). A multi-national, multi-institutional study of assessment of programming skills of first-year CS students. SIGCSE Bulletin 33(4), 125–180

James H. Block., (1978), ‘The ‘C’ in CBE’, Educational Researcher, 7(5), pp. 13-16.

Simpson E.J. (1972). The Classification of Educational Objectives in the Psychomotor Domain. Washington, DC: Gryphon House.

Harrow, A. (1972) A Taxonomy of Psychomotor Domain: A Guide for Developing Behavioral Objectives. New York: David McKay.

Lister, R. and Leaney, J. (2003), Introductory programming, criterion-referencing, SIGCSE ‘03: Proceedings of the 34th SIGCSE technical symposium on Computer science education , 143-147, ACM Press

Oliver, D., Dobele, T., Greber, M. and Roberts, T. (2004), This course has a Bloom Rating of 3.9. in Proceedings of the sixth conference on Australasian computing education - Volume 30 , Dunedin, New Zealand, 227-231, Australian Computer Society Inc

Lister, R., Adams, E.S., Fitzgerald, S., Fone, W., Hamer, J., Lindholm, M., McCartney, R., Moström, J.E., Sanders, K., Seppälä, O., Simon, B. and Thomas, L. (2004) A multi-national study of reading and tracing skills in novice programmers. Inroads - The SIGCSE Bulletin, 36(4). 119-150.

Shneider, E. and Gladkikh, O. (2006) Designing questioning strategies for information technology courses. Mann, S. and Bridgeman, N. eds. The 19th Annual Conference of the National Advisory Committee on Computing Qualifications: Preparing for the Future — Capitalising on IT , Wellington, 243- 248, National Advisory Committee on Computing Qualifications.

Thompson, E., Luxton-Reilly, A., Whalley, J. L., Hu, M. and Robbins, P. (2008), Bloom's taxonomy for CS assessment. Proceedings of the Tenth Conference Australasian Computing Education (Wollongong, NSW, Australia, January 20-23, 2008), 155-161.

Ala-Mutka, K. M. (2005). A survey of automated assessment approaches for programming assignments. Computer science education, 15(2), 83-102.

N. Truong, P. Roe and P. Bancroft, (2004), “Static Analysis of Students’ Java Programs,” Proceedings of the 6th Conference on Australasian Computing Education, Vol. 30, p. 325.

K. A. Naude, J. H. Greyling and D. Vogts, (2010), “Marking Student Programs Using Graph Similarity,” Computers & Education, Vol. 54, No. 2, pp. 545-561

Shuhida, Shuhida, Hamilton, M., D’Souza, D. (2009). A Taxonomic Study of Novice Programming Summative Assessment, Eleventh Australasian Computing Education Conference (ACE2009), Wellington, New Zealand, January 2009

Alaoutinen, S, & Smolander, K. (2010). Student self-assessment in a programming course using bloom's revised taxonomy. In Proceedings of the fifteenth annual conference on Innovation and technology in computer science education (pp. 155-159). ACM, New York, NY, USA.

Romli, R., Sulaiman, S., Zamli, K. Z. (2010). Automatic Programming Assessment and Test Data Generation a Review on its Approaches. Information Technology (ITSim), 2010 International Symposium, pp. 1186-1192

Fatima Al Shamsi, Ashraf Elnagar (2012). An Intelligent Assessment Tool for Students’ Java Submissions in Introductory Programming Courses. Journal of Intelligent Learning Systems and Applications, 2012, 4, 59-69

Rahman, K. A., & Nordin, M. J. (2007). A review on the static analysis approach in the automated programming assessment systems.

Athanasios Tsintsifas. (2002). A Framework for the Computer Based Assessment of Diagram-Based Coursework. PhD thesis.

Michael Blumenstein, Steve Green, Ann Nguyen, and Vallipuram Muthukkumarasamy, (2004), GAME: A generic automated marking environment for programming assessment. In Proceedings of the International Conference on Information Technology: Coding and Computing (ITCC’04) Volume 2, pages 212–, Washington, DC, USA, 2004. IEEE Computer Society.

Alaoutinen, S, & Smolander, K. (2010). Student self-assessment in a programming course using bloom's revised taxonomy. In Proceedings of the fifteenth annual conference on Innovation and technology in computer science education (pp. 155-159). ACM, New York, NY, USA

Srikant, S., and Aggarwal, V. (2014). A system to grade computer programming skills using machine learning. In Proc. 20th ACM SIGKDD Intl. Conf. on Knowledge Discovery and Data Mining (Aug. 2014), 1887–1896

Roozbeh Matloobi, Michael Myer Blumenstein, and Steven Green. (2009), Extensions to generic automated marking environment: Game- 2+.

Cara MacNish (2007). Longitudinal Syntactic Analysis of Laboratory Submission for Examining Problem – Solving Behavior, Advanced Learning Technologies , 2007, ICALT 2007, Seventh IEEE International Conference.

Susan A. Mengel and Vinay Yerramilli. (1999). A case study of the static analysis of the quality of novice student programs. SIGCSE Bull., 31:78–82, March 1999

Verifysoft Technology GmbH. Verifysoft (2010), Halstead metrics. http://www.verifysoft.com/en_halstead_metrics.html, June 2010.

Deek, F.P., & McHugh, J. (2000), Problem-solving methodologies and the development of critical thinking skills. Journal of Computer Science Education, 14(1-2), 6-12.

Brun, Y., & Ernst, M. D. (2004, May). Finding latent code errors via machine learning over program executions. In Proceedings of the 26th International Conference on Software Engineering (pp. 480-490). IEEE Computer Society.

Deek F. P., McHugh J. A. (1998), "A survey and critical analysis of tools for learning programming". Computer Science Education, Vol.8, n°2, p. 130-178

Secretary’s Commission on Achieving Necessary Skills (SCANS). What work requires of schools A SCANS Report for America (2000). Washington DC: US Labor Department. http://wdr.doleta.gov/SCANS/whatwork/whatwork.pdf.

ASPIRINGMINDS (2017). AUTOMATA. Retrieved 23 July 2017 at http://www.aspiringminds.com/technology/automata

Brusilovsky, P. & Sosnovsky, S. (2005). Individualized Exercises for Self-Assessment of Programming Knowledge: An Evaluation of QuizPACK, Journal on Educational Resources in Computing (JERIC), 5(3), Article 6

Ministry Higher Education (2016). Malaysia Higher Education Blueprint 2015-2025, available at https://www.acu.ac.uk/events/perspectives/datin-siti-hamisahpresentation, 21 September 2017

Koyya, P., Lee, Y, & Yang, J. (2013). Feedback for Programming Assignments Using Software-Metrics and Reference Code, ISRN Software Engineering Volume 2013 (2013), Article ID 805963, 8 pages

Higgins, C. A., Gray, G., Symeonidis, P., Tsintsifas, A. (2005). Automated Assessment and Experiences of Teaching Programming. Journal on Educational Resources in Computing (JERIC), vol. 5, pp. 5.

Wang, T., Su, X., Ma, P., Wang, Y., Wang, K. (2011). Ability-trainingoriented Automated Assessment in Introductory Programming Course. Computer. Education, Elsevier, vol. 56, pp. 220-226

Jelemenská, K. Čičák, (2012). Improved Assignments Management in MOODLE Environment. INTED2012 Proceedings, pp. 1809-1817.

Queirós, R. A. P., Leal, J. P. (2012). PETCHA: A Programming Exercises Teaching Assistant. Proceedings of the 17th ACM Annual Conference on Innovation and Technology in Computer Science Education, pp. 192-197

Milena Vujoˇsevi´c-Janiˇci´c · Mladen Nikoli´c · Duˇsan Toˇsi´c · Viktor Kuncak (2012). Software Verification and Graph Similarity for Automated Evaluation of Students’ Assignments. Information and Software Technology Volume 55, Issue 6, June 2013, Pages 1004-1016

Rivers, K. & Kenneth R. Koedinger (2013). Automatic Generation of Programming Feedback: A Data-Driven Approach, AIED 2013 Workshops Proceedings Volume 9

Fonte, D., Daniela da Cruz, Alda Lopes Gancarski, && Henriques, P. R. (2013). A Flexible Dynamic System for Automatic Grading of Programming Exercise. 2nd Symposium on Languages, Application and Technologies (SLATE 13)

Anuj Shah. (2003), Web-CAT: A Web-based Center for Automated Testing. Technical report, Virginia Polytechnic Institute and State University, Blacksburg, VA, USA, 2003

Don Colton, Leslie Fife, and Andrew Thompson. (2006), A Web-based Automatic Program Grader. Information Systems Education Journal (ISEDJ), 4(114), November 2006.

Riku Saikkonen, Lauri Malmi, and Ari Korhonen. (2001), Fully automatic assessment of programming exercises. In Proceedings of the 6th annual conference on Innovation and technology in computer science education, ITiCSE ’01, pages 133–136, New York, USA, 2001. ACM

Daniela Fonte, Ismael Vilas Boas, Daniela da Cruz, Alda Lopes Gançarski, and Pedro Rangel Henriques. (2012), Program analysis and evaluation using quimera. In ICEIS’2012 — 14th International Conference on Enterprise Information Systems, pages 209–219. INSTICC, June 2012.

S D Benford, E K Burke, E Foxley, and C A Higgins. (1995), The Ceilidh system for the automatic grading of students on programming courses. In Proceedings of the 33rd annual on Southeast regional conference, ACM-SE 33, pages 176–182, New York, NY, USA, 1995. ACM.

José Paulo Leal and Fernando Silva. (2003), Mooshak: aWeb-based multi-site programming contest system. Software: Practice and Experience, 33(6):567–581, May 2003.

D.S. Morris. (2002), Automatically grading Java programming assignments via reflection, inheritance, and regular expressions. In Frontiers in Education, 2002. FIE 2002. 32nd Annual, volume 1, pages T3G–22, 2002.

Edward L. Jones. (2000), Grading student programs - a software testing approach. In Proceedings of the second annual CCSC on Computing in Small Colleges Northwestern conference, pages 185–192, USA, 2000. Consortium for Computing Sciences in Colleges.

Emma Enstr¨om, Gunnar Kreitz, Fredrik Niemela, Pehr Soderman, and Viggo Kann. (2011), Five Years with Kattis – Using an Automated Assessment System in Teaching. In Frontiers in Education Conference (FIE), 2011. Institute of Electrical and Electronics Engineers, Piscataway, NJ, T3J–1.

Caiza, J. C., & Ramiro, Á. (2013). Programming assignments automatic grading: review of tools and implementations

Downloads

Published

2018-07-04

How to Cite

Lajis, A., Baharudin, S. A., Ab Kadir, D., Mohd Ralim, N., Mohd Nasir, H., & Abdul Aziz, N. (2018). A Review of Techniques in Automatic Programming Assessment for Practical Skill Test. Journal of Telecommunication, Electronic and Computer Engineering (JTEC), 10(2-5), 109–113. Retrieved from https://jtec.utem.edu.my/jtec/article/view/4394

Issue

Vol. 10 No. 2-5: Embracing The 4th Industrial Revolution: Innovation in Telecommunication, Electronic & Computer Engineering I

Section

Articles

License

TRANSFER OF COPYRIGHT AGREEMENT

The manuscript is herewith submitted for publication in the Journal of Telecommunication, Electronic and Computer Engineering (JTEC). It has not been published before, and it is not under consideration for publication in any other journals. It contains no material that is scandalous, obscene, libelous or otherwise contrary to law. When the manuscript is accepted for publication, I, as the author, hereby agree to transfer to JTEC, all rights including those pertaining to electronic forms and transmissions, under existing copyright laws, except for the following, which the author(s) specifically retain(s):

  1. All proprietary right other than copyright, such as patent rights
  2. The right to make further copies of all or part of the published article for my use in classroom teaching
  3. The right to reuse all or part of this manuscript in a compilation of my own works or in a textbook of which I am the author; and
  4. The right to make copies of the published work for internal distribution within the institution that employs me

I agree that copies made under these circumstances will continue to carry the copyright notice that appears in the original published work. I agree to inform my co-authors, if any, of the above terms. I certify that I have obtained written permission for the use of text, tables, and/or illustrations from any copyrighted source(s), and I agree to supply such written permission(s) to JTEC upon request.