WEB 2.0 AUTOMATED ESSAY SCORING APPLICATION AND HUMAN ESL ESSAY ASSESSMENT: A COMPARISON STUDY

Authors

  • Mohammad Radzi Manap PhD, Akademi Pengajian Bahasa, Universiti Teknologi MARA, Shah Alam, Selangor, Malaysia
  • Nor Fazlin Ramli PhD, Akademi Pengajian Bahasa, Universiti Teknologi MARA, Shah Alam, Selangor, Malaysia
  • Aini Akmar Mohd Kassim PhD, Akademi Pengajian Bahasa, Universiti Teknologi MARA, Shah Alam, Selangor, Malaysia

DOI:

https://doi.org/10.46827/ejel.v0i0.2650

Keywords:

Computer Assisted Language Learning, e-learning, web-based 2.0 application, Automated Essay Scoring

Abstract

Educators from all levels of education have comfortably embraced ICT for decades. More and more applications are being developed which has direct and indirect advantages on learning and teaching. What is more interesting is the fact that more free applications are now being made available to be used by the public. In this paper, the focus is on the comparison between a free Automated Essay Scoring (AES) web-based application called ‘PaperRater.com’ and human assessment by English instructors from a public university in Malaysia. Ten selected human assessed essays were assessed by individual lecturer (IL) and also a group of lecturers (GL). Those essays were then fed into ‘PaperRater.com’ and a comparison was made in terms of total scores generated by the application and the ones by the instructors. A descriptive statistical analysis was carried out to compare the scores. As a result, assessment for both types of essay recorded a difference of between 0.3 to 38.7 marks where bigger disparity was recorded between human assessed and computer based (AES) assessment. Overall, however, the strength of the relationship between GL and PR, based on the Pearson’s product moment correlation coefficient, was recorded at 0.678407 which means that there is a moderate positive linear relationship.

 

Article visualizations:

Hit counter

DOI

Author Biographies

  • Mohammad Radzi Manap, PhD, Akademi Pengajian Bahasa, Universiti Teknologi MARA, Shah Alam, Selangor, Malaysia

    Mohammad Radzi Manap (PhD) is a senior lecturer at the department of English and Linguistics at the Akademi Pengajian Bahasa, Universiti Teknologi MARA with more than 20 years of experience as instructors of various English language and linguistics courses at tertiary level. His research interests include the use of technology in language teaching, TESL and sociolinguistics.

  • Nor Fazlin Ramli, PhD, Akademi Pengajian Bahasa, Universiti Teknologi MARA, Shah Alam, Selangor, Malaysia

    Nor Fazlin Mohd Ramli (PhD) is a senior lecturer at the department of English and Linguistics at the Akademi Pengajian Bahasa, Universiti Teknologi MARA who has great interest in the field of technology in language learning, instructional design as well as TESL.

  • Aini Akmar Mohd Kassim, PhD, Akademi Pengajian Bahasa, Universiti Teknologi MARA, Shah Alam, Selangor, Malaysia

    Aini Akmar Mohd Kassim (PhD) is a senior lecturer at the department of English and Linguistics at the Akademi Pengajian Bahasa, Universiti Teknologi MARA.

References

Attali, Y. & Burstein, J. (2006). Automated Essay Scoring With e-rater® V.2. Journal of Technology, Learning and Assessment, 4(3). Available from http://www.jtla.org

Creswell, J. (2009). Research design: Qualitative, quantitative, and mixed methods approaches (3rd ed.). Thousand Oaks, CA: Sage.

Dikli, S. (2006). Automated Essay Scoring. Turkish Online of Distance Education. Vol. 7 (1). pp 49-62.

Grimes, D. & Warschauer, M. (2010). Utility in a Fallible Tool: A Multi-Site Case Study of Automated Writing Evaluation. Journal of Technology, Learning, and Assessment,8(6). Retrieved [22.5.2016] from http://www.jtla.org.

Hinkle D.E, Wiersma W, Jurs. (2003). 'Applied Statistics for the Behavioral Sciences'.

Kheradmand, N. and Sayadiyan, S. (2016). Comparative Investigation of the Effects of Immediate and Delayed Error Correction on the Achievement of Male and Female Iranian EFL Learners’ Writing Skill. International Journal of Social Science and Education. Vol. 6 (1). Retrieved from www.ijsse.com/sites/default/files/issues/2016/v6i1/paper-04.pdf [21 July 2015]

Moore, D. S., Notz, W. I., & Flinger, M. A. (2013). The basic practice of statistics (6th ed.). New York, NY: W. H. Freeman and Company.

Mo Zhang. (2013). Contrasting Automated and Human Scoring of Essays. R&D Connections, 21. Retrieved [11.6.16] from 2013 https://www.ets.org/Media/Research/pdf/RD_Connections_21.pdf

Nafizah Hamidun, Shafiq Hizwari Md Hashim and Nur Farhinaa Othman. (2012). Enhancing Students’ Motivation by Providing Feedback on Writing: The Case of International Students from Thailand. International Journal of Social Science and Humanity. Vol. 2 (6). Retrieved [21 July 2016] from www.ijssh.org/papers/179-A10062.pdf

Neda Kheradmand and Sima Sayadiyan. (2016). Comparative Investigation of the Effects and Delayed Error Correction on the Achievement of Male and Female Iranian EFL Learners’ Writing Skill. International Journal Soc. Sci. & Education. Vol. 6 (1). Retrieved [21 July 2016] from www.ijsse.com/sites/default/files/issues/2016/v6i1/paper-04.pdf

Ng Sing Yii, Bong Chih How, Lee Nung Kion and Hong Kian Sam. (2016). Automated Essay Scoring Feedback (AESF): An Innovative Writing Solution to the Malaysian University English Test (MUET). International Journal on E-Learning and Higher Education, 4. pp. 130-143.

Qiu Yubing. (2016). Pigai Smart Essay Scoring System and Its Implications for Teaching English Writing. Journal of Applied Science and Engineering Innovation, Vol. 3 (6). pp. 217-219. Retrieved [2 February 2018] from http://www.jasei.org/PDF/3-6/3-217-219.pdf

Saadiyah Darus, Siti Hanim Stapa, Supyan Hussin & Koo Yew Lie. (2000). A Survey of Computer–based Essay Marking (CBEM) Systems. International Conference “Education & ICT in the New Millenium’. pp 519-528. Retrieved [14.3.2019] from https://www.academia.edu/6977567/A_survey_of_computer_based_essay_marking_CBEM_system

Saadiyah Darus, Siti Hamin Stapa & Supyan Hussin. (2003). Experimenting A Computer-Based Essay Marking System At Universiti Kebangsaan Malaysia. Jurnal Teknologi, 39(E) Dis. 2003: 1–18. Retrieved [23.5.2016] from http://www.academia.edu/263332/Experimenting_a_Computer-Based_Essay_Marking_System_at_Universiti_Kebangsaan_Malaysia

Shermis, M.D., Burstein, J., Higgins, D., and Zechner, K. (2010). Automated essay scoring: Writing assessment and instruction. In E. Baker, B. McGaw, and N.S. Petersen (Eds.), International encyclopedia of education (3rd ed., pp 75-80). Oxford, England: Elsevier.

Somaye Toranj and Dariush Nejad Ansari. (2012). Automated Versus Human Essay Scoring: A Comparative Study. Theory and Practice in Language Studies, Vol. 2, No. 4, pp. 719-725, April 2012

Uzun, K. (2018). Home-Grown Automated Essay Scoring in the Literature Classroom: A Solution for Managing the Crowd? Contemporary Educational Technology, 9(4). pp.423-436. Retrieved [15.3.19] from http://doi.org/10.30935/cet.471024

Valenti, S., Neri, F., and Cucchiarelli, A. (2003). An Overview of Current Research on Automated Essay Grading. Journal of Information Technology Education. 2. pp 319-330.

Wang, J. & Brown, M. S. (2007). Automated Essay Scoring Versus Human Scoring: A Comparative Study. Journal of Technology, Learning, and Assessment, 6(2). Retrieved [25.5.2016] from http://www.jtla.org

Warschauer, M., & Grimes, D. (2008). Automated writing assessment in the classroom. Pedagogies: An International Journal, 3, pp 22–36. Retrieved [3 February 2018] from http://education.uci.edu/uploads/7/2/7/6/72769947/awe-pedagogies.pdf.

Downloads

Published

2019-09-26

Issue

Vol 5, No 1 (2019)

Section

Articles

License

Authors who publish with this journal agree to the following terms:

 

  1. Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.

 

  1. Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.

 

  1. Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).