DEFINING SEVERE GRADERS THROUGH MANY FACETED RASCH MEASUREMENT
DOI:
https://doi.org/10.46827/ejfl.v0i0.1107Paraules clau:
testing, subjectivity, reliability, severe graders, lenient graders, Rasch analysis, rater effectResum
Scoring language learners’ writing exams is a difficult task for graders since many task-relevant or irrelevant variables such as the user-friendliness of the rubric, difficulty of the task, students’ handwriting or grader characteristics (being too lenient or harsh) are involved in the process. To be able to gain valid and reliable scores, studying the variables that affect scoring procedures and seeking ways to control and minimize them are crucial concerns for institutions in order to assure their learners that their assigned scores are genuine and given in the least subjective way that could be possible. That is why analysing grader attitudes while scoring and identifying the stringent and lenient graders in the rater-pool is important not only to be able to set the best matches of graders where multiple scorings or cross-marking sessions are applied but for making those raters be aware of their scoring habits. In this exploratory study, 6 writing graders who had more than 10-year-expertise in grading writing voluntarily scored 20 student essays including two separate tasks. MFRM (Many Faceted Rasch Measurement) was used to explore graders’ marking behaviours and discover how those behaviours affect test scores of language learners. Finally, results of the study showed that graders, while they all used the same rubric and had enough expertise in grading, have significant score differences and a significant level of stringency in scoring essays.
Article visualizations:
Referències
Anastasi A, 1988. Psychological Testing, 6th Edition. New York: Macmillan.
Boone, W.J., 2016. Rasch Analysis for instrument development: Why, When and How?. CBE Life Science Education. 15-1/7, 2016.
Congdon, P.J., McQueen, J. 2000. The stability of rater severity in large-scale assessment programs. Journal of Educational Measurement, 37(2), 163–178.
Coniam, D., Falvey, P. 2007. High-stakes testing and assessment. In J. Cummins & C. Davison (Eds.), International handbook of English language teaching (pp. 457– 471). New York: Springer.
Di Nisio, R. 2010. Measure school learning through Rasch Analysis: the interpretation of results. Procedia - Social and Behavioral Sciences, Volume 9, 2010, Pages 373-377. https://doi.org/10.1016/j.sbspro.2010.12.167
Eckes, T. 2005. Examining rater effects in TestDaF writing and speaking performance assessments: A multi-faceted Rasch analysis. Language Ass. Quarterly, 2(3), 197– 221.
Kondo-Brown, K. 2002. An analysis of rater bias with FACETS in measuring Japanese L2 writing performance. Lang. Test, 19, 1-29. http://dx.doi.org/10.1191/02655322o2it218oa
Lane, S., Stone, C.A. 2006. Performance Assessment. In R. L. Brennan (Ed.): Educational Measurement (pp 387-431). Wesport, CT: ACE/Praeger.
Linacre, J.M. 1989. Many-facet Rasch measurement. Chicago: MESA Press.
Linacre, J.M. 2002. Optimizing Rating Scale Category Effectiveness. Journal of Applied Measurement, 3, 85-106.
Linacre, J.M. 2009. FACETS (Computer program, version 3.66.1). Chicago: MESA Press.
Linacre, J.M., Wright, B.D. 2002. Construction of Measures from Many-Facet Data. Journal of Applied Measurement, 3, 484-509.
Lumley, T. 2005. Assessing second language writing: The rater’s perspective. Frankfurt am Main: Peter Lang.
Lumley, T., McNamara, T.F. 1995. Rater characteristics and rater bias: Implications for training. Language Testing, 12(54), 54–71.
Lunz, M.E., Wright, B.D., Linacre, J.M. 1990. Measuring the impact of judge severity on examination scores. Applied Measurement in Education, 3(4), 331–345.
MacMillan, P.D. 2000. Classical, Generalizability, and multifaceted Rasch detection of interrater variability in large, sparse data sets. Journal of Experimental Education, 68(2), 167–190.
McNamara, T.F. 1996. Measuring second language performance. London: Longman.
Messick, S. 1995. Standard of validity and the validity of standard in performance assessment. Educational Measurement: Issues and Practice, 14(4), 5–8.
Myford, C.M., Wolfe, E.W. 2004. Detecting and Measuring Rater Effects Using Many- Facet Rasch Measurement: Part I. In E. V. Smith y R.M. Smith (Eds.). Introduction to Rasch Measurement (pp. 460-515). Maple Grove, MN: JAM Press.
Park, T. 2004. An Investigation of an ESL Placement Test of Writing Using Many- facet Rasch Measurement, Papers in TESOL & Applied Linguistics, 4, 1-21.
Prieto, G. 2011. Evaluación de la ejecución mediante el modelo Many-Facet Rasch Measurement. Psicothema, 23, 233-238. [Performance assessment using Many- Facet Rasch Measurement].
Razak, N., Khairani, A.Z., Thien, L.M. 2012. Examining Quality of Mathemtics Test Items Using Rasch Model: Preminarily Analysis. Procedia - Social and Behavioral Sciences, Volume 69, 24 December 2012, Pages 2205-2214. https://doi.org/10.1016/j.sbspro.2012.12.187
Tyndall, B., Kenyon, D. M. 1996. Validation of a new holistic rating scale using Rasch multi- faceted analysis. En A. Cumming y R. Berwick (Eds.), Validation in language testing (pp. 39-57). Clevedon: Multilingual Matters.
Wolfe, E.W. 2004. Identifying rater effects using latent trait models. Psychology Science, 46(1), 35–51.
Wright, B.D., Masters, G.N. 1982. Rating scale analysis. Chicago: MESA Press.
Wright, B.D., Linacre, J.M. 1994. Reasonable mean-square fit values. Rasch Measurement Transactions, 8, 370-371.
Wu, S.M., Tan, S. 2016. Managing rater effects through the use of FACETS analysis: the case of a university placement test, Higher Education Research & Development, 35:2, 380-394, DOI: 10.1080/07294360.2015.1087381
Descàrregues
Publicat
Número
Secció
Llicència
Authors who publish with this journal agree to the following terms:
- Authors retain copyright and grant the journal right of first publication with the work simultaneously licensed under a Creative Commons Attribution License that allows others to share the work with an acknowledgement of the work's authorship and initial publication in this journal.
- Authors are able to enter into separate, additional contractual arrangements for the non-exclusive distribution of the journal's published version of the work (e.g., post it to an institutional repository or publish it in a book), with an acknowledgement of its initial publication in this journal.
- Authors are permitted and encouraged to post their work online (e.g., in institutional repositories or on their website) prior to and during the submission process, as it can lead to productive exchanges, as well as earlier and greater citation of published work (See The Effect of Open Access).
