Educational Measurement: Issues and Practice

Abstract The remarkable strides in artificial intelligence (AI), exemplified by ChatGPT, have unveiled a wealth of opportunities and challenges in assessment. Applying cutting‐edge large language models (LLMs) and generative AI to assessment holds great promise in boosting efficiency, mitigating bias, and facilitating customized evaluations. Conversely, these innovations raise significant concerns regarding validity, reliability, transparency, fairness, equity, and test security, necessitating careful thinking when applying them in assessments. In this article, we discuss the impacts and implications of LLMs and generative AI on critical dimensions of assessment with example use cases and call for a community effort to equip assessment professionals with the needed AI literacy to harness the potential effectively.

Jiangang Hao

Alina A. von Davier

Victoria Yaneva

Susan Lottridge

Matthias von Davier

Deborah J. Harris

Educational Measurement Issues and Practice

Transforming Assessment: The Impacts and Implications of Large Language Models and Generative AI

转型评估：大型语言模型与生成式人工智能的影响与启示

Abstract In scientific literacy, knowledge integration (KI) is a scaffolding‐based theory to assist students' scientific inquiry learning. To drive students to be self‐directed, many courses have been developed based on KI framework. However, few efforts have been made to evaluate the outcome of students' learning under KI instruction. Moreover, finer‐grained information has been pursued to better understand students' learning and how it progresses over time. In this article, a normative procedure of building and choosing cognitive diagnosis models (CDMs) and attribute hierarchies was formulated under KI theory. We examined the utility of CDMs for evaluating students' knowledge status in KI learning. The results of the data analysis confirmed an intuitive assumption of the hierarchical structure of KI components. Furthermore, analysis of pre‐ and posttests using a higher‐order, hidden Markov model tracked students' skill acquisition while integrating knowledge. Results showed that students make significant progress after using the web‐based inquiry science environment (WISE) platform.

Xin Xu

Shixiu Ren

Danhui Zhang

Tao Xin

Knowledge Integration in Science Learning: Tracking Students' Knowledge Development and Skill Acquisition with Cognitive Diagnosis Models

科学学习中的知识整合：利用认知诊断模型追踪学生的知识发展与技能获取

I am very pleased to be one of the former NCME Presidents to contribute to this special issue on The Past, Present, and Future of Educational Measurement, as serving in that role was a career highlight for me. In my article, I review events in my life in measurement and how there may be lessons to be learned by others from them. This review gives the article a different flavor than most of the other contributions, but I hope readers will see relevance in my experiences to some of the decisions and situations they now face or will encounter in the future I have been employed in standardized testing for 60 years, the majority of which were spent at large testing enterprises. Much of the time I oversaw test and test program development including admissions tests, state and school assessments, and tests used in counseling, employment, and clinical settings. I estimate that many tens of millions of students have taken tests developed by my staff. I have been very active in the testing industry and profession over my career, serving on boards and committees, editing this journal, and filling the role of President for the NCME and the Association for Assessment in Counseling on the professional side and the Association of Test Publishers, the core test industry group. In my reflections, I am trying to look back on events that stand out for me and their importance at various stages of my career. Perhaps readers can draw some lessons of value to them. I tell my grandchildren that they will learn a lot from their own experiences, but that they should try also to gain from observing and thinking about the lives of others. If you keep your eyes and mind open, there is so much to learn. I finished college at night while working full time so did not have many incidental contacts with college services such as counseling. I had one required appointment with a counselor. He asked about my plans, and I replied that I wanted to continue with some aspects of psychology as I had had exceptionally good experiences in my coursework. When I said I liked every part of psychology, the counselor told me that educational psychology would be a good direction for me. I left my one and only college-level counseling session motivated to explore graduate study in educational psychology and that search led me to Teacher's College, Columbia University. This path led me into testing as a career. I am comfortable with the outcome of my decision, but I believe that more attention to career counseling for part-time students would be highly desirable. After completing college, I was sitting at my desk in the accounting department where I was working when I received a call from a Professor at Teacher's College where I had applied for graduate admissions in Educational Psychology and Measurement. The professor was Robert L. Thorndike and he wanted to know if I would be interested in receiving an NDEA Fellowship and additional university benefits as a Teacher's College PhD student? I was thrilled and accepted immediately, a life-changing decision that opened so many doors for me. The faculty included leaders in the field of measurement and my fellow students were also all people who wanted to make a difference in the world. We all saw testing as extremely valuable and wanted to play a part in developing and using tests. I kept track of some of my fellow students and they became productive contributors to universities and to the testing profession and industry. Fellowships and other support help open doors for many students, particularly when they are the first in their family seeking college and graduate degrees. Robert L. Thorndike used an example in a graduate school class presentation of an unintended consequence of the strategies measurement experts developed and used for selecting aircrew in WWII. The highest math scorers in a group of very able men were trained as Navigators rather than Pilots. After the war Pilots were more likely to stay in the service and rise in rank so the senior Air Force Leaders in future years were not the strongest math folk available. Not an intended outcome or one even considered at the time of test use. Thorndike urged us to always try to foresee likely consequences of our plans for testing and to follow up on how tests were actually used. What impacts were occurring for the diverse groups that were involved? I think about this issue whenever I am evaluating the way a test is being used. We all need to do this in all phases of testing. There is a tendency to accept some features of testing as inevitable once we have produced a strategy for dealing with what we recognize to be a problem. Realizing that unequal familiarity with what a test will cover creates an unfair situation, we strive to provide practice examinations to all test takers and to create comprehensive preparatory materials accessible to all. We should not settle, though, for the status quo. We should be seeking to make our exams and how they are used as helpful as they can be to achieve productive social and educational goals. That requires us to listen to those who criticize our work and to find ways to involve those impacted by testing in the development and evolution of tests as well as testing policies and procedures. The former President of ETS, Henry Chauncey, was speaking to a group of us who were participants in the 1964 ETS Summer Program for Graduate Students in Measurement. He said that people felt that a measure of motivation would be a good addition to the SAT when selecting students for college. In somewhat of a tongue-in-cheek way, he offered the idea of an electrified grid that students who were barefooted would need to walk across to get into a particular college. This notion was not meant to be practical and would fail many important DEI standards, but it did raise interesting issues about the measures we did use. What did they leave out? Who do they favor? I see many exchanges in my Email flow proposing innovative approaches to the use of tests in educational transitions. One common element in the discussions is how testing can be combined with instructional activities. If we could obtain the information to help students move up educational levels from how they handled prior learning, we could broaden the group about whom we had relevant knowledge and not rely as much on single- occasion tests. This would help decrease the continuing impact of Testwiseness on performance, especially on multiple-choice tests. When I was touring the ETS Campus as part of the Summer Program, we visited a section that responded to requests for student's SAT scores. I was puzzled about the fact that students were making these requests. I had not realized that this was possible as SAT scores were not sent to test takers at the time I took them as a high school student. Checking in the literature I found that up until the late 1950s, The College Board and ETS judged that students should not receive their scores as they might place too much weight on the meaning of the scores. That concern had merit, but the policy changed and now along with all other students I could see my own score. We are still addressing how to present scores on tests to test takers. The move over my career in testing is to provide increasingly greater amounts of information. One of the challenges is to try to communicate the strengths and limitations of standardized test scores. As with any sample, test scores tell only part of the story about the test taker. How can we best make that part meaningful and productive? Are we doing all we can not only to encourage proper use of test scores, but to guard against overuse of test scores including applications of scores for which no supporting evidence has been provided? As we evaluate the validity of uses of tests used in admissions at any level, we need to keep trying to communicate in a way that test takers and test score users can understand clearly. It is not enough to develop high quality tests; we want to facilitate high quality uses of scores. That must include recognizing misinterpretations and correcting them as best we can The distinguished ETS psychometrician, Fred Lord, once and only once came to my ETS office for advice! What was this Criterion Referenced Testing idea about which he was reading? He recognized how uninformed about measurement principles most of the CRT writing was. I was able to point out a core of meaning that made sense and to identify ways that CRT had the potential to be a meaningful approach. It was a brief meeting, but a career learning experience. The top measurement scholar was staying on top of new developments and turning to colleagues, including ones quite junior to him with questions. As we become more expert, we need to continue to listen to those at earlier stages of their career. The newer voices are not as likely to channel their thoughts down the same paths even as tools and challenges evolve. I once read that a senior business executive had instructed his support staff to allow access to anyone with "completely crazy ideas." I cannot say that I followed this rule myself, but I have tried to keep in mind the real risks of only listening to ideas solidly in your comfort zone. When you are making changes in testing programs, I urge you not to say that the change is going to be "easy." This triggers responses such as we give when someone says, "This isn't going to hurt." In many instances test score users are proceeding as they were taught years ago. They need to learn a new set of interpretations and explanations. The relationship between the number of questions answered correctly and test score may now be harder to understand and to communicate about. You need to explain why you are making changes and to show how you will help. The better you are at providing a productive context, the more support you will earn. Just do not say it will be easy. In 2003, at the annual Association of Test Publisher's (ATP) Conference, I met with David Foster, someone whose judgment and working style I had very much respected in roles we had at ATP. I was thinking about a career move and had five possible directions to get his help thinking about. The third on my list was test security. I had been helping design and deliver workshops for an ETS teaching/consulting program—The ETS Global Institute. Regardless of the country the workshop participants were from, cheating on tests and related security issues came up as a major concern. There seemed to me to be a place for test security services as a line of business. I never got past that item on my list. Dave said that he and a few Salt Lake City area colleagues had reached the same conclusion and were going to "meet at his cabin" later that year. Dave invited me to join them. There was no outside investor or fund to cover the airfare for me or other non-local folk coming to the meeting, but we could all stay at Dave's cabin. The "cabin" proved to have 6,000 square feet of space in a lovely home in a forested area. A dozen of us contributed to work already underway to define the company and the initial services of Caveon Test Security. I sometimes tell new measurement professionals that our professional conferences are exceptionally effective ways to build a network. There may be a conversation like the one I had with Dave Foster in their future. If you are helping professional staff develop, encourage them to attend conferences related to their work. Provide funding where you can do so. Make sure that your staff know that you are expecting them to stay abreast of developments in the measurement profession and industry. That is part of their job, not a nice-to-do digression. At conferences, participants should devote part of their time to sessions related directly to their work. When a session of this type ends, participants should go to the front of the meeting room when there are presentations meaningful to them and thank the presenters. When really impressed by a speaker, the attendee should try to establish a relationship with that speaker. One of the consequences of the growth of electronic and virtual meetings is that you need to take special steps to connect with interesting people who you have heard. There is no "front of the room" to go up to. Take steps such as sending an email, even if this does not "come naturally" to you. At in-person conferences, view sessions, receptions, refreshment breaks, and exhibit halls, as a chance to ask questions and always listen carefully to answers. It is a wonderful way to learn and grow as a measurement professional. Looking back at the different jobs and roles that I have played in testing; I am struck by a continuing need to help people make decisions. Whether it is in an educational, employment, or certification setting, someone is deciding about the qualifications or appropriateness of groups of people. Should this person be hired or admitted? Which of these people has shown that they merit certification of a set of specialized skills and knowledge? I have learned that the more I know about the circumstances for the decisions the more helpful I can be. What are the consequences for the test taker, the test program sponsor, and society of an inaccurate decision? Some misjudgments based on tests do not cause much difficulty, as when a test taker is asked to proceed more slowly through the next set of lessons. This outcome may slow a student's progress but might not have other worrisome consequences. In sharp contrast, in high-stakes testing settings, careers may be sidetracked, or harm caused to the public due to an incorrect decision about competence based on standardized tests. For our work in educational and related testing, we need to adhere to a rule from carpentry "measure twice, cut once." We need to devote time to learning from potential test users and those who will take our tests what the impacts will be. Then we must continue to take the measure of our own work going forward. That is our responsibility as measurement professionals. In the foregoing paragraphs, I have shared some of my career experiences and the lessons I draw from them. If you have reactions that you would like to share about anything I wrote, I would love to hear from you. We need to keep learning from each other for testing to continue to evolve as the world changes about us.

John Fremer

Measurement Reflections

测量反思

Abstract Over our field's 100‐year‐plus history, standardization has been a central assumption in test theory and practice. The concept's justification turns on leveling the playing field by presenting all examinees with putatively equivalent experiences. Until relatively recently, our field has accepted that justification almost without question. In this article, I present a case for standardization's antithesis, personalization. Interestingly, personalized assessment has important precedents within the measurement community. As intriguing are some of the divergent ways in which personalization might be realized in practice. Those ways, however, suggest a host of serious issues. Despite those issues, both moral obligation and survival imperative counsel persistence in trying to personalize assessment.

Randy E. Bennett

Personalizing Assessment: Dream or Nightmare?

个性化评估：梦想还是噩梦？

Abstract This article describes an amazing development of methods and models supporting educational measurement together with a much slower evolution of theory about how and what students learn and how educational measurement best supports that learning. Told from the perspective of someone who has lived through many of these changes, the article provides background on these developments and insights into challenges and opportunities for future development.

Lauress L. Wise

Daisy W. Rutstein

Educational Measurement: Models, Methods, and Theory

教育测量：模型、方法与理论

Abstract The Computer‐based Case Simulations (CCS) component of the United States Medical Licensing Examination (USMLE) Step 3 was developed to assess the decision‐making and patient‐management skills of physicians. Process data can provide deep insights into examinees’ behavioral processes related to completing the CCS assessment task. In this paper, we utilized process data to evaluate the impact of shortening allotted time limit by rescoring the CCS cases based on process data extracted at various timestamps that represented different percentages of the original allotted case time. It was found that examinees’ performance as well as the correlation between original and newly generated scores both tended to decrease as the timestamp condition became stricter. The impact of shortening allotted time limit was found marginally associated with case difficulties, but strongly dependent on the case time intensity under the original time setting.

Chunyan Liu

Monica M. Cuddy

Qiwei He

Wenli Ouyang

Cara Artman

Using Process Data to Evaluate the Impact of Shortening Allotted Case Time in a Simulation‐Based Assessment

使用过程数据评估缩短模拟评估中分配案例时间的影

Abstract Vertical scales are frequently developed using common item nonequivalent group linking. In this design, one can use upper‐grade, lower‐grade, or mixed‐grade common items to estimate the linking constants that underlie the absolute measurement of growth. Using the Rasch model and a dataset from Curriculum Associates’ i‐Ready Diagnostic in math in grades 3–7, we demonstrate how grade‐to‐grade mean differences in mathematics proficiency appear much larger when upper‐grade linking items are used instead of lower‐grade items, with linkings based on a mixture of items falling in between. We then consider salient properties of the three calibrated scales including invariance of the different sets of common items to student grade and item difficulty reversals. These exploratory analyses suggest that upper‐grade common items in vertical scaling are more subject to threats to score comparability across grades, even though these items also tend to imply the most growth.

Sanford R. Student

Derek C. Briggs

Laurie Davis

Growth across Grades and Common Item Grade Alignment in Vertical Scaling Using the Rasch Model

使用Rasch模型进行垂直标度中的年级间成长与共同项目年级对齐

Abstract This contribution to the Special Issue of EM:IP on the topic of The Past, Present and Future of Educational Measurement concentrates on the present and the future and hence focuses on the goal of improving education. The results of meta‐analyses were examined, and it was noted that the largest effect sizes were associated with actual use of formative assessments in classroom settings—hence classroom assessment (in contrast with large‐scale assessment ). The paper describes micro assessment, which focuses on in‐classroom forms of measurement, and then expands this assessment approach to focus on frames beyond that in terms of summative end‐of‐semester tests (macro). This is followed by a description of how these approaches can be combined using a construct map as the basis for developing and using assessments to span across these two levels in terms of the BEAR Assessment System (BAS). Throughout, this is exemplified using an elementary school program designed to teach students about geometry. Finally, a conclusion summarizes the discussion, and also looks to the future where a meso level of use involves end‐of‐unit tests.

Mark Wilson

What Makes Measurement Important for Education?

测量对教育为何重要？

Abbreviation	Educational Measurement
Journal Impact	2.64
Quartiles(Global)	PSYCHOLOGY, EDUCATIONAL(Q2)

ISSN	0731-1745, 1745-3992
h-index	64

Publication Information	Publisher: Wiley-Blackwell Publishing Ltd，Publishing cycle: ，Journal Type: journal，Open Access Journals: No
Basic data	Year of publication: 1982，Proportion of original research papers: 100.00%，Self Citation Rate:7.40%， Gold OA Rate: 15.79%

Educational Measurement: Issues and Practice

Journal Citation Format

A journal article with 1 author

A journal article with 2 authors

A journal article with 3 authors

A journal article with 5 or more authors

Books Citation Format

Thesis Citation Format

Web sites Citation Format

Patent Citation Format

Staying up late manually editing references? ivySCI automatically matches journals and helps you generate references with a single click.

Share Submission Experience

Transforming Assessment: The Impacts and Implications of Large Language Models and Generative AI

Knowledge Integration in Science Learning: Tracking Students' Knowledge Development and Skill Acquisition with Cognitive Diagnosis Models

Measurement Reflections

Personalizing Assessment: Dream or Nightmare?

Educational Measurement: Models, Methods, and Theory

Personalizing Assessment: Dream or Nightmare?

Measurement Reflections

Using Process Data to Evaluate the Impact of Shortening Allotted Case Time in a Simulation‐Based Assessment

Growth across Grades and Common Item Grade Alignment in Vertical Scaling Using the Rasch Model

What Makes Measurement Important for Education?