Can a language model predict for which questions another language model will answer successfully? We investigate the extent to which performance prediction is possible and dissect various factors that influence it. Our experimental setting fine-tunes DeBERTa models, which we call assessors , on the evaluation results of generative language models with up to 128 billion parameters, which we refer to as subject systems . Our analysis spans more than 100 tasks from BIG-bench. We find that the assessors can match and even exceed the subjects’ confidence in both refinement and calibration, anticipating failures at near perfect levels for some tasks. We also find that for performance prediction it can be beneficial to learn from the scores on multiple tasks or to learn from the scores of multiple subjects, but both depend on the task at hand. Lastly, we find that large and small subject systems are equally predictable, showing promise for the scalability of the predictability problem.

Wout Schellaert

Fernando Martínez-Plumed

José Hernández-Orallo

ACM Transactions on Intelligent Systems and Technology

ACM TRANSACTIONS ON INTELLIGENT SYSTEMS AND TECHNOLOGY 是一本学术期刊，以多学科的视角发表关于智能系统、适用算法和技术的最高质量论文。智能系统是使用人工智能 (AI) 技术提供重要服务（例如，作为更大系统的组件）以允许集成系统在现实世界中感知、推理、学习和智能行动的系统。ACM TIST 是每季出版（一年六期）。每期有 8-11 篇常规论文，大约有 20 个已发表的期刊页面或每篇论文 10,000 字。额外的参考文献、证明、图表或详细的实验结果可以作为单独的附录提交，而过长的论文将被自动拒绝。作者可以在其已发表论文的其他内容中包含仅在线附录，并鼓励与其他读者共享其代码和/或数据。

ACM Transactions on Intelligent Systems and Technology is an academic journal that publishes high-quality papers on intelligent systems, applicable algorithms, and technologies from a multidisciplinary perspective. Intelligent systems use artificial intelligence (AI) technologies to provide essential services, functioning as components of larger systems, enabling integrated systems to perceive, reason, learn, and act intelligently in the real world. ACM TIST is published quarterly, with six issues per year, each containing 8 to 11 regular papers, approximately 20 published journal pages, or up to 10,000 words per paper. Additional references, proofs, charts, or detailed experimental results may be submitted as separate appendices, and excessively long papers will be automatically rejected. Authors are allowed to include online-only appendices in their published papers and are encouraged to share their code and/or data with other readers.

《ACM Transactions on Intelligent Systems and Technology》是一本学术期刊，采用多学科的视角，发表关于智能系统、相关算法和技术的高质量论文。智能系统利用人工智能（AI）技术提供重要服务，作为更大系统的组成部分，使得集成系统能够在现实世界中进行感知、推理、学习和智能行动。该期刊每季度出版六期，每期包含8到11篇常规论文，约20页的已发表期刊内容，或每篇论文不超过10,000字。额外的参考文献、证明、图表或详细的实验结果可以作为附录提交，过长的论文将被自动拒绝。作者可以在已发表论文中包含仅在线附录，并鼓励与其他读者共享其代码和/或数据。

Category	Quartile
计算机科学	4
计算机科学, 计算机人工智能	4
计算机科学, 计算机信息系统	4

Analysing the Predictability of Language Model Performance

ivySCI AI Smartly Parses PDF, Answers Researchers' Questions, and Helps You Understand Papers in Seconds

Journal Info

Category	Quartile
COMPUTER SCIENCE, INFORMATION SYSTEMS	1