How far can sub-3B open language models go in zero-shot essay scoring on an 8 GB consumer GPU?

Abstract

Zero-shot essay scoring with large language models is usually demonstrated with proprietary remote models, yet the classrooms that most need it are often barred from using them. This study asks what survives when the scorer must be a sub-3B open model run locally. Four models from two families, Qwen2.5 at 0.5B, 1.5B and 3B and SmolLM2-1.7B, were evaluated on all eight Automated Student Assessment Prize prompts on one 8 GB consumer graphics processing unit, with bootstrap intervals and Holm-corrected tests. Under batch min-max aggregation, rubric-decomposed prompting improved macro quadratic weighted kappa for every model, though Qwen2.5-3B fell significantly on one prompt, and, of the two strategies, only it made all four models track the per-prompt human ceiling (Spearman ρ +0.60 to +0.85, significant for one model only). Aggregation interacted with grader calibration: one model collapsed to 0.204 under naive averaging and recovered to 0.388 under Multi-Trait Specialization's min-max normalization. Signed error fell with essay length in eleven of twelve configurations, opposite to large judge models' verbosity bias. The best configuration reached 0.388, half of the human ceiling (0.769), and stayed below a length-only baseline (0.523, itself batch min-max normalized), so sub-3B local scoring suits only formative, human-supervised feedback.

Conflict of Interest

The authors declare that they have no competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.