Keywords: Automated Essay Scoring, Large Language Models, Rubric-Based Assessment, Interpretability, Rubric Representations

Introduction

Automated essay scoring (AES) has recently started implementing neural network models, e.g., large language models (LLMs), in the grading process (Johnsi and Kumar, 2026). Agreement with human raters rose along the way, but the models remain opaque to inspection. Transparency and interpretability remain open priorities in AES investigations (Johnsi and Kumar, 2026). Control and verifiability are hard to explain in LLM-based AES systems, which is one of the biggest challenges when implementing LLM technologies in high-stakes language assessments. Test providers respond in different ways. Some rely on mixed models where human raters have the final score. Some develop feature-based machine learning models in which humans define what is measured. Others, such as the Duolingo English Test (DET), decompose the writing construct into subconstructs and constrain any LLM-derived feature score to a single subconstruct, so that the model cannot influence the holistic score except through that one slot. As stated in the report, scoring directly with deep learning “cedes both control and verifiability” (Madnani et al., 2026).¹

Although AES designs such as the DET minimize the black box impact on the score, the core question remains unsolved: what exactly an LLM scorer does on the inside. As LLMs become more and more ubiquitous in language assessment, we need to know whether a model’s internal representations contain the scoring rubrics at all, and whether each scoring rubric is controllable and verifiable separately. Current score agreement benchmark tests cannot answer this. Prior interpretability work on AES examines what a scorer has learned or which features it weights (Fiacco et al., 2023; Wang et al., 2026). We approach the topic with two AES principles in mind, control and verifiability, by adding causal intervention on the representations themselves, mirroring how these systems operate in real-world assessments.

In this study, we mechanistically audit LLM-based AES by probing and steering 25 open-weight base LLMs (0.13B to 10.31B parameters, 14 families) as they score BESTEP writing responses across three task types and six analytic scoring rubrics (see section Setup). We carry out the investigation guided by three questions. RQ1: Is the human holistic score decodable from a model’s internal activations, and where does that signal live? Which layers have the clearest scoring signals? RQ2: Do the six scoring rubrics occupy distinguishable directions in representation space? RQ3: Does steering a rubric’s direction move the model’s score specifically for that scoring rubric, or does any rubric-adjacent push move it about equally? In this paper, we choose base models as a baseline, so that fine-tuned variants (instruction-tuned, chat) can be compared against them in future studies. Our contributions are two-fold. First, we establish a white-box audit protocol for LLM-based automated essay scoring (AES) that disentangles a rubric’s decodability, separability, and causal control. Moreover, we show that analytic scoring rubrics designed for human raters do not function as intended in base LLMs, as models fail to maintain independent control across distinct evaluative dimensions. The code and the database behind every number are public for verification at https://github.com/lopentu/inside-the-scorer-paclic2026.

Related Work

AES and LLM-based scoring. As surveyed in Johnsi and Kumar (2026), AES first moved from feature-engineered machine learning methods to BERT-based deep learning, e.g., through domain-adaptive training, multi-scale essay representations, and hierarchical transfer learning (Ramesh and Sanampudi, 2022; Wang et al., 2022; Cao et al., 2020; Xue et al., 2021; Settles et al., 2020). Additionally, a second wave of AES investigations focuses on LLMs as scorers, resulting in mixed findings. Some studies report LLM ratings holding up under real psychometric scrutiny (generalizability-theory and many-facet Rasch analyses) across thousands of essays and multiple genres (Song et al., 2025; Wang et al., 2025a), and case studies where an LLM produces both a score and trait-level comments across several analytic scoring rubrics at once (Wang et al., 2025b; Cho et al., 2024). Others find the opposite: robustness failures under adversarial stress-testing and demographic bias in ratings (Kabra et al., 2022; Yamashita, 2025), systematic disagreement between LLM and human grading behavior (Mathew et al., 2026), and validity that shifts by language, genre, and proficiency level rather than holding uniformly (Mizumoto and Eguchi, 2023; Pack et al., 2024; Mansour et al., 2024; Gaggioli et al., 2025; Barbosa et al., 2025; Yavuz et al., 2025; Yancey et al., 2023). A related strand looks past scoring accuracy to classroom use, from ChatGPT’s promise and pitfalls as a writing tutor to its measurable effect on students’ own writing when used for feedback (Barrot, 2023; Ding et al., 2025).

Control and verifiability in operational scoring. High-stakes providers mostly manage the opacity of LLM-based AES systems through design. The Duolingo English Test decomposes a subconstruct into features and uses various LLMs and machine learning models across subconstructs and grader versions. Thus, an LLM-derived feature rating contributes to the score only through its own subconstruct, and the contribution of each subconstruct can be measured and explained (Madnani et al., 2026). The same report notes that it remains substantially harder to determine why an end-to-end system assigned a particular score than it is for feature-based systems.

Prior analyses of neural network AES scorers are observational. Wang et al. (2026) analyze how LLM scorers weight textual features, and Fiacco et al. (2023) align components of a transformer scorer’s activations with human-understandable feature groups. In our study, we approach this topic with high-stakes language assessment’s grading principles, i.e., control and verifiability, and intervene causally on the rubric representations.

A separate line augments scores with rubric-level explanations or rationales (Bueno et al., 2026; Chu et al., 2025; Jiao et al., 2025; Kucia et al., 2026). These works evaluate rubric outputs, whereas we evaluate internal rubric representations in LLM-based AES.

Mechanistic interpretability and faithfulness. Our probing and steering methodology draws on activation patching (Zhang and Nanda, 2023), causal concept-based explanations (Wu et al., 2023), and prior work examining whether model-generated explanations faithfully reflect internal computations (Atanasova et al., 2023; Jacovi and Goldberg, 2021; Chuang et al., 2026; Ferrando et al., 2023; Demircan et al., 2024). This concern applies to AES, because a plausible-sounding rationale does not guarantee that the model actually exercises rubric-specific control.

Our two evaluation checks are grounded in probing research. First, Elazar et al. (2021) demonstrate that a property’s being decodable from a representation does not mean the model actually uses it. Second, Hewitt and Liang (2019) show that probe accuracy is uninterpretable without a selectivity check against a memorization-based control task. In our work, RQ1 (decodability) and RQ3 (steering selectivity) directly adapt these two principles, applying them to scoring rubrics rather than standard linguistic properties.

Setup

Models. We evaluate 25 open-weight, base LLMs (non-instruction-tuned) ranging from 0.13B to 10.31B parameters with 16 to 42 layers across 14 model families, including Llama, Gemma, Falcon3, SmolLM2/3, Granite, Mistral, OLMo, TAIDE, Apertus, and others (see Appendix C, Table 6 for the full catalog). All 25 models are base models by design. Although operational LLM-based scorers, such as those in the Duolingo English Test, are typically fine-tuned models (Madnani et al., 2026), we intentionally use base models to prevent instruction or format tuning from confounding the rubric representations and score output. This design establishes a clean baseline, enabling direct comparisons with fine-tuned variants in future work.

The model parameter range also reflects a key deployment constraint highlighted in the language-testing literature. Local institutions evaluating AI-assisted assessment require practical model sizes for on-premises hosting so that private response data remain local, rather than being sent to massive cloud-hosted systems (Chuang and Yan, 2025).

BESTEP and its Scoring Rubrics. This study uses the official BESTEP writing rubrics, which human raters apply when scoring the test’s writing responses. BESTEP, which stands for “the BEST test of English Proficiency,” was developed and is operated by the Language Training & Testing Center (LTTC) in support of Taiwan’s national “Program on Bilingual Education for Students in College (BEST).” The test assesses academic English proficiency of college students across four skills: reading, listening, writing and speaking. We adopt the writing scoring rubrics from the official test guidelines and have verified with LTTC stakeholders that human raters apply these same public rubrics. The only difference is that human raters complete training workshops with authentic test responses before live scoring.

The writing section comprises three task parts, designed to evaluate three distinct subconstructs. Part 1 (Answering Questions) is a short factual-response task. Part 2 (Expressing Opinions) asks test-takers to write emails expressing their opinions on campus events to stakeholders. Part 3 (Integrated Writing) is the longest and most demanding task, requiring test-takers to write a full essay.

Writing responses are scored with six analytic scoring rubrics, though not every rubric applies to every part (see Table 1). Grammar and vocabulary apply across all three parts, spelling and punctuation are scored exclusively in the short-answer task (Part 1), and argument relevance is evaluated solely in the integrated essay (Part 3).

Dataset. All 25 models are probed and scored on the same fixed set of 75 human-rated BESTEP writing responses (the gold-75), spanning two test forms and three task parts (see Table 2).

Activation extraction. For every model, we extract residual-stream activations at every layer for each of the gold-75 responses, taken at the final token of the grading prompt, where the model would write its score. These activations are the shared input to the probing (RQ1), contrastive-direction (RQ2), and steering (RQ3) procedures. These procedures are described in the next three sections.

Code and Data Availability. Code, the analysis scripts, and the result dataset behind every reported number are available at https://github.com/lopentu/inside-the-scorer-paclic2026. They are released for noncommercial research use under the PolyForm Noncommercial License 1.0.0. However, the original and degraded essay pairs (Appendix A) containing real exam responses are not yet public due to copyright constraints. They are available for research purposes upon request.

Experiments

Score-Information Representation

RQ1. Is the human-graded holistic score decodable from a model’s internal activations, and where does that signal live? Which layers contain the highest-quality human scoring signals?

Method. For each combination of [model, layer, part], hereafter referred to as a cell, we train a Ridge regression probe on final-token residual-stream activations to predict the human-rated holistic band score. Each cell probe is evaluated using leave-one-out cross-validation (LOOCV; Figure 1): in each fold, one response is held out, and the cell probe is trained on the remaining N-1 responses (features standardized and the ridge penalty selected within each training round). All held-out predictions are pooled to compute a single R2 per cell (R2 is negative when predictions are worse than guessing the mean). We report the peak-R2 layer, the peak R2 itself, and the peak-to-final-layer drop, treating the pooled-form scope (both test forms combined) as our primary result. To verify that peak R2 scores exceed chance-level baselines from the probing procedure, each cell undergoes a label-shuffling permutation test (1,000 score permutations). Because the peak layer is selected as the maximum across 16 to 42 candidate layers, we report the peak layer’s p-value both raw and Bonferroni-corrected over that model’s total layers.

Results. Human-rated score information is decodable in most models, but not equally for every part, and not in a simple “easy parts early, hard parts late” order. Against the permutation null, the peak layer beats chance in all 25 models for Part 3 and in 22 of 25 for Part 2 after the layer correction, but in only 9 of 25 for Part 1 (24 of 25 uncorrected). The one consistent timing pattern: Part 1 (the shortest, short-answer task) peaks latest or second-latest among the three parts in 23 of 25 models (Figure 2). Across the 75 model-part cells, the signal usually fades toward the final layer. The last layer keeps about two-thirds of the peak R2 (median drop 34%), and in 14 out of the 75 cells, it is worse than guessing the average score (R2 below zero).

Peak timing and decodability strength diverge across models. Part 1 produces the lowest peak R2 in 24 of 25 models and is never the highest among the three parts. Mean peak R2 across models scales with task length: 0.432 for Part 1, 0.633 for Part 2, and 0.773 for Part 3 (Table 3, Figure 3). Part 1 is also the least stable across models (peak-R2 variance of 0.0338, versus 0.0095 for Part 2 and 0.0038 for Part 3). This lower and more volatile human-score decodability stems from Part 1 having the fewest gold responses (n=21), the shortest response lengths (about 47 words, vs. about 117 for Part 2 and about 214 for Part 3), and a coarser rating scale, yielding less textual evidence and fewer data points for probes estimation.

For the other two longer tasks, models carry information about essay quality that a label-shuffle null cannot explain. For BESTEP’s shortest task type the signal peaks later, is weaker and less stable, and is above the corrected null in only 9 of 25 models. We cannot explain this pattern mechanistically here, but RQ2 returns to it. The Bonferroni correction is conservative because adjacent layers are correlated, and 1,000 shuffles bound how small a p-value can get (Limitations).

Rubric Representation

RQ2. Do the six scoring rubrics occupy distinguishable directions in representation space?

Method. RQ2 uses the same 75 gold-standard responses plus rubric-targeted clean-versus-degraded mutations, since holistic scores blend scoring rubrics and cannot support rubric-geometry claims on their own. Mutations are generated by prompting llama3.3:70b (via Ollama) with one of 12 rubric-quoting specs, one per [part, scoring rubric] combination the rubric scores (Table 1), each spec quoting the official BESTEP band descriptors for that scoring rubric and instructing the model to degrade the essay to the low band on that dimension only, holding every other dimension, and meaning, constant (length preserved to within 10%). Two trained BESTEP graders verified each mutated essay against its intended target dimension before use (Appendix A gives the procedure and worked examples). For each [model, layer, part, scoring rubric] cell we estimate a contrastive direction as the normalized difference between clean and mutated-essay activation centroids, and its split-half reliability by recomputing the direction from random halves of the essay pairs. We compare same-rubric to different-rubric cosine similarity within the same essay part (a same-part, cross-rubric comparison avoids confounding “different rubric” with “different essay”), retaining only reliability-gated rows (split-half reliability at least 0.6).

Results.

Separation is uneven across the assessment. Averaged across 1,470 reliability-gated layer-part cells, the mean same-versus-different geometry gap is 0.334 (74.5% of cells positive), but this is not stable across task parts: Part 1’s four scoring rubrics are, on average, negative (mean gap -0.220, 6.2% of rows positive), no more separated from each other than a single scoring rubric is from itself across parts, while Parts 2 and 3 separate cleanly (89.7% and 99.2% of cells positive): reliable, rubric-specific directions are broadly available in Parts 2 and 3 but sparse in Part 1.

Pair by pair, the range is wide. Pooling all reliability-gated rows by literal rubric pair, using disattenuated cosine similarity (a correction for each direction’s own measurement noise, so a value of 1 means statistically indistinguishable from the same rubric and 0 means unrelated), the 12 comparable pairs range from anti-correlated to statistically identical (three of the 15 possible pairs never co-occur in the same part and are not comparable at all; Table 4, alongside RQ3’s steering results below). argument_relevance paired with grammar is the most separated pair (-0.102, anti-correlated beyond noise), and argument_relevance, a Part-3-only scoring rubric, carries most of the separation signal in the whole rubric. At the opposite end, grammar paired with spelling_punctuation sits at the 1.0 ceiling: the two are statistically indistinguishable from the same rubric, and spelling_punctuation paired with vocabulary is nearly as entangled (0.984). Both of the most-entangled pairs are Part-1-only comparisons, consistent with Part 1’s negative geometry gap above.

So separation is broad in the two longer, more integrated task types, concentrated in one scoring rubric (argument relevance) beyond that, and absent in the shortest task type.

Causal Specificity

RQ3. Does steering a rubric’s direction move the model’s score specifically for that scoring rubric, or does any rubric-adjacent push move it about equally?

Method. RQ3 reuses RQ2’s rubric directions and tests them causally rather than only representationally. A direction is scaled to unit length and added to the residual stream at one layer, at every token position of the grading prompt, with magnitude alpha times that layer’s mean residual-stream norm (alpha = 0.05, 0.1, 0.25, 0.5), so strength is comparable across layers and models. The outcome is the change in the model’s own expected band score: the probability-weighted mean over the digit tokens at the answer position of the same grading prompt (no learned readout), minus the unsteered baseline. For degraded essays with a known target scoring rubric we inject four kinds of direction: the matching rubric direction (self), a mismatched rubric direction (other), an orthogonal control (a rubric direction with the axis shared by all scoring rubrics in the part removed, which tests whether overlap between directions hides specificity), and a generic holistic direction (the direction the RQ1 probe uses to predict the human band score, which tests whether any score-raising push works). Self and other effects are paired at the finest valid unit, [model, layer, part, form, degradation, alpha], giving 74,304 paired cells across all 25 models, and we test whether the matching direction beats a mismatched one by a practically meaningful margin.

Results.

Steering works for five of six scoring rubrics. Grammar, vocabulary, cohesion, spelling/punctuation, and task relevance each show a clearly positive self effect (0.168 to 0.359 score points). Only argument_relevance’s own direction produces no detectable effect: it neither raises nor lowers the score.

Steering is not selective. Pooled across all paired cells, the matching direction beats a mismatched one by only 0.0074 score points on average (95% CI [0.0036, 0.0117]; 1.74% of the absolute self effect, 95% CI [0.85%, 2.74%]), and this holds under every robustness check we ran: the gap holds at the cell level (0.0085, 95% CI [0.0051, 0.0119]), never exceeds 0.0083 under leave-one-model/family-out, and a cluster bootstrap that resamples whole model families rather than individual models, accounting for shared training lineage such as the three Llama-3.x checkpoints, gives an almost identical estimate (0.0073, 95% CI [0.0035, 0.0108]).

The pooled number combines two different patterns. Broken out pair by pair (Figure 4), among the four scoring rubrics that both work and are reliably measured (grammar, vocabulary, cohesion, spelling/punctuation), the matching direction shows no detectable advantage over a mismatched one in any of their 10 testable head-to-head comparisons (the other two ordered pairs, cohesion with spelling/punctuation, are never degraded in the same part and cannot be tested). In one of the ten, vocabulary detectably loses to spelling/punctuation, not because vocabulary is weak, but because spelling/punctuation happens to be the single strongest general-purpose lever of the six. argument_relevance loses every comparison it has data for, for the different reason that its own direction never moves the score. Re-run separately within each task part (36 within-part comparisons among comparably-strong scoring rubrics), the same pattern holds in every part: not one instance of a scoring rubric beating a comparably strong rival specifically because it matched the essay’s degradation.

Two controls calibrate these effects (Table 5). The orthogonal control does not behave like a flat baseline: it lowers the score slightly and more so at larger alpha, a small movement with the opposite sign of any working self effect, so the model is not indifferent to an injected direction, only to which scoring rubric it targets. The generic holistic direction raises the score less than self or other at every alpha tested, which argues against a pure “generic tampering” account of the specificity gap above; its alpha slope falls inside the self/other range, so it behaves like a weaker member of the same family of levers. At the strongest alpha (0.5) the average self effect is 0.361 score points on the 1-6 band scale, about 6% of the band range, a moderate intervention.

Tying back to RQ2, a pair’s representational separation (Table 4, above) does predict how large a causal difference that pair shows under steering (r=0.717 across all comparable pairs), but it does not predict which direction wins.

RQ2’s directions are reliability-gated, but RQ3 injects the raw, non-disattenuated direction regardless of noise, and different scoring rubrics’s directions are already substantially correlated (Table 4). Noise and this overlap could be masking a larger true specificity, and three checks address that. The first restricts to progressively more reliable directions, which raises the specificity ratio from 1.8% (all cells) to 5.4% (RQ2 geometry available) to 3.1% at reliability at least 0.7 to 8.3% at the top reliability quartile (self 0.670 vs. other 0.605, gap 95% CI [0.057, 0.072]): less reliable directions do understate true specificity, but even the best-measured quarter shows self only modestly outperforming other. A second check removes the axis shared by all scoring rubrics in a part from each direction (the orthogonal-control condition above) and repeats self vs. other on these overlap-corrected directions, using data already collected. If overlap were concealing real independent control, this should reveal it, and it does not. Both self_orth and other_orth average slightly negative (-0.093, -0.100), and their gap (0.0064, 95% CI [0.0003, 0.0125], ratio 4.82%) matches the reliability-gated result: once the shared component is removed, the rubric-specific residual is not a strong positive lever for either scoring rubric, so most of the raw effect was the shared component, not overlap hiding independent control.

A third check isolates what a direction shares only with its specific rival. For each pair of scoring rubrics we strip from each direction its projection onto the other, so that the matching and mismatched directions lose exactly their mutual overlap (pairs too collinear to leave a residual are skipped), and inject the two residuals next to a norm-matched random direction as a noise floor. Across all 25 models (884 pair cells), both residuals raise the score above the noise floor, by 0.194 (matching) and 0.181 (mismatched) points, so rubric-specific residual structure exists and moves the score. The matching residual still does not win: the gap is 0.013 points (95% CI [-0.015, 0.042]), about 4.5% of the average absolute matching effect, and the interval includes zero. Pairs below and above a cosine of 0.6 give the same picture (gaps of 0.012 and 0.014). Removing the overlap pair by pair leaves the levers interchangeable.

Discussion

Taken together, the three research questions give one account of what a rubric-based LLM scorer does inside. The model carries decodable information about essay quality (RQ1), strongly for the two longer tasks and weakly for the shortest. It represents at least some scoring rubrics as distinguishable directions rather than one undifferentiated quality signal (RQ2). But distinguishable representation does not translate into independently controllable behavior (RQ3). Steering a rubric direction reliably moves the score, yet pushing the matching direction is, for the scoring rubrics that are both effective and reliably measured, essentially interchangeable with pushing a different one. The model appears to exercise one shared control with several equally effective handles, not six independent, rubric-specific ones.

This matters for how neural AES is justified. Operational providers that need control and verifiability get it by construction: in a decomposed design each subconstruct has its own slot, and a neural feature can influence the score only through its slot (Madnani et al., 2026). An end-to-end LLM scorer offers no such slot, and our audit asks whether it contains something equivalent on its own. In these base models it does not: scoring rubric information is present and partly separable, but it is not exposed as separate controls. High agreement with human raters, or the mere decodability of scoring rubrics, is therefore not evidence that a model uses those scoring rubrics as separately controllable scoring levers. Controllability is a diagnostic that score agreement cannot supply.

These results also speak to the rubrics themselves. Analytic scoring rubrics are written for human raters, who are trained to treat each criterion as a separate judgment. In these base models the criteria are partly represented, with distinct directions for the two longer tasks, but they do not work as separate levers: none of the 10 testable comparisons among the four reliably measured scoring rubrics shows a detectable advantage for the matching direction. Rubrics designed for human raters therefore do not function in base LLMs as they are intended to function for human raters, although, as the next paragraph explains, human ratings may show the same pattern.

What the diagnostic means is open, because shared control has two readings. One is that the model collapses scoring rubrics that the test keeps apart. The other is that the model faithfully reproduces the structure of human analytic ratings, in which dimensions are known to move together: halo, the carry-over of a rater’s overall impression into separate trait scores, is a documented rater effect in the scoring of written composition (Engelhard Jr, 1994; Myford and Wolfe, 2004), and its relation to rating accuracy is itself contested (Fisicaro, 1988). Our gold set carries only holistic band scores, so we cannot separate the two readings here. Either way, a provider cannot assume that rubric labels from a neural scorer are separately controllable without checking.

The evidence for separation and for nonselective steering also comes from partly different scoring rubrics, and the two should be read together. argument_relevance carries most of the separation signal in RQ2 but its steering effect is undetectable, whereas the four scoring rubrics that steer and are reliably measured have moderately to highly similar directions (Table 4). Nonselective steering among overlapping directions is what overlap alone would predict, which is why the pairwise check in RQ3 matters: with each pair’s overlap removed, both residuals still move the score and still do so about equally.

A related result concerns what the score responds to. The generic holistic direction, learned from the human band scores, raises the model’s score less than the rubric directions do at every strength we tested (Table 5), and the strongest lever of all is spelling/punctuation, a surface feature. This suggests, but does not show, that the model’s own score leans more on surface language features than on the quality signal that RQ1 can read out of its activations. Information that is decodable is not necessarily information that the model uses.

A further caution is that these base models agree only modestly with human raters (Limitations), so shared control could simply reflect weak scorers. It does not appear to. Among the seven models whose own scores correlate with human scores at r of 0.6 or higher, the matching direction still beats a mismatched one by only 0.011 score points (95% CI [0.007, 0.015]; 1.9% of the absolute self effect), about the same as in the full set.

The pattern is not uniform across the assessment design. Part 1, BESTEP’s shortest task, shows the weakest and least stable score signal (RQ1) and no meaningful rubric separation (RQ2), while the two longer, more integrated parts support a stronger signal and real, if not independently controllable, rubric structure. Validity claims should be checked at that grain, not assumed to generalize across task format.

Limitations

All 25 models are base (non-instruction-tuned). We chose them as a baseline, but it means we cannot say how instruction-tuned, chat, or task-fine-tuned variants, which resemble deployed scorers more closely, would behave. The gold set is 75 essays from one assessment (BESTEP), which bounds essay-level diversity. Model coverage tops out at 10.31B parameters, so results may not hold for substantially larger models. argument_relevance’s own steering direction never reaches significance; whether that reflects a representational gap or a scoring rubric that needs more than surface language features to steer is open. More models, other writing assessments, and non-English rubric-based assessments could test whether the findings travel.

This paper tests how scoring rubrics are represented and controlled inside the models, not whether the models score like human raters. The human band score is a fixed reference only: it is the target of the RQ1 probe and the source of the generic holistic direction in RQ3. The models’ own scores agree only modestly with it (pooled Pearson r across the 75 essays: median 0.41, 7 of 25 models at r ≥ 0.6, none at 0.8), so the findings describe imperfect zero-shot base scorers, not validated scoring systems.

Two limits concern the degraded essays that define every rubric direction. They were generated by one LLM, and two trained BESTEP graders then checked every one of the 104: each, separately, saw an original and its degraded version, was told the essay had been degraded on one scoring rubric, and had to name which, and both confirmed the intended scoring rubric every time (Appendix A). That shows the target scoring rubric is recognizable, not that nothing else changed. The grammar/spelling_punctuation similarity ceiling in Table 4, and the near-ceiling spelling/vocabulary pair, can arise because errors in one dimension leave traces in another even in “pure” stimuli (Appendix A gives an example), so we do not read those pairs as evidence about the model alone. Cleaner single-dimension manipulations are future work.

Because the gold set carries no rubric-level human ratings, we cannot test whether shared control reflects the models or the halo structure of human analytic ratings (Discussion); a disattenuated human inter-scoring rubric correlation matrix would let a future study separate the two.

Three statistical limits apply. First, the RQ1 correction for layer selection is Bonferroni, which is conservative because adjacent layers are correlated, and 1,000 label shuffles set a floor on each p-value; a max-over-layers permutation test would be exact. Second, the paired-cell analysis is not a simple random sample: layers, alpha values, and models sharing a training lineage are correlated. The primary inference averages within model before pooling, and the family-clustered bootstrap gives an essentially unchanged estimate, but a full hierarchical model across family, essay, scoring rubric, and layer would be a stronger test. Third, the pairwise check removes overlap between two directions at a time, and pairs too collinear to leave a residual are skipped, so the most entangled pairs are the ones it cannot speak to. How best to isolate one direction’s rubric-specific component from a correlated one is itself an active methodological question (Elazar et al., 2021).

Conclusion

LLM-based essay scoring is attractive but opaque, and high-stakes testing asks for control and verifiability. We examined the scorer from the inside, asking whether an LLM scorer’s scoring rubrics are represented and independently controllable. Across 25 open-weight base models, holistic scores are decodable (clearly for the two longer task types, weakly for the shortest) and scoring rubrics are partly separable, but neither property implies independent causal control: steering a rubric direction moves the score as a general lever for the great majority of the scoring rubrics this design can measure reliably, and removing each pair’s overlap does not change that. This may reflect the models, the halo structure of the human ratings they imitate, or both; rubric-level human ratings would be needed to tell them apart. Independent controllability is a diagnostic that any rubric-based LLM scorer should be examined with before its rubric labels are treated as separately meaningful, and on this evidence, base LLMs do not yet offer it. Instruction-tuned and fine-tuned scorers are the natural next comparison.

Degradation Examples

RQ2’s contrastive directions are estimated from clean-versus-degraded essay pairs (§RQ2 Method). Each degradation is produced by prompting llama3.3:70b with a spec that quotes the official BESTEP low-band descriptor for one scoring rubric and instructs the model to hold every other dimension constant. The first example below degrades one source essay on two adjacent scoring rubrics, and the second shows a cohesion degradation, to make the intended separation (and its limits) concrete.

Part 1, grammar (target: verb form, agreement, word order only).

• Original: “Three tours will be given. What should I prepare for the training? The place in my school that I visit most often is the classroom, which is the place I learn knowledges about my major.”
• Grammar-degraded: “Three tour will be give. What I should prepare for the training? The place in my school that I visiting most often are the classroom, which the place I learn knowledges about my major.”
• Spelling/punctuation-degraded (same source sentence): “Three tuors will be given. What shoud I prepair for the trainning. the place in my school that i visit most often is the clasroom which is the place i lern knowleges about my major.”

The grammar mutant changes verb forms, subject-verb agreement, and word order while leaving spelling intact; the spelling/punctuation mutant misspells content words and drops punctuation while leaving grammar intact, but it also drops sentence-initial capitalization (“the place”, “i visit”), a surface feature a grammar-sensitive signal could partly pick up on. This is a concrete illustration of the limitation discussed below: even a mutation that targets one dimension by instruction can leave a residual trace in an adjacent one.

Part 2, cohesion (target: connectives and sentence-joining only).

• Original: “I wrote this email in regards to the new food court on campus. First of all, I want to say it is a good change for the school because the food court can help save the time, the students won’t need to leave the campus to get food. […] In this way, the variety of the options and the opening hours might be appealing to more students.”
• Cohesion-degraded: “I wrote this email about the new food court on campus. It is a good change for the school. The food court can help save the time. The students won’t need to leave the campus to get food. […] The variety of the options might be appealing to more students. The opening hours might be appealing to more students.”

Here the degradation removes connectives (“First of all”, “In this way”) and splits joined sentences into flat, simple ones, with no change to individual sentences’ grammar or vocabulary, a cleaner separation than the grammar/spelling pair above, consistent with cohesion’s lower same-rubric entanglement in Table 4.

Verification. Every degraded essay was checked by two trained BESTEP essay graders before use. Working separately, each grader was shown an original essay together with its degraded version, was told that the degraded version had been downgraded on one scoring rubric, and had to identify which. All 104 degraded essays (20 for Part 1, 36 for Part 2, and 48 for Part 3, that is 5, 9, or 12 for each part and scoring rubric) were checked this way. Both graders confirmed the intended scoring rubric for every one, a consensus of 100%, and the check was run as a random double sample and as a cross-comparison between the graders. This establishes that the intended scoring rubric is recognizable in each degraded essay. It does not establish that no other dimension changed, and the grammar/spelling example above shows that adjacent dimensions can carry a trace. The degraded essay pairs contain exam responses and are not yet public; they are available from the authors for research purposes on request.

Per-Model Depth Profiles

Figure 5 shows the RQ1 depth-profile curve (Figure 2’s pooled version, disaggregated) for each of the 25 models individually, for readers who want to check whether the pooled pattern in Figure 2 holds for a specific model rather than only in aggregate.

Model Catalog

Table 6 lists the 25 open-weight base models probed and steered in this study, sorted by parameter count.

References