Keywords: Automated Essay Scoring, Large Language Models, Rubric-Based Assessment, Interpretability, Rubric Representations

Introduction

Automated essay scoring (AES) has recently started implementing neural network models, e.g., large language models (LLMs), in the grading process (Johnsi and Kumar, 2026). Agreement with human raters rose along the way, but the models remain opaque to inspection. Transparency and interpretability remain open priorities in AES investigations (Johnsi and Kumar, 2026). Control and verifiability are hard to explain in LLM-based AES systems, which is one of the biggest challenges when implementing LLM technologies in high-stakes language assessments. Test providers respond in different ways. Some rely on mixed models where human raters have the final score. Some develop feature-based machine learning models in which humans define what is measured. Others, such as the Duolingo English Test (DET), decompose the writing construct into subconstructs and constrain any LLM-derived feature score to a single subconstruct, so that the model cannot influence the holistic score except through that one slot. As stated in the report, scoring directly with deep learning “cedes both control and verifiability” (Madnani et al., 2026).¹

Although AES designs such as the DET minimize the black box impact on the score, the core question remains unsolved: what exactly an LLM scorer does on the inside. As LLMs become more and more ubiquitous in language assessment, we need to know whether a model’s internal representations contain the scoring rubrics at all, and whether each scoring rubric is controllable and verifiable separately. Current score agreement benchmark tests cannot answer this. Prior interpretability work on AES examines what a scorer has learned or which features it weights (Fiacco et al., 2023; Wang et al., 2026). We approach the topic with two AES principles in mind, control and verifiability, by adding causal intervention on the representations themselves, mirroring how these systems operate in real-world assessments.

In this study, we mechanistically audit LLM-based AES by probing and steering 25 open-weight base LLMs (0.13B to 10.31B parameters, 14 families) as they score BESTEP writing responses across three task types and six analytic scoring rubrics (see section Setup). We carry out the investigation guided by three questions. RQ1: Is the human holistic score decodable from a model’s internal activations, and where does that signal live? Which layers have the clearest scoring signals? RQ2: Do the six scoring rubrics occupy distinguishable directions in representation space? RQ3: Does steering a rubric’s direction move the model’s score specifically for that scoring rubric, or does any rubric-adjacent push move it about equally? In this paper, we choose base models as a baseline, so that fine-tuned variants (instruction-tuned, chat) can be compared against them in future studies. Our contributions are two-fold. First, we establish a white-box audit protocol for LLM-based automated essay scoring (AES) that disentangles a rubric’s decodability, separability, and causal control. Moreover, we show that analytic scoring rubrics designed for human raters do not behave as independent controls in base LLMs: for the rubrics that steer the score and are reliably measured, steering one rubric’s direction moves the score about as much as steering a different one, even among the models whose scores agree best with human raters. The code and the database behind every number are public for verification at https://github.com/lopentu/inside-the-scorer-paclic2026.

Related Work

AES and LLM-based scoring. As surveyed in Johnsi and Kumar (2026), AES first moved from feature-engineered machine learning methods to BERT-based deep learning, e.g., through domain-adaptive training, multi-scale essay representations, and hierarchical transfer learning (Ramesh and Sanampudi, 2022; Wang et al., 2022; Cao et al., 2020; Xue et al., 2021; Settles et al., 2020). Additionally, a second wave of AES investigations focuses on LLMs as scorers, resulting in mixed findings. Some studies report LLM ratings holding up under real psychometric scrutiny (generalizability-theory and many-facet Rasch analyses) across thousands of essays and multiple genres (Song et al., 2025; Wang et al., 2025a), and case studies where an LLM produces both a score and trait-level comments across several analytic scoring rubrics at once (Wang et al., 2025b; Cho et al., 2024). Others find the opposite: robustness failures under adversarial stress-testing and demographic bias in ratings (Kabra et al., 2022; Yamashita, 2025), systematic disagreement between LLM and human grading behavior (Mathew et al., 2026), and validity that shifts by language, genre, and proficiency level rather than holding uniformly (Mizumoto and Eguchi, 2023; Pack et al., 2024; Mansour et al., 2024; Gaggioli et al., 2025; Barbosa et al., 2025; Yavuz et al., 2025; Yancey et al., 2023). A related strand looks past scoring accuracy to classroom use, from ChatGPT’s promise and pitfalls as a writing tutor to its measurable effect on students’ own writing when used for feedback (Barrot, 2023; Ding et al., 2025).

Control and verifiability in operational scoring. High-stakes providers mostly manage the opacity of LLM-based AES systems through design. The Duolingo English Test decomposes a subconstruct into features and uses various LLMs and machine learning models across subconstructs and grader versions. Thus, an LLM-derived feature rating contributes to the score only through its own subconstruct, and the contribution of each subconstruct can be measured and explained (Madnani et al., 2026). The same report notes that it remains substantially harder to determine why an end-to-end system assigned a particular score than it is for feature-based systems.

Prior analyses of neural network AES scorers are observational. Wang et al. (2026) analyze how LLM scorers weight textual features, and Fiacco et al. (2023) align components of a transformer scorer’s activations with human-understandable feature groups. In our study, we approach this topic with high-stakes language assessment’s grading principles, i.e., control and verifiability, and intervene causally on the rubric representations.

A separate line augments scores with rubric-level explanations or rationales (Bueno et al., 2026; Chu et al., 2025; Jiao et al., 2025; Kucia et al., 2026). These works evaluate rubric outputs, whereas we evaluate internal rubric representations in LLM-based AES.

Mechanistic interpretability and faithfulness. Our probing and steering methodology draws on activation patching (Zhang and Nanda, 2023), causal concept-based explanations (Wu et al., 2023), and prior work examining whether model-generated explanations faithfully reflect internal computations (Atanasova et al., 2023; Jacovi and Goldberg, 2021; Chuang et al., 2026; Ferrando et al., 2023; Demircan et al., 2024). This concern applies to AES, because a plausible-sounding rationale does not guarantee that the model actually exercises rubric-specific control.

Our two evaluation checks are grounded in probing research. First, Elazar et al. (2021) demonstrate that a property’s being decodable from a representation does not mean the model actually uses it. Second, Hewitt and Liang (2019) show that probe accuracy is uninterpretable without a selectivity check against a memorization-based control task. In our work, RQ1 (decodability) and RQ3 (steering selectivity) directly adapt these two principles, applying them to scoring rubrics rather than standard linguistic properties.

Setup

Models. We evaluate 25 open-weight, base LLMs (non-instruction-tuned) ranging from 0.13B to 10.31B parameters with 16 to 42 layers across 14 model families, including Llama, Gemma, Falcon3, SmolLM2/3, Granite, Mistral, OLMo, TAIDE, Apertus, and others (see Appendix C, Table 6 for the full catalog). All 25 models are base models by design. Although operational LLM-based scorers, such as those in the Duolingo English Test, are typically fine-tuned models (Madnani et al., 2026), we intentionally use base models to prevent instruction or format tuning from confounding the rubric representations and score output. This design establishes a clean baseline, enabling direct comparisons with fine-tuned variants in future work.

The model parameter range also reflects a key deployment constraint highlighted in the language-testing literature. Local institutions evaluating AI-assisted assessment require practical model sizes for on-premises hosting so that private response data remain local, rather than being sent to massive cloud-hosted systems (Chuang and Yan, 2025).

BESTEP and its Scoring Rubrics. This study uses the official BESTEP writing rubrics, which human raters apply when scoring the test’s writing responses. BESTEP, which stands for “the BEST test of English Proficiency,” was developed and is operated by the Language Training & Testing Center (LTTC) in support of Taiwan’s national “Program on Bilingual Education for Students in College (BEST).” The test assesses the academic English proficiency of college students across four skills: reading, listening, writing and speaking. We adopt the writing scoring rubrics from the official test guidelines and have verified with LTTC stakeholders that human raters apply these same public rubrics. The only difference is that human raters complete training workshops with authentic test responses before live scoring.

The writing section comprises three task parts, designed to evaluate three distinct subconstructs. Part 1 (Answering Questions) is a short factual-response task. Part 2 (Expressing Opinions) asks test-takers to write emails expressing their opinions on campus events to stakeholders. Part 3 (Integrated Writing) is the longest and most demanding task, requiring test-takers to write a full essay.

Writing responses are scored with six analytic scoring rubrics, though not every rubric applies to every part (see Table 1). Grammar and vocabulary apply across all three parts, spelling and punctuation are scored exclusively in the short-answer task (Part 1), and argument relevance is evaluated solely in the integrated essay (Part 3).

Dataset. All 25 models are probed and scored on the same fixed set of 75 human-rated BESTEP writing responses (the gold-75), spanning two test forms and three task parts (see Table 2).

Activation extraction. For every model, we extract residual-stream activations at every layer for each of the gold-75 responses, taken at the final token of the grading prompt, where the model would write its score. These activations are the shared input to the probing (RQ1), contrastive-direction (RQ2), and steering (RQ3) procedures. These procedures are described in the next three sections.

Code and Data Availability. Code, the analysis scripts, and the result dataset behind every reported number are available at https://github.com/lopentu/inside-the-scorer-paclic2026. They are released for noncommercial research use under the PolyForm Noncommercial License 1.0.0. However, the original and degraded essay pairs (Appendix A) containing real exam responses are not yet public due to copyright constraints. They are available for research purposes upon request.

Experiments

Score-Information Representation

RQ1. Is the human-graded holistic score decodable from a model’s internal activations, and where does that signal live? Which layers contain the highest-quality human scoring signals?

Method. For each combination of [model, layer, part], hereafter referred to as a cell, we train a Ridge regression probe on final-token residual-stream activations to predict the human-rated holistic band score. Each cell probe is evaluated using leave-one-out cross-validation (LOOCV; Figure 1): in each fold, one response is held out, and the cell probe is trained on the remaining N-1 responses (features centered and the ridge penalty selected within each training round). All held-out predictions are pooled to compute a single R² per cell (R² is negative when predictions are worse than guessing the mean). We report the peak-R² layer, the peak R² itself, and the peak-to-final-layer drop, treating the pooled-form scope (both test forms combined) as our primary result. To verify that peak R² scores exceed chance-level baselines from the probing procedure, each cell undergoes a label-shuffling permutation test (1,000 score permutations). Because the peak layer is selected as the maximum across 16 to 42 candidate layers, we report the peak layer’s p-value both raw and Bonferroni-corrected over that model’s total layers.

Results. Human-rated score information is decodable in most models, but not equally for every part, and not in a simple “easy parts early, hard parts late” order. Against the permutation null, the peak layer beats chance in all 25 models for Part 3 and in 22 of 25 for Part 2 after the layer correction, but in only 9 of 25 for Part 1 (24 of 25 uncorrected). We call a peak significant at p < 0.05. Raw p-values sit at the permutation floor (p = 0.001) for 25 of 25 models in Part 3, 21 in Part 2 and 9 in Part 1, so the Bonferroni-corrected p-values (medians 0.032, 0.034 and 0.096) are limited by the 1,000 shuffles, not by the data. The one consistent timing pattern: Part 1 (the shortest, short-answer task) peaks latest or second-latest among the three parts in 23 of 25 models (Figure 2). This holds in 8 of the 9 models whose Part 1 peak passes the corrected test, but where it does not pass, the peak location is poorly estimated. Across the 75 model-part cells, the signal usually fades toward the final layer. The last layer keeps about two-thirds of the peak R² (median drop 34%), and in 14 out of the 75 cells it is worse than guessing the average score (R² below zero).

Peak timing and decodability strength diverge across models. Part 1 produces the lowest peak R² in 24 of 25 models and is never the highest among the three parts. Mean peak R² across models scales with task length: 0.432 for Part 1, 0.633 for Part 2, and 0.773 for Part 3 (Table 3, Figure 3). Part 1 is also the least stable across models (peak-R² variance of 0.0338, versus 0.0095 for Part 2 and 0.0038 for Part 3). This lower and more volatile human-score decodability stems from Part 1 having the fewest gold responses (n=21), the shortest response lengths (about 47 words, vs. about 117 for Part 2 and about 214 for Part 3), and a coarser rating scale, yielding less textual evidence and fewer data points for probe estimation.

For the other two longer tasks, models carry information about essay quality that a label-shuffle null cannot explain. For BESTEP’s shortest task type the signal peaks later, is weaker and less stable, and is above the corrected null in only 9 of 25 models. We cannot explain this pattern mechanistically here, but RQ2 returns to it. The Bonferroni correction is conservative because adjacent layers are correlated, and 1,000 shuffles bound how small a p-value can get (Limitations).

Rubric Representation

RQ2. Do the six scoring rubrics occupy distinguishable directions in representation space? If so, how separated are they from each other?

Method. To evaluate rubric separability, RQ2 uses rubric-targeted clean-versus-degraded essay mutations of the gold-standard responses (n=26). This is because holistic scores blend individual rubrics and cannot isolate rubric-geometry dynamics on their own. The mutations were generated by prompting Llama 3.3 (70B, via Ollama) with one of 12 rubric-quoting specifications corresponding to each valid [part, scoring rubric] combination (Table 1).

The source essays are the 26 gold responses rated at the top band of Part 1’s 1 to 5 scale (n=5) and at a high band on the 1 to 6 scales of Part 2 (n=9) and Part 3 (n=12). Each is degraded once for every rubric scored in its part. Each specification quotes the official BESTEP band descriptors for a target rubric and instructs the model to degrade the response to a low band along that single dimension while holding all other dimensions, overall meaning, and response length (within 10%) constant. Each mutated response was verified by two trained BESTEP human raters independently against its target dimension prior to analysis (see Appendix A for full procedures and worked examples).

For each [model, layer, part, rubric] cell, we estimate a contrastive direction: the mean activation of the clean essays minus the mean activation of their degraded versions, taken at the same position as in RQ1 and scaled to unit length. Each direction rests on 5, 9, or 12 essay pairs (Parts 1 to 3). To measure how stable a direction is, we split its pairs at random into two halves, estimate a direction from each half, and take the cosine similarity between the two; the average over 20 random splits is the split-half reliability. Two directions are compared by their cosine similarity at the same layer, and a comparison is kept only if both directions have a split-half reliability of at least 0.6. For each direction, the geometry gap is its mean similarity to the same rubric in the other parts (same rubric) minus its mean similarity to the other rubrics in the same part (different rubric). Different rubrics are compared only within a part, so that a difference between two rubrics is not confused with a difference between tasks or essays. A positive gap means that a rubric’s direction is more similar to itself than to its neighbors, that is, that it has a direction of its own.

Results. Rubric directions are separate in the two longer tasks (Parts 2 and 3) but not in Part 1. A positive geometry gap means that a rubric’s direction is more similar to itself than to the other rubrics in the same part. Across 1,470 reliability-gated layer-part cells, the mean gap is 0.334, and 74.5% of cells are positive. This average covers only rubrics scored in more than one part; spelling/punctuation and argument relevance are scored in a single part, have no same-rubric comparison across parts, and are not included. The gap differs sharply by part. In Part 2 (mean gap 0.372, 89.7% of cells positive) and Part 3 (0.609, 99.2%), the rubrics have directions of their own. In Part 1 the mean gap is negative (−0.220, 6.2% of cells positive), especially for grammar and vocabulary. This suggests that different rubrics are more alike than a single rubric is across parts.

Pair by pair, Table 4 gives the similarity between each two rubrics’ directions: 1 means the same direction, 0 means unrelated, and a negative value means partly opposite directions. Because each direction rests on only 5 to 12 essay pairs (made from high-band gold essays) and carries noise, we correct the similarities for it (disattenuated cosine similarity). Among the 15 possible rubric pairs, 12 can be compared. The other three (argument relevance with spelling/punctuation, argument relevance with task relevance, and cohesion with spelling/punctuation) are never simultaneously used to score in the same part. The 12 comparable pairs range from −0.102 to 1.0. The most separate pair is argument_relevance with grammar (−0.102, anti-correlated beyond measurement noise), and argument_relevance, a Part-3-only rubric, appears in three of the five least similar pairs. The most alike are grammar with spelling_punctuation (1.0, statistically indistinguishable from the same rubric) and spelling_punctuation with vocabulary (0.984). Both are Part-1-only comparisons, in line with Part 1’s negative gap.

In short, rubric separation is clear in Parts 2 and 3, most visibly pair by pair for argument_relevance, and absent in Part 1.

Causal Specificity

RQ3. Does steering a rubric’s direction move the model’s score specifically for that scoring rubric, or does any rubric-adjacent push move it about equally?

Method. RQ3 tests the rubric directions from RQ2 causally, not only representationally. We add a direction to the residual stream at one layer, at every token position of the grading prompt. The direction has unit length and is scaled by alpha times the mean L2 norm of that layer’s residual stream, measured at the answer position of the clean and degraded essay prompts (alpha = 0.05, 0.1, 0.25, 0.5), so the strength is comparable across layers and models. The outcome is the change in the model’s own expected band score. At the answer position of the grading prompt, where the model would write its score, we take the model’s logits for the digit tokens 0 to the maximum band (5 for Part 1, 6 for Parts 2 and 3), convert them to probabilities with a softmax over those digits only, and compute the probability-weighted mean of the digit values. This takes one forward pass and uses neither a separate scoring prompt nor a learned readout. The outcome is this expected band with the direction injected minus the same quantity without it (the unsteered baseline). For degraded essays with a known target rubric, we inject four kinds of direction:

• the matching rubric direction (self);
• a mismatched rubric direction (other);
• an orthogonal control, a rubric direction with the axis shared by all rubrics in the part removed, which tests whether overlap between directions hides specificity;
• a generic holistic direction, the direction the RQ1 probe uses to predict the human band score, which tests whether any score-raising push works.

Self and other effects are paired at the finest valid unit, [model, layer, part, form, degradation, alpha], which gives 74,304 paired cells across the 25 models. We test whether the matching direction beats a mismatched one by a practically meaningful margin, which we define as more than 0.05 score points or more than 10% of the self effect.

Results. Steering moves the score for five of the six rubrics, but not selectively: the direction that matches the essay’s degraded rubric works barely better than a mismatched one.

Steering works. Grammar, vocabulary, cohesion, spelling/punctuation, and task relevance each have a clearly positive effect when their own direction is injected (self effect, 0.168 to 0.359 score points). Only argument_relevance has no detectable self effect (−0.083, 95% CI [−0.195, 0.024]): its direction neither raises nor lowers the score.

Steering is not selective. Pooled across all paired cells, the matching direction beats a mismatched one by only 0.0074 score points (95% CI [0.0036, 0.0117]), which is 1.74% of the absolute self effect (95% CI [0.85%, 2.74%]). Both confidence intervals lie well inside our margin, so the advantage is statistically detectable but practically negligible. This holds under every robustness check we ran: at the cell level (0.0085, 95% CI [0.0051, 0.0119]); when any one model or any one model family is left out (never above 0.0083); and in a cluster bootstrap that resamples whole model families to account for shared training lineage, such as the three Llama-3.x checkpoints (0.0073, 95% CI [0.0035, 0.0108]).

Pair by pair. The pooled number combines two patterns (Figure 4). First, among the four rubrics that both steer the score and are reliably measured (grammar, vocabulary, cohesion, and spelling/punctuation), the matching direction shows no detectable advantage over a mismatched one in any of the 10 testable head-to-head comparisons. The other two ordered pairs, cohesion with spelling/punctuation, cannot be tested because the two are never degraded in the same part. In one of the ten, vocabulary detectably loses to spelling/punctuation, not because vocabulary is weak, but because spelling/punctuation is the strongest general-purpose lever of the six. Second, argument_relevance loses all three of its comparisons, because its own direction never moves the score.

Within each task part. We also compared every pair of rubrics scored in the same part (18 pairs in all). For each pair, we added the two directions’ gaps, one on each rubric’s own essays. This cancels a rubric’s general strength as a lever and leaves only the advantage of matching. The matching advantage ranges from −0.015 to 0.030 score points. It is detectable in 9 of the 18 pairs and reversed in one, but it is small: in the detectable pairs it is 2% to 14% of the pair’s own effects (median 6%), and the largest values involve the weak Part 1 task-relevance direction.

Controls. Two controls calibrate these effects (Table 5), where Delta is the mean change in the expected band score in score points and Slope is how much that change grows per unit of alpha, both averaged over models. The orthogonal control is not a flat baseline: it lowers the score slightly (Delta −0.097) and more so at larger alpha (Slope −0.288). So the model is not indifferent to an injected direction, only to which rubric it targets, and the score-raising effect lies mostly in the axis that the rubric directions share (Check 2 below). The generic holistic direction raises the score (Delta 0.130) but less than the self or other direction at every alpha tested, which argues against a pure “generic tampering” account of the specificity gap above. Its alpha slope (0.550) is close to those of self (0.589) and other (0.562), so it behaves like a weaker member of the same family of levers. At the strongest alpha (0.5), the average self effect is 0.361 score points, about 6% of the 6-point band scale, a moderate intervention.

Link to RQ2. A pair’s separation in RQ2 (Table 4) predicts how large the causal difference for that pair is under steering (r=0.717 across the 24 ordered pairs, that is, the 12 pairs in both directions), but it does not predict which direction wins. The evidence for separation and for nonselective steering also comes partly from different rubrics: argument_relevance is among the most separate rubrics in RQ2 but has no steering effect, whereas the four rubrics that steer and are reliably measured have moderately to highly similar directions (Table 4), so the two results should be read together (Discussion).

Could noise or overlap hide a larger specificity? RQ2’s directions are reliability-gated, but RQ3 injects the raw, non-disattenuated directions regardless of noise, and different rubrics’ directions are already substantially correlated (Table 4). Noise and this overlap could mask a larger true specificity, so we ran three checks.

Check 1: more reliable directions. The specificity ratio is 1.8% across all cells (a cell-level figure, which is why it differs slightly from the model-level 1.74% above), 5.4% for cells that meet the RQ2 reliability gate (at least 0.6), 3.1% at a reliability of at least 0.7, and 8.3% in the top reliability quartile (about 0.85 or higher; self 0.670 vs. other 0.605, gap 95% CI [0.057, 0.072]). The rise is not monotone, but the most reliable directions show the highest ratio, so less reliable directions may understate true specificity. Even the best-measured quarter shows self only modestly outperforming other.

Check 2: remove the shared axis. We remove the axis shared by all rubrics in a part from each direction (the orthogonal-control condition) and repeat the self-versus-other comparison on these overlap-corrected directions, using data already collected. If overlap were hiding real independent control, this would reveal it, and it does not. Both the self and other effects are slightly negative (−0.093 and −0.100), and their gap (0.0064, 95% CI [0.0003, 0.0125]) is about 5% of the mean absolute size of the corrected self effect (0.133 across models, larger than the −0.093 average because the sign differs across models), close to the 5.4% in Check 1. Once the shared component is removed, the rubric-specific residual is not a strong positive lever for either rubric, so most of the raw effect was the shared component, not overlap hiding independent control.

Check 3: overlap with one specific rival. For each pair of rubrics, we strip from each direction its projection onto the other, so that the matching and mismatched directions lose exactly their mutual overlap (pairs too collinear to leave a residual are skipped). We inject the two residuals next to a norm-matched random direction as a noise floor. Across all 25 models (884 pair cells), both residuals raise the score above the noise floor, by 0.194 (matching) and 0.181 (mismatched) points, so rubric-specific residual structure exists and moves the score. This does not conflict with Check 2. The random-direction noise floor is itself negative (−0.101), so the corrected directions in Check 2 (−0.093) sit at the noise level, while the residuals here (+0.093 and +0.080) sit well above it, probably because removing only one rival’s overlap leaves part of the lever that all rubrics share. The matching residual still does not win: the gap is 0.013 points (95% CI [−0.015, 0.042]), about 4.5% of the average absolute matching effect (about 0.30, since the 0.194 above is measured from a noise floor that is itself negative), and the interval includes zero. Pairs with a cosine below and at or above 0.6 give the same picture (gaps of 0.012 and 0.014). Removing the overlap pair by pair leaves the levers interchangeable.

Models that agree best with human raters. These base models agree only modestly with the human band (Limitations), so shared control could reflect weak scorers. Among the seven models whose own scores correlate with the human band at r ≥ 0.6 (an exploratory cut-off set after seeing the agreement values), the matching direction beats a mismatched one by 0.011 score points (95% CI [0.007, 0.015]), which is 1.9% of the absolute self effect, against 1.74% in the full set. Models with better agreement therefore show the same shared control.

Discussion

Taken together, the three research questions give one account of what a rubric-based LLM scorer does inside. The human score can be read out of the models’ activations (RQ1). This is clear for the two longer tasks, where the peak layer beats a label-shuffle null after correction in 25 of 25 models (Part 3) and 22 of 25 (Part 2), and only tentative for the shortest task, where 9 of 25 models do. The models also represent at least some scoring rubrics as partly separable directions, rather than as one undifferentiated quality signal (RQ2). But separable representations do not give independent control (RQ3). Steering a rubric’s direction reliably moves the score, yet for the rubrics that are both effective and reliably measured, pushing the matching direction is essentially interchangeable with pushing a different one. The model appears to exercise one shared control with five working handles of different strength, not six independent, rubric-specific ones.

This matters for how neural AES is justified. Operational providers that need control and verifiability get them by design. In a decomposed system, each subconstruct has its own slot, and a neural feature can influence the score only through that slot (Madnani et al., 2026). An end-to-end LLM scorer has no such slot, so our audit asks whether it contains something equivalent on its own. In these base models it does not. Rubric information is present and partly separable, but it is not exposed as separate controls. High agreement with human raters, or the mere decodability of rubrics, is therefore not evidence that a model uses those rubrics as separately controllable levers. Controllability is a diagnostic that score agreement cannot supply, and the models that agree best with human raters, discussed below, show this directly.

These results also bear on the rubrics themselves. Analytic rubrics are written for human raters, who are trained to treat each criterion as a separate judgment. In these base models, the criteria are partly represented, with separable directions for the two longer tasks, but they do not work as separate levers. None of the 10 testable comparisons among the four reliably measured rubrics shows a detectable advantage for the matching direction, and a within-part test finds only small advantages in some pairs (typically 6% of the pair’s own effects, at most 14%; RQ3). In that sense, rubrics designed for human raters do not behave as independent controls in base LLMs.

Why the control is shared is an open question, because there are two readings. In one, the model collapses rubrics that the test keeps apart. In the other, the model faithfully reproduces the structure of human analytic ratings, in which dimensions are known to move together. Halo, the carry-over of a rater’s overall impression into separate trait scores, is a documented rater effect in the scoring of written composition (Engelhard Jr, 1994; Myford and Wolfe, 2004), and its relation to rating accuracy is itself contested (Fisicaro, 1988). Our gold set carries only holistic band scores, so we cannot separate the two readings here. Either way, a provider cannot assume that rubric labels from a neural scorer are separately controllable without checking.

One might worry that shared control simply reflects weak scorers, since these base models agree only modestly with human raters (Limitations). The data do not suggest that. Among the seven models whose own scores correlate with human scores at r of 0.6 or higher, the matching direction still beats a mismatched one by only 0.011 score points (95% CI [0.007, 0.015]; 1.9% of the absolute self effect), about the same as in the full set (RQ3). Higher agreement with human raters did not bring independent control.

The evidence for separation and for nonselective steering also comes from partly different rubrics, so the two should be read together. argument_relevance appears in three of the five least similar pairs in RQ2, but its steering effect is undetectable. The four rubrics that steer and are reliably measured, in contrast, have moderately to highly similar directions (Table 4). Nonselective steering among overlapping directions is what overlap alone would predict, which is why the pairwise check in RQ3 matters. With each pair’s overlap removed, both residuals still move the score and still do so about equally.

A related result concerns what the score responds to. The generic holistic direction, learned from the human band scores, raises the model’s score less than the rubric directions do at every strength we tested (Table 5), and the strongest lever of all is spelling/punctuation, a surface feature. This suggests, but does not show, that the model’s own score leans more on surface language features than on the quality signal that RQ1 can read out of its activations. The decodable signal also usually fades toward the final layer, with a median drop of 34% from the peak and worse than guessing the average score in 14 of 75 model-part cells. Information that is decodable is not necessarily information that the model uses.

Finally, the pattern is not uniform across the assessment. Part 1, BESTEP’s shortest task, shows the weakest and least stable score signal (RQ1; only 9 of 25 models pass the corrected permutation test, so its later peak is tentative) and no meaningful rubric separation (RQ2). The two longer, more integrated parts support a stronger signal and real, if not independently controllable, rubric structure. Validity claims should be checked at that grain, not assumed to generalize across task format.

Limitations

All 25 models are base (non-instruction-tuned) models. We chose them as a baseline, but this means we cannot say how instruction-tuned, chat, or task-fine-tuned variants, which resemble deployed scorers more closely, would behave. The gold set is 75 essays from one assessment (BESTEP), which limits essay-level diversity, and model coverage tops out at 10.31B parameters, so the results may not hold for substantially larger models. All models also see the same grading prompt, and we steer by adding one direction at one layer, so other prompts or other kinds of intervention could give different results. argument_relevance’s own steering direction never reaches significance; whether that reflects a representational gap or a rubric that needs more than surface language features to steer is open. More models, other writing assessments, and non-English rubric-based assessments could test whether the findings travel.

This paper tests how scoring rubrics are represented and controlled inside the models, not whether the models score like human raters. The human band score is a fixed reference only: it is the target of the RQ1 probe and the source of the generic holistic direction in RQ3. The models’ own scores agree only modestly with it (pooled Pearson r across the 75 essays: median 0.41, 7 of 25 models at r ≥ 0.6, none at 0.8), so the findings describe imperfect zero-shot base scorers, not validated scoring systems. Our check on the seven best-agreeing models is also exploratory. It is small, and we chose the cut-off of 0.6 after seeing the agreement values.

Two limits concern the degraded essays that define every rubric direction. The first is their number: all 104 come from only 26 gold responses at band 5, so each direction rests on 5 to 12 essay pairs. The second is how they were made. One LLM generated them, and two trained BESTEP graders then checked every one of the 104. Each grader, separately, saw an original and its degraded version, was told the essay had been degraded on one scoring rubric, and had to name which, and both confirmed the intended rubric every time (Appendix A). That shows the target rubric is recognizable, not that nothing else changed. The grammar/spelling_punctuation similarity ceiling in Table 4, and the near-ceiling spelling/vocabulary pair, can arise because errors in one dimension leave traces in another even in “pure” stimuli (Appendix A gives an example), so we do not read those pairs as evidence about the model alone. Cleaner single-dimension manipulations are future work.

Because the gold set carries no rubric-level human ratings, we cannot test whether shared control reflects the models or the halo structure of human analytic ratings (Discussion). A disattenuated human inter-rubric correlation matrix would let a future study separate the two.

Several statistical limits apply. The RQ1 correction for layer selection is Bonferroni, which is conservative because adjacent layers are correlated, and 1,000 label shuffles set a floor on each p-value; a max-over-layers permutation test would be exact. The paired-cell analysis is not a simple random sample, because layers, alpha values, and models sharing a training lineage are correlated. The primary inference averages within model before pooling, and the family-clustered bootstrap gives an essentially unchanged estimate, but a full hierarchical model across family, essay, rubric, and layer would be a stronger test. The pairwise check removes overlap between two directions at a time, and pairs too collinear to leave a residual are skipped, so the most entangled pairs are the ones it cannot speak to. How best to isolate one direction’s rubric-specific component from a correlated one is itself an active methodological question (Elazar et al., 2021). Finally, “no detectable advantage” is not proof of equivalence for a single pair, because the intervals for single pairs are wide (for example, [−0.018, 0.112] score points for grammar against vocabulary); only the pooled estimate is tight. We also did not correct for multiple comparisons across the 10 head-to-head and 18 within-part comparisons, and the correlation of 0.717 in RQ3 counts each of the 12 pairs twice, so its effective sample is about 12 pairs.

Conclusion

LLM-based essay scoring is attractive but opaque, and high-stakes testing asks for control and verifiability. We examined base LLMs prompted as scorers from the inside, asking whether their scoring rubrics are represented and whether they can be controlled independently. Across 25 open-weight base models, holistic scores are decodable (clearly for the two longer task types, weakly for the shortest) and scoring rubrics are partly separable, but neither property implies independent causal control. Steering a rubric’s direction moves the score as a general lever for all four rubrics that this design can measure reliably (and for five of the six overall), and removing each pair’s overlap does not change that. The same holds among the models whose scores agree best with human raters. This may reflect the models, the halo structure of human ratings, or both, and rubric-level human ratings would be needed to tell them apart. Any rubric-based LLM scorer should therefore be examined for independent controllability before its rubric labels are treated as separately meaningful, and on this evidence, base LLMs do not yet offer it. Instruction-tuned and fine-tuned scorers are the natural next comparison.

Degradation Examples

RQ2’s contrastive directions are estimated from clean-versus-degraded essay pairs (§RQ2 Method). Each degradation is produced by prompting llama3.3:70b with a spec that quotes the official BESTEP low-band descriptor for one scoring rubric and instructs the model to hold every other dimension constant. The first example below degrades one source essay on two adjacent scoring rubrics, and the second shows a cohesion degradation, to make the intended separation (and its limits) concrete.

Part 1, grammar (target: verb form, agreement, word order only).

• Original: “Three tours will be given. What should I prepare for the training? The place in my school that I visit most often is the classroom, which is the place I learn knowledges about my major.”
• Grammar-degraded: “Three tour will be give. What I should prepare for the training? The place in my school that I visiting most often are the classroom, which the place I learn knowledges about my major.”
• Spelling/punctuation-degraded (same source sentence): “Three tuors will be given. What shoud I prepair for the trainning. the place in my school that i visit most often is the clasroom which is the place i lern knowleges about my major.”

The grammar mutant changes verb forms, subject-verb agreement, and word order while leaving spelling intact; the spelling/punctuation mutant misspells content words and drops punctuation while leaving grammar intact, but it also drops sentence-initial capitalization (“the place”, “i visit”), a surface feature a grammar-sensitive signal could partly pick up on. This is a concrete illustration of the limitation discussed below: even a mutation that targets one dimension by instruction can leave a residual trace in an adjacent one.

Part 2, cohesion (target: connectives and sentence-joining only).

• Original: “I wrote this email in regards to the new food court on campus. First of all, I want to say it is a good change for the school because the food court can help save the time, the students won’t need to leave the campus to get food. […] In this way, the variety of the options and the opening hours might be appealing to more students.”
• Cohesion-degraded: “I wrote this email about the new food court on campus. It is a good change for the school. The food court can help save the time. The students won’t need to leave the campus to get food. […] The variety of the options might be appealing to more students. The opening hours might be appealing to more students.”

Here the degradation removes connectives (“First of all”, “In this way”) and splits joined sentences into flat, simple ones, with no change to individual sentences’ grammar or vocabulary, a cleaner separation than the grammar/spelling pair above, consistent with cohesion’s lower same-rubric entanglement in Table 4.

Verification. Every degraded essay was checked by two trained BESTEP essay graders before use. Working separately, each grader was shown an original essay together with its degraded version, was told that the degraded version had been downgraded on one scoring rubric, and had to identify which. All 104 degraded essays (20 for Part 1, 36 for Part 2, and 48 for Part 3, that is 5, 9, or 12 for each part and scoring rubric, made from 26 band-5 gold responses) were checked this way. Both graders confirmed the intended scoring rubric for every one, a consensus of 100%, and the check was run as a random double sample and as a cross-comparison between the graders. This establishes that the intended scoring rubric is recognizable in each degraded essay. It does not establish that no other dimension changed, and the grammar/spelling example above shows that adjacent dimensions can carry a trace. The degraded essay pairs contain exam responses and are not yet public; they are available from the authors for research purposes on request.

Per-Model Depth Profiles

Figure 5 shows the RQ1 depth-profile curve (Figure 2’s pooled version, disaggregated) for each of the 25 models individually, for readers who want to check whether the pooled pattern in Figure 2 holds for a specific model rather than only in aggregate.

Model Catalog

Table 6 lists the 25 open-weight base models probed and steered in this study, sorted by parameter count.

References