Keywords: Automated Essay Scoring, Large Language Models, Rubric-Based Assessment, Interpretability, Construct Validity

Introduction

Automated essay scoring (AES) has moved from hand-built features to neural models and, more recently, to general-purpose large language models (LLMs) (Johnsi and Kumar, 2026). Agreement with human raters rose along the way, but the models became harder to inspect. Transparency and interpretability remain open priorities in the AES literature (Johnsi and Kumar, 2026). They matter most in high-stakes testing, where a provider must control what a scorer responds to and verify that its scores reflect the intended construct. Providers respond in different ways. Some rely on human raters, or on feature-based machine learning in which humans define what is measured. The Duolingo English Test decomposes its writing construct into subconstructs and confines any LLM-derived feature to a single subconstruct, so that the model cannot influence the score except through that one slot. Its authors are explicit about the trade-off: scoring directly with deep learning “cedes both control and verifiability” (Madnani et al., 2026).¹

That design keeps the black box out of the score; it does not show what a neural scorer does inside. If LLMs are to score essays more directly, we need to know whether a model’s internal representations contain the rubric’s criteria at all, and whether those criteria can be controlled one at a time. Score agreement cannot show this, because a model can track holistic quality without using the rubric’s analytic structure. Prior interpretability work on AES examines what a scorer has learned or which features it weights (Fiacco et al., 2023; Wang et al., 2026); we add causal intervention on the representations themselves.

We therefore audit LLM-based AES from the inside, probing and steering 25 open-weight base LLMs (0.13B to 10.31B parameters, 14 families) as they score BESTEP (Taiwan’s Ministry-of-Education English exam) responses across three task types and six analytic criteria. RQ1: is the holistic score decodable from a model’s activations, and where does that signal live? RQ2: do the six criteria occupy distinguishable directions in representation space? RQ3: does steering a criterion’s direction move the score specifically for that criterion, or does any rubric-adjacent push move it about equally? We choose base models as a baseline, so that fine-tuned variants (instruction-tuned, chat) can later be compared against it. Our contributions are a white-box audit protocol for rubric-based AES that separates decodability, separability, and causal control, and evidence that, in these models, the first two do not imply the third. The code and the database behind every number are public at https://github.com/lopentu/inside-the-scorer-paclic2026.

Related Work

AES and LLM-based scoring. Automated essay scoring (surveyed in Johnsi and Kumar (2026)) first moved from feature-engineered methods to BERT-based deep learning: domain-adaptive training, multi-scale essay representations, and hierarchical transfer learning each closed part of the gap to human-rater agreement (Ramesh and Sanampudi, 2022; Wang et al., 2022; Cao et al., 2020; Xue et al., 2021; Settles et al., 2020). A second wave prompts general-purpose LLMs directly as scorers or raters, with a mixed record. Some studies report LLM ratings holding up under real psychometric scrutiny, generalizability-theory and many-facet Rasch analyses across thousands of essays and multiple genres (Song et al., 2025; Wang et al., 2025a), and case studies where an LLM produces both a score and trait-level comments across several analytic criteria at once (Wang et al., 2025b; Cho et al., 2024). Others find the opposite: robustness failures under adversarial stress-testing and demographic bias in ratings (Kabra et al., 2022; Yamashita, 2025), systematic disagreement between LLM and human grading behavior (Mathew et al., 2026), and validity that shifts by language, genre, and proficiency level rather than holding uniformly (Mizumoto and Eguchi, 2023; Pack et al., 2024; Mansour et al., 2024; Gaggioli et al., 2025; Barbosa et al., 2025; Yavuz et al., 2025; Yancey et al., 2023). A related strand looks past scoring accuracy to classroom use, from ChatGPT’s promise and pitfalls as a writing tutor to its measurable effect on students’ own writing when used for feedback (Barrot, 2023; Ding et al., 2025).

Control and verifiability in operational scoring. High-stakes providers mostly manage opacity through design. The Duolingo English Test partitions features by subconstruct, so a deep-learning feature, including a fine-tuned LLM rating, reaches the score only through its own subconstruct, and the contribution of each subconstruct can be measured and explained (Madnani et al., 2026). The same report notes that explaining why an end-to-end neural scorer assigned a score remains substantially harder. Prior analyses of neural AES scorers are observational: Wang et al. (2026) analyze how LLM scorers weight textual features, and Fiacco et al. (2023) align components of a transformer scorer’s activations with human-understandable feature groups. We intervene causally on the representations.

Rubric-aware and rationale-based scoring. A separate line augments scores with rubric-level explanations or rationales (Bueno et al., 2026; Chu et al., 2025; Jiao et al., 2025; Kucia et al., 2026). These works evaluate rubric outputs; we evaluate rubric representations.

Mechanistic interpretability and faithfulness. Our probing and steering methodology builds on activation-patching practice (Zhang and Nanda, 2023), causal concept-based explanation (Wu et al., 2023b), and work questioning whether model-generated explanations are faithful to internal computation (Atanasova et al., 2023; Jacovi and Goldberg, 2021; Chuang et al., 2026; Ferrando et al., 2023; Demircan et al., 2024), a concern that applies to AES, where a plausible-sounding rationale is not evidence of criterion-specific control. The probing literature motivates our two checks. Elazar et al. (2021) show that a property’s decodability from a representation does not mean the model actually uses that property, only that it is present, and Hewitt and Liang (2019) show that a probe’s accuracy is uninterpretable without a selectivity check against a control task the probe could only solve by memorization rather than by reading the representation. RQ1’s decodability result and RQ3’s steering-selectivity test correspond to these two checks, applied to rubric criteria rather than linguistic properties.

Criterion-adjacent NLP. For the grammar and vocabulary criteria specifically, we draw on grammatical error correction (Bryant et al., 2023; Wu et al., 2023a; Fang et al., 2023) and lexical and proficiency assessment (Aleksandrova and Pouliot, 2023; Bannò et al., 2024; Bannò et al., 2025) literatures, and for inter-rater reliability, the standard kappa statistic (Cohen, 1960).

Setup

Models. We probe and steer 25 open-weight, base (non-instruction-tuned) LLMs spanning 0.13B to 10.31B parameters and 16 to 42 layers across 14 model families (Llama, Gemma, Falcon3, SmolLM2/3, Granite, Mistral, OLMo, TAIDE, Apertus, and others; see Appendix C, Table 6 for the full catalog), selected for complete output coverage across every phase of the pipeline (probing, geometry, and steering). All 25 are base models, by design. Operational scorers are often fine-tuned (the Duolingo English Test, for instance, uses LLM features fine-tuned on human ratings (Madnani et al., 2026)), but a base model carries no instruction or format tuning that could confound the score readout, so it gives a clean baseline against which instruction-tuned, chat, and other fine-tuned variants can be compared in future work. The parameter range also reflects a deployment constraint raised in the language-testing literature: local institutions weighing AI-assisted assessment need models practical to host on-premises, keeping response data local, rather than only the largest cloud-hosted systems (Chuang and Yan, 2025).

Dataset. All models are probed and scored on the same fixed set of 75 human-verified BESTEP writing responses (the gold-75), spanning two test forms and three task parts (Table 1): Part 1 (Answering Questions, a short factual-response task), Part 2 (Expressing Opinions), and Part 3 (Integrated Writing, the longest and most demanding task). BESTEP’s rubric assesses six analytic criteria, but not every criterion applies to every part (Table 3): grammar and vocabulary apply throughout, spelling and punctuation is scored only in the short-answer part (Part 1), and argument relevance only in the integrated-essay part (Part 3).

Activation extraction. For every model, we extract final-token residual-stream activations at every layer for each of the 75 responses; these activations are the shared input to the probing (RQ1), contrastive-direction (RQ2), and steering (RQ3) procedures described in the next three sections.

Code and Data Availability. Code, the analysis scripts, and the database slice behind every reported number are available at https://github.com/lopentu/inside-the-scorer-paclic2026. They are released for noncommercial research use under the PolyForm Noncommercial License 1.0.0. The degraded essay pairs (Appendix A) contain exam responses and are not yet public; they are available from the authors for research purposes on request.

Experiments

Score-Information Representation

Method. For each (model, layer, part) cell we fit a Ridge regression probe on final-token residual-stream activations to predict the response’s holistic band score, using leave-one-out cross-validation (LOOCV; Figure 1): each of the 21 (Part 1), 24 (Part 2), or 30 (Part 3) responses is held out once, the probe is trained on the rest, and every held-out prediction is pooled to compute one R2 per cell. We report the peak-R2 layer, the peak R2 itself, and the peak-to-final-layer drop, using the pooled-form scope (both test forms combined) as the headline result. To check that a peak R2 exceeds what the probing procedure produces on its own, each cell also gets a label-shuffle permutation test (1,000 shuffles of the scores). Because the peak is the best of 16 to 42 layers, we report the peak layer’s p-value both raw and Bonferroni-corrected over that model’s layers.

Results. Score information is decodable in most models, but not equally for every part, and not in a simple “easy parts early, hard parts late” order. Against the permutation null, the peak layer beats chance in all 25 models for Part 3 and in 22 of 25 for Part 2 after the layer correction, but in only 9 of 25 for Part 1 (24 of 25 uncorrected). The one consistent timing pattern: Part 1 (the shortest, short-answer task) peaks latest or second-latest among the three parts in 23 of 25 models (Figure 2). A response-length explanation for this timing does not hold up: relative peak-to-final signal drop ranges from 0.0% to 2611.3% across models, and the part with the largest drop varies by model, so timing alone cannot be read as a complexity gradient.

Peak timing and decodability strength diverge. Part 1 has the lowest peak R2 in 24 of 25 models and is never the highest; across models, mean peak R2 is 0.432 for Part 1, 0.633 for Part 2, and 0.773 for Part 3 (Table 2, Figure 3). Part 1 is also the least stable across models (peak-R2 variance 0.0338, versus 0.0095 for Part 2 and 0.0038 for Part 3). Part 1 has the fewest gold responses (21), the shortest responses (about 47 words on average, versus about 117 for Part 2 and about 214 for Part 3), and the coarsest scale, which plausibly explains why its signal is both weaker and more model-dependent: less textual evidence and fewer observations for the probe to exploit.

For the two longer tasks, models carry information about essay quality that a label-shuffle null cannot explain. For BESTEP’s shortest task type the signal peaks later, is weaker and less stable, and is above the corrected null in only 9 of 25 models. We cannot explain this pattern mechanistically here, but RQ2 and RQ3 return to it. The Bonferroni correction is conservative because adjacent layers are correlated, and 1,000 shuffles bound how small a p-value can get (Limitations).

Construct Representation

Method. RQ2 uses the same 75 gold-standard responses plus criterion-targeted clean-versus-degraded mutations, since holistic scores blend criteria and cannot support construct-geometry claims on their own. Mutations are generated by prompting llama3.3:70b (via Ollama) with one of 12 rubric-quoting specs, one per (part, criterion) combination the rubric scores (Table 3), each spec quoting the official BESTEP band descriptors for that criterion and instructing the model to degrade the essay to the low band on that dimension only, holding every other dimension, and meaning, constant (length preserved to within 10%). Two trained BESTEP graders verified each mutated essay against its intended target dimension before use (Appendix A gives the procedure and worked examples). For each (model, layer, part, criterion) cell we estimate a contrastive direction as the normalized difference between clean and mutated-essay activation centroids, and its split-half reliability by recomputing the direction from random halves of the essay pairs. We compare same-criterion to different-criterion cosine similarity within the same essay part (a same-part, cross-criterion comparison avoids confounding “different construct” with “different essay”), retaining only reliability-gated rows (split-half reliability at least 0.6).

Results.

Separation is uneven across the assessment. Averaged across 1,470 reliability-gated layer-part cells, the mean same-versus-different geometry gap is 0.334 (74.5% of cells positive), but this is not stable across task parts: Part 1’s four criteria are, on average, negative (mean gap -0.220, 6.2% of rows positive), no more separated from each other than a single criterion is from itself across parts, while Parts 2 and 3 separate cleanly (89.7% and 99.2% of cells positive): reliable, criterion-specific directions are broadly available in Parts 2 and 3 but sparse in Part 1.

Pair by pair, the range is wide. Pooling all reliability-gated rows by literal criterion pair, using disattenuated cosine similarity (a correction for each direction’s own measurement noise, so a value of 1 means statistically indistinguishable from the same construct and 0 means unrelated), the 12 comparable pairs range from anti-correlated to statistically identical (three of the 15 possible pairs never co-occur in the same part and are not comparable at all; Table 4, alongside RQ3’s steering results below). argument_relevance paired with grammar is the most separated pair (-0.102, anti-correlated beyond noise), and argument_relevance, a Part-3-only criterion, carries most of the separation signal in the whole rubric. At the opposite end, grammar paired with spelling_punctuation sits at the 1.0 ceiling: the two are statistically indistinguishable from the same construct, and spelling_punctuation paired with vocabulary is nearly as entangled (0.984). Both of the most-entangled pairs are Part-1-only comparisons, consistent with Part 1’s negative geometry gap above.

So separation is broad in the two longer, more integrated task types, concentrated in one criterion (argument relevance) beyond that, and absent in the shortest task type.

Causal Specificity

Method. RQ3 reuses RQ2’s criterion directions and tests them causally rather than only representationally. A direction is scaled to unit length and added to the residual stream at one layer, at every token position of the grading prompt, with magnitude alpha times that layer’s mean residual-stream norm (alpha = 0.05, 0.1, 0.25, 0.5), so strength is comparable across layers and models. The outcome is the change in expected band score: the probability-weighted mean over the digit tokens at the answer position of the same grading prompt (no learned readout), minus the unsteered baseline. For degraded essays with a known target criterion we inject four kinds of direction: the matching criterion direction (self), a mismatched rubric direction (other), an orthogonal control (a criterion direction with the axis shared by all criteria in the part removed, which tests whether overlap between directions hides specificity), and a generic holistic direction (the direction the RQ1 probe uses to predict the band score, which tests whether any score-raising push works). Self and other effects are paired at the finest valid unit, (model, layer, part, form, degradation, alpha), giving 74,304 paired cells across all 25 models, and we test whether the matching direction beats a mismatched one by a practically meaningful margin.

Results.

Steering works for five of six criteria. Grammar, vocabulary, cohesion, spelling/punctuation, and task relevance each show a clearly positive self effect (0.168 to 0.359 score points). Only argument_relevance’s own direction produces no detectable effect: it neither raises nor lowers the score.

Steering is not selective. Pooled across all paired cells, the matching direction beats a mismatched one by only 0.0074 score points on average (95% CI [0.0036, 0.0117]; 1.74% of the absolute self effect, 95% CI [0.85%, 2.74%]), and this holds under every robustness check we ran: the gap holds at the cell level (0.0085, 95% CI [0.0051, 0.0119]), never exceeds 0.0083 under leave-one-model/family-out, and a cluster bootstrap that resamples whole model families rather than individual models, accounting for shared training lineage such as the three Llama-3.x checkpoints, gives an almost identical estimate (0.0073, 95% CI [0.0035, 0.0108]).

The pooled number combines two different patterns. Broken out pair by pair (Figure 4), among the four criteria that both work and are reliably measured (grammar, vocabulary, cohesion, spelling/punctuation), the matching direction shows no detectable advantage over a mismatched one in 11 of their 12 head-to-head comparisons; the one exception is vocabulary losing to spelling/punctuation, not because vocabulary is weak, but because spelling/punctuation happens to be the single strongest general-purpose lever of the six. argument_relevance loses every comparison it has data for, for the different reason that its own direction never moves the score. Re-run separately within each task part (36 within-part comparisons among comparably-strong criteria), the same pattern holds in every part: not one instance of a criterion beating a comparably strong rival specifically because it matched the essay’s degradation.

Two controls calibrate these effects (Table 5). The orthogonal control does not behave like a flat baseline: it lowers the score slightly and more so at larger alpha, a small movement with the opposite sign of any working self effect, so the model is not indifferent to an injected direction, only to which criterion it targets. The generic holistic direction raises the score less than self or other at every alpha tested, which argues against a pure “generic tampering” account of the specificity gap above; its alpha slope falls inside the self/other range, so it behaves like a weaker member of the same family of levers. At the strongest alpha (0.5) the average self effect is 0.361 score points on the 1-6 band scale, about 6% of the band range, a moderate intervention.

Tying back to RQ2, a pair’s representational separation (Table 4, above) does predict how large a causal difference that pair shows under steering (r=0.717 across all comparable pairs), but it does not predict which direction wins.

RQ2’s directions are reliability-gated, but RQ3 injects the raw, non-disattenuated direction regardless of noise, and different criteria’s directions are already substantially correlated (Table 4). Noise and this overlap could be masking a larger true specificity, and three checks address that. The first restricts to progressively more reliable directions, which raises the specificity ratio from 1.8% (all cells) to 5.4% (RQ2 geometry available) to 3.1% at reliability at least 0.7 to 8.3% at the top reliability quartile (self 0.670 vs. other 0.605, gap 95% CI [0.057, 0.072]): less reliable directions do understate true specificity, but even the best-measured quarter shows self only modestly outperforming other. A second check removes the axis shared by all criteria in a part from each direction (the orthogonal-control condition above) and repeats self vs. other on these overlap-corrected directions, using data already collected. If overlap were concealing real independent control, this should reveal it, and it does not. Both self_orth and other_orth average slightly negative (-0.093, -0.100), and their gap (0.0064, 95% CI [0.0003, 0.0125], ratio 4.82%) matches the reliability-gated result: once the shared component is removed, the criterion-specific residual is not a strong positive lever for either criterion, so most of the raw effect was the shared component, not overlap hiding independent control.

A third check isolates what a direction shares only with its specific rival. For each pair of criteria we strip from each direction its projection onto the other, so that the matching and mismatched directions lose exactly their mutual overlap (pairs too collinear to leave a residual are skipped), and inject the two residuals next to a norm-matched random direction as a noise floor. Across all 25 models (884 pair cells), both residuals raise the score above the noise floor, by 0.194 (matching) and 0.181 (mismatched) points, so criterion-specific residual structure exists and moves the score. The matching residual still does not win: the gap is 0.013 points (95% CI [-0.015, 0.042]), about 4.5% of the average absolute matching effect, and the interval includes zero. Pairs below and above a cosine of 0.6 give the same picture (gaps of 0.012 and 0.014). Removing the overlap pair by pair leaves the levers interchangeable.

Discussion

Taken together, the three research questions give one account of what a rubric-based LLM scorer does inside. The model carries decodable information about essay quality (RQ1), strongly for the two longer tasks and weakly for the shortest. It represents at least some rubric criteria as distinguishable directions rather than one undifferentiated quality signal (RQ2). But distinguishable representation does not translate into independently controllable behavior (RQ3). Steering a rubric direction reliably moves the score, yet pushing the matching direction is, for the criteria that are both effective and reliably measured, essentially interchangeable with pushing a different one. The model appears to exercise one shared control with several equally effective handles, not six independent, criterion-specific ones.

This matters for how neural AES is justified. Operational providers that need control and verifiability get it by construction: in a decomposed design each subconstruct has its own slot, and a neural feature can influence the score only through its slot (Madnani et al., 2026). An end-to-end LLM scorer offers no such slot, and our audit asks whether it contains something equivalent on its own. In these base models it does not: criterion information is present and partly separable, but it is not exposed as separate controls. High agreement with human raters, or the mere decodability of rubric criteria, is therefore not evidence that a model uses those criteria as separately controllable scoring levers. Controllability is a diagnostic that score agreement cannot supply.

What the diagnostic means is open, because shared control has two readings. One is that the model collapses criteria the rubric keeps apart. The other is that the model faithfully reproduces the structure of human analytic ratings, in which dimensions are known to move together: halo, the carry-over of a rater’s overall impression into separate trait scores, is a documented rater effect in the scoring of written composition (Engelhard Jr, 1994; Myford and Wolfe, 2004), and its relation to rating accuracy is itself contested (Fisicaro, 1988). Our gold set carries only holistic band scores, so we cannot separate the two readings here. Either way, a provider cannot assume that criterion labels from a neural scorer are separately controllable without checking.

The evidence for separation and for nonselective steering also comes from partly different criteria, and the two should be read together. argument_relevance carries most of the separation signal in RQ2 but its steering effect is undetectable, whereas the four criteria that steer and are reliably measured have moderately to highly similar directions (Table 4). Nonselective steering among overlapping directions is what overlap alone would predict, which is why the pairwise check in RQ3 matters: with each pair’s overlap removed, both residuals still move the score and still do so about equally.

The pattern is not uniform across the assessment design. Part 1, BESTEP’s shortest task, shows the weakest and least stable score signal (RQ1) and no meaningful criterion separation (RQ2), while the two longer, more integrated parts support a stronger signal and real, if not independently controllable, criterion structure. Validity claims should be checked at that grain, not assumed to generalize across task format.

Limitations

All 25 models are base (non-instruction-tuned). We chose them as a baseline, but it means we cannot say how instruction-tuned, chat, or task-fine-tuned variants, which resemble deployed scorers more closely, would behave. The gold set is 75 essays from one assessment (BESTEP), which bounds essay-level diversity. Model coverage tops out at 10.31B parameters, so results may not hold for substantially larger models. argument_relevance’s own steering direction never reaches significance; whether that reflects a representational gap or a criterion that needs more than surface language features to steer is open. More models, other writing assessments, and non-English rubric-based assessments could test whether the findings travel.

Two limits concern the degraded essays that define every criterion direction. They were generated by one LLM, and two trained BESTEP graders then checked every one of the 104: each, separately, saw an original and its degraded version, was told the essay had been degraded on one rubric criterion, and had to name which, and both confirmed the intended criterion every time (Appendix A). That shows the target criterion is recognizable, not that nothing else changed. The grammar/spelling_punctuation similarity ceiling in Table 4, and the near-ceiling spelling/vocabulary pair, can arise because errors in one dimension leave traces in another even in “pure” stimuli (Appendix A gives an example), so we do not read those pairs as evidence about the model alone. Cleaner single-dimension manipulations are future work.

Because the gold set carries no criterion-level human ratings, we cannot test whether shared control reflects the models or the halo structure of human analytic ratings (Discussion); a disattenuated human inter-criterion correlation matrix would let a future study separate the two.

Three statistical limits apply. First, the RQ1 correction for layer selection is Bonferroni, which is conservative because adjacent layers are correlated, and 1,000 label shuffles set a floor on each p-value; a max-over-layers permutation test would be exact. Second, the paired-cell analysis is not a simple random sample: layers, alpha values, and models sharing a training lineage are correlated. The primary inference averages within model before pooling, and the family-clustered bootstrap gives an essentially unchanged estimate, but a full hierarchical model across family, essay, criterion, and layer would be a stronger test. Third, the pairwise check removes overlap between two directions at a time, and pairs too collinear to leave a residual are skipped, so the most entangled pairs are the ones it cannot speak to. How best to isolate one direction’s construct-specific component from a correlated one is itself an active methodological question (Elazar et al., 2021).

Conclusion

LLM-based essay scoring is attractive but opaque, and high-stakes testing asks for control and verifiability. We examined the scorer from the inside, asking whether an LLM scorer’s rubric criteria are represented and independently controllable. Across 25 open-weight base models, holistic scores are decodable (clearly for the two longer task types, weakly for the shortest) and rubric criteria are partly separable, but neither property implies independent causal control: steering a rubric direction moves the score as a general lever for the great majority of the criteria this design can measure reliably, and removing each pair’s overlap does not change that. This may reflect the models, the halo structure of the human ratings they imitate, or both; criterion-level human ratings would be needed to tell them apart. Independent controllability is a diagnostic that any rubric-based LLM scorer should be examined with before its rubric labels are treated as separately meaningful, and on this evidence, base LLMs do not yet offer it. Instruction-tuned and fine-tuned scorers are the natural next comparison.

Degradation Examples

RQ2’s contrastive directions are estimated from clean-versus-degraded essay pairs (§RQ2 Method). Each degradation is produced by prompting llama3.3:70b with a spec that quotes the official BESTEP low-band descriptor for one criterion and instructs the model to hold every other dimension constant. The first example below degrades one source essay on two adjacent criteria, and the second shows a cohesion degradation, to make the intended separation (and its limits) concrete.

Part 1, grammar (target: verb form, agreement, word order only).

• Original: “Three tours will be given. What should I prepare for the training? The place in my school that I visit most often is the classroom, which is the place I learn knowledges about my major.”
• Grammar-degraded: “Three tour will be give. What I should prepare for the training? The place in my school that I visiting most often are the classroom, which the place I learn knowledges about my major.”
• Spelling/punctuation-degraded (same source sentence): “Three tuors will be given. What shoud I prepair for the trainning. the place in my school that i visit most often is the clasroom which is the place i lern knowleges about my major.”

The grammar mutant changes verb forms, subject-verb agreement, and word order while leaving spelling intact; the spelling/punctuation mutant misspells content words and drops punctuation while leaving grammar intact, but it also drops sentence-initial capitalization (“the place”, “i visit”), a surface feature a grammar-sensitive signal could partly pick up on. This is a concrete illustration of the limitation discussed below: even a mutation that targets one dimension by instruction can leave a residual trace in an adjacent one.

Part 2, cohesion (target: connectives and sentence-joining only).

• Original: “I wrote this email in regards to the new food court on campus. First of all, I want to say it is a good change for the school because the food court can help save the time, the students won’t need to leave the campus to get food. […] In this way, the variety of the options and the opening hours might be appealing to more students.”
• Cohesion-degraded: “I wrote this email about the new food court on campus. It is a good change for the school. The food court can help save the time. The students won’t need to leave the campus to get food. […] The variety of the options might be appealing to more students. The opening hours might be appealing to more students.”

Here the degradation removes connectives (“First of all”, “In this way”) and splits joined sentences into flat, simple ones, with no change to individual sentences’ grammar or vocabulary, a cleaner separation than the grammar/spelling pair above, consistent with cohesion’s lower same-criterion entanglement in Table 4.

Verification. Every degraded essay was checked by two trained BESTEP essay graders before use. Working separately, each grader was shown an original essay together with its degraded version, was told that the degraded version had been downgraded on one rubric criterion, and had to identify which. All 104 degraded essays (20 for Part 1, 36 for Part 2, and 48 for Part 3, that is 5, 9, or 12 per part-criterion cell) were checked this way. Both graders confirmed the intended criterion for every one, a consensus of 100%, and the check was run as a random double sample and as a cross-comparison between the graders. This establishes that the intended criterion is recognizable in each degraded essay. It does not establish that no other dimension changed, and the grammar/spelling example above shows that adjacent dimensions can carry a trace. The degraded essay pairs contain exam responses and are not yet public; they are available from the authors for research purposes on request.

Per-Model Depth Profiles

Figure 5 shows the RQ1 depth-profile curve (Figure 2’s pooled version, disaggregated) for each of the 25 models individually, for readers who want to check whether the pooled pattern in Figure 2 holds for a specific model rather than only in aggregate.

Model Catalog

Table 6 (see page 10) lists the 25 open-weight base models probed and steered in this study, sorted by parameter count.

References