TestGRAD

01 · Problem

Selection must be grounded in behavior.

SWE agents have complementary strengths: an agent that fails on one issue may still produce the best patch for another. An ensemble runs several agents on the same issue and then has to select one of their candidate patches.

An LLM judge can compare the candidates, but without execution evidence it sees surface plausibility rather than repository behavior, and it can be sensitive to presentation order. Selection should be test-based: generate tests, execute every candidate patch, and use the outcomes to identify the correct one. The hard part is getting tests that actually tell the candidates apart.

The gap

Existing test generators are limited optimizers.

No effective loss

A test that simply passes or fails is not necessarily useful for selection. Generators are not optimized to separate the candidates by behavior.

Differential lossStage 2 · the target

Incomplete update direction

Most create a test from the issue and, at most, delete conflicting tests by name matching. Existing fixtures, helpers and stale assertions are left untouched.

Full CRUD gradientStage 4 · the direction

One-off generation

When the first test does not discriminate, nothing turns that failure into the next attempt.

Failure Pattern MomentumStage 3 · memory across attempts

02 · Method

Test-suite evolution as gradient descent.

TestGRAD treats the repository test suite as the parameter state of an optimizer.

In the forward pass it executes the current suite on every candidate patch and checks whether their behaviors differ. In the backward pass an LLM optimizer turns this execution signal, together with momentum mined from earlier failures, into the next CRUD edit of the suite. The loop ends as soon as one test separates the candidates, or when the iteration budget is used up.

The analogy · Table 1 of the paper

Gradient descent with momentum, role by role.

Role	Gradient descent with momentum	TestGRAD
State	θ_tCurrent parameter vector.	y_tCurrent candidate test suite.
Objective	𝓛(θ_t)Loss function.	𝓛(y_t)Differential loss, defined by execution.
Retention	β ∈ [0, 1]How much of the previous velocity is retained.	β = MinSupportMinimum frequency a failure sequence must reach. Lowering it retains more sequences and gives momentum broader influence.
Momentum	β ∇_t−1Accumulated direction from previous steps.	m_t = Mine_β(M_t−1 ‖ ∇_t−1)Distilled from recurring failure sequences in memory M, including the latest failed update.
Update direction	∇_t = β ∇_t−1 + (1 − β) ∂𝓛/∂θ_tInstantaneous gradient combined with momentum.	∇_TestGrad = Optimizer(𝓛, y_t, m_t)Current feedback combined with momentum, written by the LLM optimizer.
Apply	θ_t+1 = θ_t − η ∇_tParameter update.	y_t+1 = y_t + ∇_TestGradA combination of CRUD operations on the test suite.

How to readEach row pairs one element of gradient descent with momentum (middle) with the part of TestGRAD that plays the same role (right).

What it showsA one-step CRUD edit plays the role of the instantaneous gradient; a recurring failed edit sequence plays the role of momentum carried across attempts. The correspondence is between roles: the gradient is a code edit written by an LLM, not a numerical derivative.

Overview · Figure 1 of the paper, redrawn

One iteration, four stages.

How to readStart at the left, follow the arrows to the loss, then along the bottom row back to the test suite. Numbers mark the four stages explained below. Blue is what TestGRAD optimizes and writes; green is the exit.

What it showsCandidate patches from N agents and the current suite y_t are executed together (1). If some test separates the candidates, the loss is zero (2) and a checker, invoked once with a separate context, uses that result to select the final patch. Otherwise the failed update goes into failure memory and is mined into momentum (3), and the LLM optimizer writes the next CRUD update (4). In the experiments the loop runs for at most 8 steps.

1Stage 1 · Forward pass

Run the suite on every candidate patch.

The forward pass is purely execution-based. TestGRAD runs the current suite y_t against every candidate patch and records pass or fail. A suite is useful for selection only if at least one test is differential: it passes on some candidates and fails on others. Uniform outcomes, all pass or all fail, carry no selection signal.

Test	p₁	p₂	p₃	p₄	Signal
test_a	✓	✓	✓	✓	uniform · no signal
test_b	✕	✕	✕	✕	uniform · no signal
test_c	✕	✓	✕	✓	differential

p₁

p₂

p₃

p₄

test_a

✓

uniform · no signal

test_b

uniform · no signal

test_c

✓

differential

How to readRows are tests, columns are candidate patches; each cell is one execution, pass (✓) or fail (✕). Illustrative outcomes.

What it showstest_a and test_b give the same result on every candidate, so they cannot help choose. test_c passes on p₂ and p₄ and fails on p₁ and p₃: with it, the suite separates the candidates.

2Stage 2 · Differential loss

The loss is zero once a test separates the patches.

The loss is defined by execution, following the differential-testing intuition that behavioral differences across executions expose semantic distinctions. It gives the optimizer an explicit target, and it is different from the pass counting that existing pipelines use for the final choice.

TestGRAD · differential loss

Search for behavioral contrast

𝓛(y_t) = 0 if y_t contains a differential test; 1 otherwise

𝓛 = 0Optimization stops. A checker agent, invoked once outside the loop with a separate context, uses the differential result to select the final patch. 𝓛 = 1The failed update becomes the optimization signal for the next iteration.

Contrast · pass-count selection (Agentless, TRAE)

Pick the patch that passes the most tests

This rewards passing, not contrast. Non-differential tests can still favor broadly plausible or under-tested patches, and nothing drives the generator toward a test whose outcome differs across candidates.

In the ablation (RQ3), replacing the differential loss and checker with this rule lowers Pass@1 from 84.2% to 80.8%.

3Stage 3 · Failure Pattern Momentum

Mine repeated failures into momentum.

If an update leaves the suite non-differential, it has failed to separate the candidates, and TestGRAD writes it into failure memory M. Instead of replaying every raw failure, it mines maximal frequent failure sequences from that memory:

m_t = Mine_β(M_t−1 ‖ ∇_t−1)

A sequence is kept only if it appears in at least β failed test suites (2 in the experiments). The result records which directions have already failed, so the next update can move away from the same dead end. Unlike a summary written by another LLM call, it is deterministic and faithful to the observed failures; unlike appending the raw history, it stays small (RQ4).

Example · Figure 3 of the papersphinx-doc__sphinx-10673

From warning checks to the toctree's structure

(a) current suite y_t · warning check

# Read and Reuse: app/warning fixtures
def test_toctree_special_pages(
    app, status, warning):
  app.build()
  warnings = warning.getvalue()
  assert "...genindex'" not in warnings
  assert "...modindex'" not in warnings
  assert "...search'" not in warnings
  assert 'index' in
      app.env.toctree_includes

Uniform outcomes on the candidates: non-differential.

(b) momentum m_t = Mine(M)

|M| = 3 failed suites
-> retained frequent sequence
-> all converge on warning checking

  app.build()
  warn = warning.getvalue()
  assert "...genindex'" not in warn
  assert "...modindex'" not in warn
  assert "...search'"   not in warn

Dead end: "assert no warning"
  exhausted (3/3 attempts)
Steer: inspect AST via assert_node()

The shared core of the three failures, and a steer away from it.

(c) updated suite y_t+1 · AST probe

# avoid dead-end warning-check
# new: probe toctree AST structure
def test_toctree_index(app):
  app.build()
  toctree = app.env.tocs['index']
  node = toctree[0][1][1]
  assert_node(node, addnodes.toctree,
    caption="Indices",
    entries=[(None, 'genindex'),
             (None, 'modindex'),
             (None, 'search')])

A discriminative signal for the checker.

How to readLeft to right: the failed suite, the momentum mined from failure memory (highlighted lines are the retained frequent sequence), and the suite the optimizer writes next.

What it showsThree failed suites in memory all converge on checking warning text, which gives uniform outcomes. Momentum marks that direction as a dead end and steers the optimizer toward AST inspection; the new test probes the toctree with assert_node() and produces a discriminative signal.

4Stage 4 · Backward pass

A full CRUD update to the repository tests.

The optimizer receives the loss, the current suite and the momentum, and returns the next test-space gradient, ∇_t = Optimizer(𝓛, y_t, m_t); the new suite is y_t+1 = y_t + ∇_t. The gradient is an edit over the original repository test suite. Existing pipelines mostly create a test and, at most, delete conflicting ones by name; repository tests often depend on fixtures, setup routines and helpers, and on expected values that must change with the fix.

Optimizer prompt · Figure 2 of the paper, abridged

Given issue {I}, current test suite {y_t}, execution result {loss L(y_t)}, and momentum {m_t}:

Explain why the current suite is still non-differential.
Identify the failed direction preserved in momentum.
Return one CRUD update over the original repository tests: [R]ead + Reuse, [C]reate, [U]pdate, [D]elete.

The loss sets the target and the momentum names the direction that already failed, so the update moves away from previous non-discriminative attempts.

The four update directions

Read + Reuse beyond C/D

Collect the repository's testing knowledge: fixtures, imports, assertion utilities, setup conventions.

Create

Add new discriminative tests.

Update beyond C/D

Revise stale assertions or test bodies whose expected behavior changes with the fix.

Delete

Remove only tests confirmed obsolete, rather than deleting by name matching.

What Read collects · supplementary material

Six aspects, derived from 200 sampled SWE-bench instances (SWE-bench Verified excluded). Each appears in over 60% of the target test files; the bar shows how often.

96%Project APIs and imports

91%Assertion conventions

83%setUp and fixtures

79%Test base classes

71%Expected values

64%Helper functions

Example · supplementary materialdjango__django-12325

`OneToOneField` to a parent model should imply `parent_link=True`

R Read · tests/invalid_models_tests/

@isolate_apps('invalid_models_tests')
class InvalidModelsTests(SimpleTestCase):
    def test_missing_parent_link(self):
        with self.assertRaisesMessage(
                ImproperlyConfigured, msg):
    self.assertEqual(Model.check(), [])
    self.assertEqual(Model.check(), [Error(id=...)])
from django.core.checks import Error, Warning

Four pieces of infrastructure are extracted (blue) and reused.

C Create a discriminative test

+ @isolate_apps('invalid_models_tests')
+ class TestParentLink(SimpleTestCase):
+     def test_onetoone_parent(self):
+         class Place(models.Model): pass
+         class ParkingLot(Place):
+             other = OneToOneField(
+                 Place, CASCADE,
+                 related_name='other_parking')
+         self.assertEqual(
+             ParkingLot.check(), [])

Asserts the new behavior, with no configuration boilerplate.

U Update an existing assertion

  def test_render(self):
      OneToOneField(
          'migrations.Tag',
          CASCADE,
-         parent_link=False,
+         parent_link=True,
          primary_key=True,
          to_field='id',
          serialize=False,
          ...)

The updated test passes exactly on the correct patch and fails on the rest.

D Delete a stale test

- def test_missing_link(self):
-     msg = ('Add parent_link=True'
-            ' to ParkingLot.parent.')
-     with self.assertRaisesMessage(
-             ImproperlyConfigured, msg):
-         class ParkingLot(Place):
-             parent = OneToOneField(
-                 Place, CASCADE)
-         ParkingLot.check()

Its expected exception is no longer raised after the fix.

Create/Delete only, same issue

The reproduction script hand-rolls setup (DJANGO_SETTINGS_MODULE, django.setup(), app_label) and asserts the old ImproperlyConfigured, so it passes on both correct and incorrect patches.
The stale test_render is either deleted by name matching, losing a test that one edit would have made a strong discriminator, or left unchanged, so that it fails on the correct patch and passes on the incorrect ones.

How to readThe four boxes are the parts of one CRUD update. Blue lines in Read are the infrastructure that the created test reuses. The dashed box shows what a Create/Delete-only pipeline does on the same issue.

What it showsBefore the fix, this model raised ImproperlyConfigured; the fix turns that exception into accepted behavior, so a useful test must assert the new behavior, not reproduce the old crash. Reading the suite first makes the new test run inside Django's own test setup, and updating test_render keeps a high-signal test instead of discarding it or letting it penalize the correct patch.

03 · Evaluation

84.2% Pass@1 on SWE-bench Verified.

All selection methods receive the same candidate patches and differ only in the selection module. Pass@1 is the fraction of the 500 instances resolved under the SWE-bench harness after one patch is selected per instance. The single-agent baseline, Livesweagent alone, resolves 79.2%.

BenchmarkSWE-bench Verified

500 human-validated GitHub issues, run in the official SWE-bench Docker images.

Candidates6 leaderboard submissions

Livesweagent, Sonar, Trae_doubao, OpenHands, Gemini-Live, Atlassian, added in this order (N = 1…6).

BaselinesAugment · Agentless · TRAE

LLM judge without execution; Create/Delete test generation; Create/Delete plus repository browsing.

TestGRAD3 base LLMs as optimizer

Minimax-m2.7, Deepseek-v3.2, GLM-5.1; at most 8 optimization steps; minimum support 2.

RQ1 · overall effectiveness · 4-agent ensemble

Highest Pass@1 with each of the three base LLMs.

How to readEach panel uses one base LLM for every method, so within a panel only the selection method differs. The dashed line is the single-agent baseline; each stem runs from it to the method's Pass@1, and the small number is the gain over it.

What it showsTestGRAD reaches 84.2%, 83.6% and 84.0%, 4.4 to 5.0 points above the single agent and 3.4 to 3.6 points above the strongest baseline in each panel. TRAE and Agentless add at most 1.4 and 1.2 points. Augment, which judges without execution, does not improve on the single agent (−0.6 to ±0.0).

RQ2 · ensemble size N = 1…6

Ahead of TRAE at every size, best at N = 4.

TestGRADTRAEOracleExpected (random)AdversaryPass@1 (%)

How to readAgents are added one at a time in the order listed above. Oracle counts an instance as solved if any candidate is correct, Adversary only if every candidate is correct, and Expected is random selection; the grey band spans Oracle to Adversary. The blue column marks N = 4, the setting used in RQ1.

What it showsTestGRAD stays above TRAE for every N ≥ 2; with Minimax-m2.7 the gap grows from 1.2 points at N = 2 to 3.6 at N = 4. Pass@1 peaks at N = 4 and dips slightly at N = 5 and 6, when weaker agents join (Gemini-Live 77.4%, Atlassian 76.8%). At N = 4 the Oracle is 87.0%: a 2.8-point gap remains where a correct patch exists but the generated tests did not separate it.

RQ3 · ablation · Minimax-m2.7

Removing any component lowers Pass@1.

PartVariantsingle agent 79.2Δ pp

CompleteTestGRAD84.2—

CRUD gradientw/o Read · no test-suite understanding81.2−3.0

w/o Update · no in-place test updates81.8−2.4

w/o Read and Update79.4−4.8

Lossw/o differential loss · pass-count selection80.8−3.4

Momentumw/o Failure Pattern Momentum82.6−1.6

Optimizerw/o iterative optimization75.0−9.2

72768084

How to readEach row removes or replaces one part of TestGRAD; Δ is the drop in percentage points from the complete system, measured on the same instances. The dashed line marks the single-agent baseline (79.2%).

What it showsWithin the CRUD gradient, Read matters most: without it, generated tests fall back on bare imports and ad-hoc setups that often fail to run in the project's test runner. Without Update, tests that assert old expected values are deleted instead of corrected. Removing both leaves something close to a Create/Delete generator (−4.8). Pass-count selection instead of the differential loss costs 3.4 points, removing momentum 1.6, and removing iterative optimization altogether 9.2.

RQ4 · Failure Pattern Momentum · token cost

Momentum keeps the failure history compact.

Raw memory (all failed suites, verbatim)Failure Pattern Momentumavg. tokens, log scale

How to readAverage tokens of failure-history context per optimization iteration, on a log scale. Raw memory appends every failed test suite verbatim; momentum keeps only the mined frequent sequences. Italic labels are compression ratios (raw / momentum) at iterations 4 and 6, as labelled in the paper.

What it showsRaw memory keeps growing, to 16.4k tokens by iteration 6 with Deepseek-v3.2, while momentum stays between a few dozen and about 150 tokens. The ratio exceeds 100× with Deepseek-v3.2 and is 25–39× with GLM-5.1 and 33–34× with Minimax-m2.7. Removing momentum also lowers Pass@1, from 84.2% to 82.6% (RQ3).

Overhead · per instance · supplementary material

More tokens than the baselines; time close to TRAE.

	Avg. time (min)			Avg. tokens (K)
Method	MiniMax-m2.7	Deepseek-v3.2	GLM-5.1	MiniMax-m2.7	Deepseek-v3.2	GLM-5.1
Augment	1.3	1.8	1.4	2.8	3.2	2.9
Agentless	7.2	8.6	7.5	14.8	16.2	15.3
TRAE	19.3	26.7	20.8	318	412	334
TestGRAD	12.2	23.0	20.7	670	1,670	1,307

How to readAverage time and tokens per instance. TestGRAD values are measured from run logs; baseline values are estimated from conversation logs and API call traces.

What it showsTestGRAD uses the most tokens, because Read explores the repository and up to eight optimization steps may run. Its time is comparable to TRAE and shorter with MiniMax-m2.7 (12.2 vs. 19.3 min), since the loop stops as soon as a discriminative test is found.

04 · Cite

Citation

The paper is under review. Citation details will be added when it is public.