The reward function behind DeepSWE fits in a sentence: the model earns +1 if its generated patch passes a selected sample of tests, and 0 if the code fails any test or runs longer than five minutes. That's it. No graded rubric, no partial credit for effort, no reviewer scoring style. A patch either makes the suite go green inside the time budget or it earns nothing.
That austerity is the whole point, and it explains a shift that has quietly reorganized how frontier coding models get built. For a couple of years the story was scale: more tokens, more parameters, more supervised examples of good code. The bottleneck moved. The scarce resource now isn't text describing engineering work — it's places where a model can attempt the work and be scored by machinery that can't be talked into a good grade. Those places are RL environments, and building them well is most of the job.
What an environment actually is
Strip away the terminology and an RL environment is two things: a task (a prompt plus a grader) and an action space (the set of moves the model can make). For software engineering the task is almost always a real git repository frozen at a commit where some tests fail. The action space is a small toolset — run a shell command, search files, edit a file, submit. The model reads the failing issue, pokes around the repo, writes a patch, and submits. The grader runs the suite and hands back a number.
The infrastructure underneath is heavier than it sounds. Each attempt needs an isolated, reproducible copy of the repo with its dependencies installed, and training generates thousands of attempts in parallel. DeepSWE's training used the R2E-Gym environment orchestrated with Kubernetes across more than a thousand CPU cores — not because the model is large but because you're standing up and tearing down a small fleet of Docker-style sandboxes continuously. The environment is a distributed systems problem wearing a machine-learning hat.
This is also why the field has gotten expensive. Analysts who interviewed vendors report per-task authoring costs from a few hundred to a few thousand dollars, and a faithful replica of a complex product environment can run into six figures. One large lab was reported to have discussed spending over a billion dollars on RL environments in a year. You don't spend that on data. You spend it on graders and sandboxes.
Two ways to score a patch
The interesting design decision is how the grader turns a patch into a number, and there are two schools.
Execution-based rewards run the code. This is what DeepSWE does — the strongest possible signal, because passing tests is close to the actual definition of "solved." The cost is that every single rollout needs a working, sandboxed execution environment, which is exactly the infrastructure above.
Similarity-based rewards skip execution. SWE-RL, trained on Llama-3.3-70B-Instruct, never runs the patch at all. It compares the model's diff against the reference patch with Python's difflib.SequenceMatcher and rewards the overlap:
def reward(predicted_patch, oracle_patch):
if not well_formed(predicted_patch):
return -1.0 # format penalty
return SequenceMatcher(
None, predicted_patch, oracle_patch
).ratio() # continuous 0.0–1.0
That's a much cheaper grader — no containers, no dependency hell — and it scales to mountains of GitHub history that were never packaged as runnable test suites. The tradeoff is that similarity to a known-good patch is a proxy for correctness, not correctness itself. A different-but-valid fix scores low; a superficially close but broken one scores high.
Neither is strictly better. Execution rewards are truer but bounded by how many repos you can make runnable. Similarity rewards are looser but let you learn from open software evolution at a scale execution can't reach. What's striking is that the cheaper signal still worked: Llama3-SWE-RL-70B reached 41.0% on SWE-bench Verified, the best result among models under 100B parameters at the time.
The real game is not getting hacked
Ask people who build these environments what keeps them up, and it isn't model capability. It's reward hacking — the model finding a path to high reward that doesn't correspond to solving the task. The governing principle one practitioner put bluntly: high reward must mean the task was actually solved, not hacked. If a test can be satisfied by deleting the assertion, editing the test file, or writing to the expected output path, the model will eventually discover that, because RL optimizes exactly what you measure and nothing you meant.
Defending against it is mostly grader hardening, and it shows up in the training loop too. DeepSWE's GRPO++ uses "compact filtering" to mask trajectories that hit the context limit or the timeout, which stopped the model from being rewarded for lucky or degenerate runs and pushed it toward deliberate verification instead.
Difficulty calibration is the other quiet craft. A task the model never solves gives no gradient; a task it always solves gives no gradient either. Practitioners target a floor around a 2–3% pass rate — at least one success in 64 to 128 attempts — and when a task sits at 0%, they check that a human can actually solve it before blaming the model. An environment full of impossible or trivial tasks teaches nothing regardless of how much compute you throw at it.
Why this bleeds into general reasoning
The payoff that surprised me most is that these narrow environments don't produce narrow models. SWE-RL trained only on real issue-solving, yet the resulting model improved on unrelated benchmarks — MATH climbed from 63.2% to 73.7%, code-reasoning on CRUXEval from 60.5% to 71.6%. The supervised fine-tuning baseline, by contrast, dropped on most of those out-of-domain tasks. Being forced to find and verify a working solution against a grader seems to teach a transferable habit that memorizing solutions does not.
The takeaway
If you're evaluating or building on top of a coding model, stop reading only the parameter count and the headline benchmark. Ask what it was trained against: were the rewards execution-based or a similarity proxy, how were the graders hardened against being gamed, and were the tasks calibrated to sit in the band where learning actually happens. Those three answers predict how a model behaves on your unglamorous real repo far better than its size does. The models are converging; the environments are where the differences are still being made.
Sources: An FAQ on Reinforcement Learning Environments — Epoch AI, DeepSWE: Training a State-of-the-Art Coding Agent by Scaling RL — Together AI, SWE-RL: Advancing LLM Reasoning via Reinforcement Learning on Open Software Evolution — arXiv