Benchmarks and recorded runs

Batch e2c01df3-a5b9-58d2-bc9b-f51d996be91c. Fifty distinct task IDs, one recorded attempt each, no cancelled runs or terminal model-request failures. All saved grader verdicts and required-test counts reconcile with the outcome records. No benchmark was rerun for this review.

Recorded configuration: OpenRouter; model ID anthropic/claude-sonnet-5.5; provider route anthropic; medium thinking. These are recorded route labels, not independent verification of the provider’s underlying model identity. Each run records a 3,000-second and 400-tool-call limit. Task commits and image digests are in the run details.

The batch records $5.03 in estimated model usage. This excludes AWS infrastructure and is not an invoice. The task selection method and how representative this sample is have not been independently established. Treat this as a selected internal sample, rather than an official score or model comparison.

The benchmark grader tests the generated patch against the task’s expected outcome. In-task checks are the checks Rovara Code records and compares during the coding session.

This batch has separate grader evidence for all 50 tasks. Its in-task check records show 48 not verified and 2 verified. Both tasks with verified in-task checks were unresolved by the benchmark grader. A check can pass while the requested fix remains incomplete.

Five unresolved tasks contain an agent success claim. Their grader evidence still records failure. Inspect the saved output rather than treating that claim as the result.

Required tests are the task’s bug and regression groups combined. Full task IDs are in the downloadable outcomes.

Raw output excerpts. Environment setup and database logs are omitted here; the downloadable output is unchanged.

test_help_text (auth_tests.test_validators.UserAttributeSimilarityValidatorTest) ... ok
test_validate (auth_tests.test_validators.UserAttributeSimilarityValidatorTest) ... ok
test_validate_property (auth_tests.test_validators.UserAttributeSimilarityValidatorTest) ... ok
test_help_text (auth_tests.test_validators.CommonPasswordValidatorTest) ... ok
test_validate (auth_tests.test_validators.CommonPasswordValidatorTest) ... ok
test_validate_custom_list (auth_tests.test_validators.CommonPasswordValidatorTest) ... ok
test_validate_django_supplied_file (auth_tests.test_validators.CommonPasswordValidatorTest) ... ok
test_help_text (auth_tests.test_validators.MinimumLengthValidatorTest) ... ok
test_validate (auth_tests.test_validators.MinimumLengthValidatorTest) ... ok
test_help_text (auth_tests.test_validators.NumericPasswordValidatorTest) ... ok
test_validate (auth_tests.test_validators.NumericPasswordValidatorTest) ... ok
test_empty_password_validator_help_text_html (auth_tests.test_validators.PasswordValidationTest) ... ok
test_get_default_password_validators (auth_tests.test_validators.PasswordValidationTest) ... ok
test_get_password_validators_custom (auth_tests.test_validators.PasswordValidationTest) ... ok
test_password_changed (auth_tests.test_validators.PasswordValidationTest) ... ok
test_password_changed_with_custom_validator (auth_tests.test_validators.PasswordValidationTest) ... ok
test_password_validators_help_text_html (auth_tests.test_validators.PasswordValidationTest) ... ok
test_password_validators_help_text_html_escaping (auth_tests.test_validators.PasswordValidationTest) ... ok
test_password_validators_help_texts (auth_tests.test_validators.PasswordValidationTest) ... ok
test_validate_password (auth_tests.test_validators.PasswordValidationTest) ... ok
test_ascii_validator (auth_tests.test_validators.UsernameValidatorsTests) ... ok
test_unicode_validator (auth_tests.test_validators.UsernameValidatorsTests) ... ok

----------------------------------------------------------------------
Ran 22 tests in 0.047s

OK

>>>>> Test Exit Code: 0

Task: django__django-11099. Run: d13ed785-404c-56fc-8787-e95f88170d65. Recorded session began 30 September 2026 at 23:22 UTC, using Claude Sonnet 4.6 through Amazon Bedrock.

The patch, test output, and grader report are unchanged copies of saved local benchmark artifacts, inspected on 5 October. Their SHA-256 hashes are recorded. The request is a summary derived from the recorded prompt. The prompt itself suggested the regex-anchor fix and instructed the agent to edit source files only; the evaluation harness supplied the tests.

The grader’s FAIL_TO_PASS and PASS_TO_PASS groups describe benchmark-defined expectations. They are not a fresh before/after run by this website reviewer. No new evaluation was run here.

This is one selected historical task, not an aggregate solve rate, a model comparison, the full Django test suite, customer proof, or qualification of the current runtime verification loop. It does not show the live Slack-to-PR workflow. The benchmark generated a patch, not an Rovara Code pull request.

Django’s historical upstream PR provides problem context; it was not submitted by Rovara Code. Django is not a customer or endorser. Django license · Record notes.

“Finished” counts terminal task × model × repeat slots. “Graded” excludes model-request and execution failures. A stopped batch can have planned tasks that never started. A batch with no finished runs has no measured solve rate. Retried attempts are retained separately.