Methodology
Translation Benchmark
A reproducible test set for checking whether generated translations preserve observable behavior.
This benchmark is intentionally small and transparent. It is a starting point for regression checks, not evidence that every translation is correct.
Latest recorded run
Run on 13 August 2026 (Asia/Shanghai) against the configured openrouter/free route: 3 passed and 0 failed out of 3 cases.
| Case | Pair | Result | Duration |
|---|---|---|---|
| js-python-even-sum | JavaScript → Python | Pass | 6.1 s |
| python-typescript-user-lookup | Python → TypeScript | Pass | 20.4 s |
| english-python-slug | Natural Language → Python | Pass | 5.4 s |
What is measured
Each case defines a source language, a target language, a focused input snippet, and observable checks for the returned text. A run records the model route, time, raw output, and pass or fail result.
Current cases
- Sum even integers: JavaScript → Python
- Typed user lookup: Python → TypeScript
- Plain English slug helper: Natural Language → Python
How to reproduce it
- Download the current public benchmark JSON, including cases, checks, raw outputs, durations, and provider route.
- Run each source snippet and translated result in an isolated test environment.
- Compare observable output, error behavior, and edge cases defined by each case. Text pattern checks are not execution tests.
- Record the date and provider route when comparing future runs.
Known limitations
The provider route can select different models over time, generated output is nondeterministic, and text assertions cannot prove semantic equivalence. Behavioral execution tests require a separately isolated runner with strict resource limits.
The public JSON snapshot makes the current evidence inspectable without requiring access to a source repository.