Methodology
Translation Benchmark
A reproducible test set for checking whether generated translations preserve observable behavior.
This benchmark is intentionally small and transparent. It is a starting point for regression checks, not evidence that every translation is correct.
Latest recorded run
Run on 13 August 2026 (Asia/Shanghai) against the configured openrouter/free route: 3 passed and 0 failed out of 3 cases.
| Case | Pair | Result | Duration |
|---|---|---|---|
| js-python-even-sum | JavaScript → Python | Pass | 6.1 s |
| python-typescript-user-lookup | Python → TypeScript | Pass | 20.4 s |
| english-python-slug | Natural Language → Python | Pass | 5.4 s |
What is measured
Each case defines a source language, a target language, a focused input snippet, and observable checks for the returned text. A run records the model route, time, raw output, and pass or fail result.
Current cases
- Sum even integers: JavaScript → Python
- Typed user lookup: Python → TypeScript
- Plain English slug helper: Natural Language → Python
How to reproduce it
- Clone the repository and install dependencies.
- Start the app with a configured server-side model key.
- Run
npm run benchmark -- --base-url http://localhost:3000. - Inspect every output; pattern checks are not a substitute for execution tests.
Known limitations
The provider route can select different models over time, generated output is nondeterministic, and text assertions cannot prove semantic equivalence. Behavioral execution tests require a separately isolated runner with strict resource limits.
The test cases, raw generated outputs, assertions, and runner are stored in the repository's benchmarks/ directory.