React Native Evals is our benchmark suite for testing how coding models handle React Native tasks. A solver is the model being benchmarked: it attempts the task. A judge is the model that checks the generated files against the task's requirements. We keep the files, scores, logs, and run settings so we can trace and compare the results.
We started by running the benchmark by hand. We connected models through OpenCode, adjusted configuration and credentials, started each run, watched for failures, and collected the output ourselves. That worked with a small model list. Once it grew, each release added more runs to manage and more chances to confuse a model regression with a provider problem or an operator mistake.
We now coordinate the process with Apache Airflow, a platform for defining and running workflows made of ordered tasks. A human operator selects the models, eval scope, and repeat count once. Airflow validates that request, starts one worker for each solver/judge pair, and reports each pair's progress and failures separately.
MLflow Tracking keeps the run settings, metrics, and output files connected to the result that produced them. The React Native Evals runner acts as the evaluation harness: it invokes the solver and judge through OpenCode, an AI coding agent. People still check the analyzer output before anything is published.
The workflow follows six steps:
- An operator selects the models, evals, and repeat count.
- Airflow schedules and monitors the selected solver/judge pairs.
- The harness runs the solver, then the judge checks the output.
- MLflow keeps the run configuration and artifacts together when tracking is enabled.
- Reviewers verify the summaries and analyzer output.
- Verified artifacts feed the public benchmark results.
-1600w.avif)
Stage 1: choose the run
Operators usually start the workflow from the Airflow user interface. They can also send the same request through the Airflow API.
The operator chooses one judge, one or more solvers, the evals to run, and the repeat count. Operators can enter supported model IDs when a run starts, so testing a new supported model does not require an automation code change or an edit to a YAML model matrix. Operators can run the full benchmark repeatedly or choose a smaller task set for a smoke test.
The request records the exact Evals commit, meaning the specific version of the Evals repository used for that benchmark, along with the solver and judge models, selected tasks, repeat count, and runtime settings. Keeping those inputs with the outputs lets us compare like-for-like runs and reproduce the setup later. The Evals commit pins the benchmark code and task definitions to one concrete repository version; model outputs can still vary between repeats.
Stage 2: solve and judge each task
Airflow starts a separate containerized worker for each selected solver/judge pair. Each worker keeps its workspace, logs, and failures separate. If a provider times out or a model returns malformed output, the operator can inspect that pair instead of searching through one combined log.
The React Native Evals runner is the evaluation harness. It invokes both models through OpenCode. During the solver stage, the harness hides unselected evals, reference solutions, and judge-only requirements. It restores the requirements before judging, while the reference solution remains unavailable.
Each pair completes its configured repeats in sequence. Airflow records each worker's state and marks the workflow successful only when all required pairs finish.
Stage 3: keep the run record and files together
Workers are short-lived, so we need a durable record of their outputs after execution ends. We use MLflow Tracking to keep each model pair's configuration, status, metrics, generated files, scores, and summaries together for comparison and review.
Each solver/judge pair has its own MLflow experiment, which keeps all runs for that pair together. Every Airflow workflow execution creates one tracking run for each selected pair, and that record contains the configured repeats. Airflow checks that each expected repeat produced its output before it marks the workflow successful.
MLflow is enabled by default, but the runner can operate without it. If tracking is disabled, the runner can use a separate export for the normalized run files.
Stage 4: review and publish the result
After the configured repeats succeed with MLflow enabled, the runner builds a dated archive in the format expected by the React Native Evals results analyzer. Reviewers can download it, combine it with other result sources when needed, and process it without rebuilding the folder structure by hand.
We inspect the generated summaries and analyzer output before committing any result files used by the public benchmark. Human review remains the publication gate.
What changed for the operator
One request now replaces repeated setup and monitoring for each solver/judge pair. Automation handles execution, judging, progress reporting, logs, and artifact packaging. People focus on checking whether the result is valid.
Adding a supported model now requires its model ID, a judge, and an eval scope rather than a code change or model-matrix edit. In current team practice, adding a supported model and completing its evals can take about a day instead of taking about a week with the manual setup. This comparison only covers completed evals. It does not include review or publication.
Because the benchmark runs as an Airflow workflow, we can also schedule it or trigger it from a specific external event. Human review remains the final gate before results reach the public benchmark.
The latest reviewed React Native Evals results are available at rn-evals.vercel.app.
