Check text-to-SQL execution accuracy

Use execution checks when two different SQL statements can answer the same question. EvalGate executes the reference and generated queries against the same database snapshot, compares their results, and reports execution accuracy. This command runs locally, requires no EvalGate credentials, and sends no SQL or fixture rows to EvalGate.

This capability requires Python on Linux or WSL. SQL execution fails closed on Windows and macOS because its hard worker memory boundary is Linux-specific. Parsing and row-comparison helpers remain portable.This capability is Python-only. The TypeScript SDK does not invoke Python on your behalf. Use an SDK build that advertises the sql command; source code and an installed package release are separate availability boundaries.

Prepare a fictional fixture

Use a dedicated fixture database with fictional or approved de-identified data. Avoid a production database. For example, create a SQLite fixture:

import sqlite3

with sqlite3.connect("fixture.db") as db:
    db.executescript("""
        CREATE TABLE accounts(id INTEGER PRIMARY KEY, balance REAL);
        CREATE TABLE transactions(id INTEGER PRIMARY KEY, account_id INTEGER, amount REAL);
        INSERT INTO accounts VALUES (1, 12.50), (2, 20.00);
        INSERT INTO transactions VALUES (1, 1, 10.00), (2, 1, 2.50), (3, 2, 20.00);
    """)

Write one case per line in sql-cases.jsonl:

{"id":"balances","question":"What is each account's total transaction amount?","generatedSql":"SELECT account_id, SUM(amount) FROM transactions GROUP BY account_id","referenceSql":"SELECT id, balance FROM accounts","floatTolerance":0.000001}
{"id":"account-order","question":"List account ids in ascending order.","generatedSql":"SELECT id FROM accounts ORDER BY id","referenceSql":"SELECT id FROM accounts ORDER BY id","orderMatters":true}

Each case requires a unique nonempty id, question, generatedSql, and referenceSql. orderMatters defaults to false; floatTolerance defaults to 0.000000001 and must be finite and nonnegative. Unknown fields, duplicate ids, invalid types, and empty files are refused. The command accepts at most 10,000 cases, a regular case file of at most 32 MiB, ids of at most 256 UTF-8 bytes, JSONL lines of at most 1 MiB, and 100,000 UTF-8 bytes per SQL statement. Inputs are read with a byte bound before JSON parsing; pipes and device files are refused. The reader streams LF-delimited lines within the byte limits; blank padding does not create a list of every line, and Unicode separators inside JSON values remain part of the value.

evalgate sql check --cases sql-cases.jsonl --db sqlite:///fixture.db --format json
evalgate sql check --cases sql-cases.jsonl --db sqlite:///fixture.db \
  --timeout-ms 5000 --max-rows 10000 --min-accuracy 0.95

sqlite:///fixture.db resolves relative to the working directory. Use sqlite:////absolute/path/fixture.db for an absolute path. The database must already exist; the checker never creates it.

Understand comparisons

Column names are ignored. Column count and position must match, including for empty result sets. By default, rows are compared as a multiset: order is ignored, duplicate rows retain their multiplicity, NULL stays NULL, and strings compare exactly. Finite floats may differ by at most the absolute floatTolerance; decimals retain exact decimal precision. Use orderMatters: true only when the reference query orders by a unique key or otherwise resolves every tie. Ordering on non-unique values alone does not make the database’s tie order deterministic. The report contains an outcome for every case:

Outcome	Meaning
`match`	Generated and reference results agree.
`mismatch`	Results differ in values, multiplicity, column count, or required order.
`generated_error`	Generated SQL could not be parsed or executed.
`reference_error`	The reference is invalid, refused, timed out, or exceeds the row cap. Fix the case.
`refused`	Generated SQL is outside the single read-only SELECT/CTE contract.
`timeout`	Generated execution or result comparison exceeded its deadline.
`too_many_rows`	Generated results exceed `--max-rows`.
`too_many_bytes`	Generated results exceed the per-value or aggregate byte budget.
`resource_limit`	The isolated worker hit its hard memory/CPU boundary or could not return a bounded report.

Execution accuracy is matches / (case count - reference errors). Reference errors are invalid evidence, and the gate stays inconclusive even if all other cases match. Refusal, timeout, row/byte limits and worker resource failures also keep the gate inconclusive, including when a low accuracy threshold was supplied. A file with no valid reference cases has unavailable accuracy. Reports omit SQL, questions, connection strings, database error text, and result rows by default. --show-diff explicitly includes at most five rows from each side for mismatches, at most twenty columns per row, with values clipped to bounded previews. Decimal, date/time, UUID and nested values carry type tags in the preview. A diff exceeding the worker report budget is omitted and marked diffTruncated. Across the entire job, the parent caps the serialized report at 16 MiB and drops additional optional diffs before retaining each case; diffTruncatedCaseCount records how many previews were dropped. Every completed case id and outcome stays in the report. If even the base outcomes cannot fit, the whole job returns WEAK_EVIDENCE with complete: false and no accuracy, rather than scoring a partial cohort. Those rows may contain fixture data; review their handling before saving or sharing a report. The report’s case ids are the ids you supplied.

Exit code	Symbol	Meaning
`0`	`PASS`	Accuracy meets the minimum and all reference cases are valid.
`1`	`SCORE_BELOW`	Accuracy is below `--min-accuracy` (default `1.0`).
`5`	`BAD_ARGS`	Invalid input, options, unsupported platform, missing SQLite file, or refused database configuration.
`7`	`WEAK_EVIDENCE`	Invalid reference, refused query, timeout, row/byte limit, or worker resource failure.
`11`	`IO_ERROR`	The PostgreSQL connection or infrastructure is unavailable.

PostgreSQL fixtures

Install the Python SDK’s sql extra to enable psycopg. Supply a dedicated fixture role with SELECT access only. The checker refuses superusers, elevated role attributes, table write grants, and membership in server-file or server-program roles. Configure the fixture role itself without write grants; the client check is an additional guard.

evalgate sql check --cases sql-cases.jsonl \
  --db 'postgresql://fixture_reader@localhost/fixture' --format json

Prefer PostgreSQL’s ordinary credential facilities, such as a password file, over putting passwords in shell history. The connection uses read-only session defaults and a read-only repeatable-read transaction per case. Named server cursors fetch one row at a time. statement_timeout bounds each server statement, and the worker also checks its deadline while fetching/comparing. Queries run with search_path=pg_catalog; unqualified fixture tables are validated and rewritten as quoted public tables. Use explicit schema qualification for other fixture schemas. Only a fixed set of pure built-in functions is supported, including common aggregates, math/string/date functions and JSON/array construction. Unknown functions, session-control functions and custom schema functions are refused. Views, foreign tables, row-level security, generated columns and custom-typed columns are unsupported because their definitions can hide function calls. Materialized views contain stored data and may be read when their types satisfy the same checks.

Execution boundaries

SQLite opens the file with URI mode=ro, sets query_only=ON, disables extension loading, and uses an authorizer that allows SELECT/read operations only. On Python versions with SQLite connection limits, individual SQLite values are capped at 1 MB. PostgreSQL uses server-enforced read-only transactions and a role without write grants. Both additionally parse SQL and refuse multiple statements, non-SELECT commands, write CTEs, SELECT INTO, locking queries, and non-allowlisted functions, custom operators/types and file/extension operations. --timeout-ms accepts 1–300,000; --max-rows accepts 1–100,000. Each query fetches at most the cap plus one row to detect overflow. Each value has a 1 MiB accounting budget, and each query has an 8 MiB aggregate result budget. SQLite also enforces its stricter 1 MB value limit on Python versions with connection limits. Every case executes in a fresh worker with a hard 512 MiB Linux address-space limit, a CPU-time ceiling, and disabled core dumps. Python starts in isolated mode so the bootstrap ignores the working directory and inherited PYTHONPATH; the verified SDK package root is added only after hard limits are installed. The parent enforces a wall-clock deadline of three query timeouts plus ten seconds for worker setup and returns a resource failure if the worker dies. Only a bounded report (maximum 128 KiB per case), never raw result sets, crosses the worker pipe. This protects the CLI process even when a driver buffers a single huge field before the value budget can be checked. These are fixture execution checks, not a general sandbox for hostile database extensions or untrusted database administrators. The worker limit does not bound the PostgreSQL server process or server-side extension memory; configure server resource limits and use a disposable database you control with trusted table definitions. The separate portable-quality query sandbox continues to refuse raw SQL. For SDK contributors, the ordinary Python test lane exercises SQLite and the PostgreSQL transport contract. Real PostgreSQL proofs run in a separate lane and require a disposable database with role/schema creation permissions:

EVALGATE_SQL_POSTGRES_URL='postgresql://postgres@localhost/fixture' \
  python -m pytest tests/sql_check_postgres.py

Run this from src/packages/sdk-python after installing the development and sql extras. The native lane creates a temporary fixture schema and a dedicated read-only role, removes both afterward, and fails if its database prerequisite is missing. It checks decimal precision, empty-result columns, server-enforced read-only behavior, role refusal, timeout, row caps, and SQL refusal.

Get Started

Core Concepts

Guides

SDK Reference

Platform

Check text-to-SQL execution accuracy

Prepare a fictional fixture

Understand comparisons

PostgreSQL fixtures

Execution boundaries

​Prepare a fictional fixture

​Understand comparisons

​PostgreSQL fixtures

​Execution boundaries

Prepare a fictional fixture

Understand comparisons

PostgreSQL fixtures

Execution boundaries