> ## Documentation Index
> Fetch the complete documentation index at: /docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Check text-to-SQL execution accuracy

> Compare generated SQL with reference results on a bounded, read-only SQLite or PostgreSQL fixture database.

Use execution checks when two different SQL statements can answer the same
question. EvalGate executes the reference and generated queries against the
same database snapshot, compares their results, and reports execution accuracy.
This command runs locally, requires no EvalGate credentials, and sends no SQL
or fixture rows to EvalGate.

> [!NOTE]
>
> This capability requires Python on Linux or WSL. SQL execution fails closed
> on Windows and macOS because its hard worker memory boundary is Linux-specific.
> Parsing and row-comparison helpers remain portable.
>
> This capability is Python-only. The TypeScript SDK does not invoke Python on
> your behalf. Use an SDK build that advertises the `sql` command; source code
> and an installed package release are separate availability boundaries.

## Prepare a fictional fixture

Use a dedicated fixture database with fictional or approved de-identified data.
Avoid a production database. For example, create a SQLite fixture:

```python theme={null}
import sqlite3

with sqlite3.connect("fixture.db") as db:
    db.executescript("""
        CREATE TABLE accounts(id INTEGER PRIMARY KEY, balance REAL);
        CREATE TABLE transactions(id INTEGER PRIMARY KEY, account_id INTEGER, amount REAL);
        INSERT INTO accounts VALUES (1, 12.50), (2, 20.00);
        INSERT INTO transactions VALUES (1, 1, 10.00), (2, 1, 2.50), (3, 2, 20.00);
    """)
```

Write one case per line in `sql-cases.jsonl`:

```jsonl theme={null}
{"id":"balances","question":"What is each account's total transaction amount?","generatedSql":"SELECT account_id, SUM(amount) FROM transactions GROUP BY account_id","referenceSql":"SELECT id, balance FROM accounts","floatTolerance":0.000001}
{"id":"account-order","question":"List account ids in ascending order.","generatedSql":"SELECT id FROM accounts ORDER BY id","referenceSql":"SELECT id FROM accounts ORDER BY id","orderMatters":true}
```

Each case requires a unique nonempty `id`, `question`, `generatedSql`, and
`referenceSql`. `orderMatters` defaults to `false`; `floatTolerance` defaults
to `0.000000001` and must be finite and nonnegative. Unknown fields, duplicate
ids, invalid types, and empty files are refused. The command accepts at most
10,000 cases, a regular case file of at most 32 MiB, ids of at most 256
UTF-8 bytes, JSONL lines of at most 1 MiB, and 100,000 UTF-8 bytes per SQL statement. Inputs are read with
a byte bound before JSON parsing; pipes and device files are refused.
The reader streams LF-delimited lines within the byte limits; blank padding
does not create a list of every line, and Unicode separators inside JSON values
remain part of the value.

```bash theme={null}
evalgate sql check --cases sql-cases.jsonl --db sqlite:///fixture.db --format json
evalgate sql check --cases sql-cases.jsonl --db sqlite:///fixture.db \
  --timeout-ms 5000 --max-rows 10000 --min-accuracy 0.95
```

`sqlite:///fixture.db` resolves relative to the working directory. Use
`sqlite:////absolute/path/fixture.db` for an absolute path. The database must
already exist; the checker never creates it.

## Understand comparisons

Column names are ignored. Column count and position must match, including for
empty result sets. By default, rows are compared as a multiset: order is
ignored, duplicate rows retain their multiplicity, `NULL` stays `NULL`, and
strings compare exactly. Finite floats may differ by at most the absolute
`floatTolerance`; decimals retain exact decimal precision.

Use `orderMatters: true` only when the reference query orders by a unique key
or otherwise resolves every tie. Ordering on non-unique values alone does not
make the database's tie order deterministic.

The report contains an outcome for every case:

| Outcome | Meaning |
| - | - |
| `match` | Generated and reference results agree. |
| `mismatch` | Results differ in values, multiplicity, column count, or required order. |
| `generated_error` | Generated SQL could not be parsed or executed. |
| `reference_error` | The reference is invalid, refused, timed out, or exceeds the row cap. Fix the case. |
| `refused` | Generated SQL is outside the single read-only SELECT/CTE contract. |
| `timeout` | Generated execution or result comparison exceeded its deadline. |
| `too_many_rows` | Generated results exceed `--max-rows`. |
| `too_many_bytes` | Generated results exceed the per-value or aggregate byte budget. |
| `resource_limit` | The isolated worker hit its hard memory/CPU boundary or could not return a bounded report. |

Execution accuracy is `matches / (case count - reference errors)`. Reference
errors are invalid evidence, and the gate stays inconclusive even if all other
cases match. Refusal, timeout, row/byte limits and worker resource failures also
keep the gate inconclusive, including when a low accuracy threshold was supplied.
A file with no valid reference cases has unavailable accuracy.

Reports omit SQL, questions, connection strings, database error text, and result
rows by default. `--show-diff` explicitly includes at most five rows from each
side for mismatches, at most twenty columns per row, with values clipped to
bounded previews. Decimal, date/time, UUID and nested values carry type tags
in the preview. A diff exceeding the worker report budget is omitted and
marked `diffTruncated`. Across the entire job, the parent caps the serialized
report at 16 MiB and drops additional optional diffs before retaining each
case; `diffTruncatedCaseCount` records how many previews were dropped. Every
completed case id and outcome stays in the report. If even the base outcomes
cannot fit, the whole job returns `WEAK_EVIDENCE` with `complete: false` and no
accuracy, rather than scoring a partial cohort. Those rows
may contain fixture data; review their handling before saving or sharing a
report. The report's case ids are the ids you supplied.

| Exit code | Symbol | Meaning |
| - | - | - |
| `0` | `PASS` | Accuracy meets the minimum and all reference cases are valid. |
| `1` | `SCORE_BELOW` | Accuracy is below `--min-accuracy` (default `1.0`). |
| `5` | `BAD_ARGS` | Invalid input, options, unsupported platform, missing SQLite file, or refused database configuration. |
| `7` | `WEAK_EVIDENCE` | Invalid reference, refused query, timeout, row/byte limit, or worker resource failure. |
| `11` | `IO_ERROR` | The PostgreSQL connection or infrastructure is unavailable. |

## PostgreSQL fixtures

Install the Python SDK's `sql` extra to enable `psycopg`. Supply a dedicated
fixture role with SELECT access only. The checker refuses superusers, elevated
role attributes, table write grants, and membership in server-file or
server-program roles. Configure the fixture role itself without write grants;
the client check is an additional guard.

```bash theme={null}
evalgate sql check --cases sql-cases.jsonl \
  --db 'postgresql://fixture_reader@localhost/fixture' --format json
```

Prefer PostgreSQL's ordinary credential facilities, such as a password file,
over putting passwords in shell history. The connection uses read-only session
defaults and a read-only repeatable-read transaction per case. Named server
cursors fetch one row at a time. `statement_timeout` bounds each server
statement, and the worker also checks its deadline while fetching/comparing.

Queries run with `search_path=pg_catalog`; unqualified fixture tables are
validated and rewritten as quoted `public` tables. Use explicit schema
qualification for other fixture schemas. Only a fixed set of pure built-in
functions is supported, including common aggregates, math/string/date functions
and JSON/array construction. Unknown functions, session-control functions and
custom schema functions are refused. Views, foreign tables, row-level security,
generated columns and custom-typed columns are unsupported because their
definitions can hide function calls. Materialized views contain stored data
and may be read when their types satisfy the same checks.

## Execution boundaries

SQLite opens the file with URI `mode=ro`, sets `query_only=ON`, disables extension
loading, and uses an authorizer that allows SELECT/read operations only. On
Python versions with SQLite connection limits, individual SQLite values are
capped at 1 MB. PostgreSQL uses server-enforced read-only transactions and a
role without write grants. Both additionally parse SQL and refuse multiple
statements, non-SELECT commands, write CTEs, SELECT INTO, locking queries, and
non-allowlisted functions, custom operators/types and file/extension operations.

`--timeout-ms` accepts 1–300,000; `--max-rows` accepts 1–100,000. Each query fetches
at most the cap plus one row to detect overflow. Each value has a 1 MiB accounting
budget, and each query has an 8 MiB aggregate result budget. SQLite also enforces
its stricter 1 MB value limit on Python versions with connection limits.

Every case executes in a fresh worker with a hard 512 MiB Linux address-space
limit, a CPU-time ceiling, and disabled core dumps. Python starts in isolated
mode so the bootstrap ignores the working directory
and inherited `PYTHONPATH`; the verified SDK package root is added only after
hard limits are installed. The parent enforces a
wall-clock deadline of three query timeouts plus ten seconds for worker setup
and returns a resource failure if the worker dies. Only a bounded report
(maximum 128 KiB per case), never raw result sets, crosses the worker pipe.
This protects the CLI process even when a driver buffers a single huge field
before the value budget can be checked.

These are fixture execution
checks, not a general sandbox for hostile database extensions or untrusted
database administrators. The worker limit does not bound the PostgreSQL server
process or server-side extension memory; configure server resource limits and
use a disposable database you control with trusted table definitions. The separate
portable-quality query sandbox continues to refuse raw SQL.

For SDK contributors, the ordinary Python test lane exercises SQLite and the
PostgreSQL transport contract. Real PostgreSQL proofs run in a separate lane
and require a disposable database with role/schema creation permissions:

```bash theme={null}
EVALGATE_SQL_POSTGRES_URL='postgresql://postgres@localhost/fixture' \
  python -m pytest tests/sql_check_postgres.py
```

Run this from `src/packages/sdk-python` after installing the development and
`sql` extras. The native lane creates a temporary fixture schema and a dedicated
read-only role, removes both afterward, and fails if its database prerequisite
is missing. It checks decimal precision, empty-result columns, server-enforced
read-only behavior, role refusal, timeout, row caps, and SQL refusal.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.