Capability & release status

The Python source line is at 3.15.0; that version is an unpublished candidate, not a release.

• Understand the product before writing evals

  localMaturity not recorded in contract

  Build and confirm an evidence-backed product theory before drafting evaluation cases.

  No web route · 4 CLI commands · 1 governed operations

  Understand the product before writing evals evidence: No evidence recorded

• Set up a project safely

  hybridMaturity not recorded in contract

  Establish durable organization/repository/root linkage, pin cloud evidence to exact commits, and verify local and cloud readiness separately.

  Web: /setup · 12 CLI commands · 2 governed operations

  Set up a project safely evidence: No evidence recorded

• Run and gate evaluations

  hybridMaturity not recorded in contract

  Execute behavioral specifications, inspect durable runs, and block regressions.

  Web: /evaluations · 2 CLI commands · 0 governed operations

  Run and gate evaluations evidence: No evidence recorded

• Author and compare behavior

  cloudMaturity not recorded in contract

  Import or generate cases, run variants with live progress, compare calibrated evidence, and promote a winner.

  Web: /playgrounds · 3 CLI commands · 0 governed operations

  Author and compare behavior evidence: No evidence recorded

• Govern prompt versions

  cloudMaturity not recorded in contract

  Create, review, approve, publish, resolve, and roll back immutable prompt versions.

  Web: /prompts · 6 CLI commands · 0 governed operations

  Govern prompt versions evidence: No evidence recorded

• Curate governed datasets

  hybridMaturity not recorded in contract

  Create, import, version, publish, and bind durable regression datasets.

  Web: /datasets · 2 CLI commands · 0 governed operations

  Curate governed datasets evidence: No evidence recorded

• Install opinionated domain coverage

  cloudMaturity not recorded in contract

  Discover detailed first-party domain scenarios, inspect criterion-level gates, and atomically install a governed dataset and evaluation.

  Web: /eval-packs · 2 CLI commands · 0 governed operations

  Install opinionated domain coverage evidence: No evidence recorded

• Run reproducible experiments

  cloudMaturity not recorded in contract

  Execute immutable variants, compare exact results, decide, and publish the winner.

  Web: /experiments · 1 CLI commands · 0 governed operations

  Run reproducible experiments evidence: No evidence recorded

• Develop governed scorers

  cloudMaturity not recorded in contract

  Test, calibrate, review, approve, publish, and bind exact scorer versions.

  Web: /measurement · 1 CLI commands · 0 governed operations

  Develop governed scorers evidence: No evidence recorded

• Investigate production traces

  cloudMaturity not recorded in contract

  Query bounded trace evidence and turn failures into reviews, candidates, monitors, datasets, or Playground cases.

  Web: /evidence · 1 CLI commands · 0 governed operations

  Investigate production traces evidence: No evidence recorded

• Monitor production continuously

  cloudMaturity not recorded in contract

  Version monitors, simulate, activate, backfill, triage alerts, and verify remediation evidence.

  Web: /online-evals · 1 CLI commands · 0 governed operations

  Monitor production continuously evidence: No evidence recorded

• Cluster and remediate failure topics

  cloudMaturity not recorded in contract

  Inspect, label, split, merge, accept, and convert production failure topics into governed work.

  Web: /insights · 1 CLI commands · 0 governed operations

  Cluster and remediate failure topics evidence: No evidence recorded

• Attribute a failure and prove the repair holds

  cloudMaturity not recorded in contract

  Attribute a recurring production failure to the harness component that caused it, measure single-component repairs against a sealed holdout, adopt one, and leave a release gate that blocks its reintroduction.

  Web: /improvement · 6 CLI commands · 0 governed operations

  Attribute a failure and prove the repair holds evidence: No evidence recorded

• Improve a repository feature locally and confirm it independently

  hybridMaturity not recorded in contract

  Optimize one repository prompt on an isolated branch under an approved Optimization Contract, freeze the candidate, import the campaign, confirm it on reserved cases through the customer's own runners, and hand a person the report and draft pull request. The program-driven loop is TypeScript CLI only; Python reaches the hosted import and confirmation through the generic API command.

  Web: /change-assessments/import, /change-assessments · 10 CLI commands · 0 governed operations

  Improve a repository feature locally and confirm it independently evidence: No evidence recorded

• Operate evidence dashboards

  cloudMaturity not recorded in contract

  Author, publish, render, share, and inspect dashboards whose widgets fail independently.

  Web: /dashboards · 1 CLI commands · 0 governed operations

  Operate evidence dashboards evidence: No evidence recorded

• Discover the repository AI stack

  cloudMaturity not recorded in contract

  Scan one exact commit, retain protected detector evidence, construct an immutable AI system graph, and recommend missing evaluation packs.

  Web: /repository-intelligence · 4 CLI commands · 2 governed operations

  Discover the repository AI stack evidence: No evidence recorded

• Use bounded evidence Copilot

  cloudMaturity not recorded in contract

  Assemble redacted evidence, answer dataset questions from bounded server-side queries with their scope and row citations, propose reviewable actions (including new draft datasets and evaluations), and apply only authorized revisions without blocking manual work.

  Web: /copilot · 1 CLI commands · 0 governed operations

  Use bounded evidence Copilot evidence: No evidence recorded

• Configure governed model routing

  cloudMaturity not recorded in contract

  Create encrypted gateway configurations, check health, discover models, and explain routing before execution.

  Web: /settings · 1 CLI commands · 0 governed operations

  Configure governed model routing evidence: No evidence recorded

• Run governed adversarial testing

  cloudMaturity not recorded in contract

  Version risk packs and campaigns, review attacks, triage findings, verify remediation, and sign evidence.

  Web: /red-team · 3 CLI commands · 0 governed operations

  Run governed adversarial testing evidence: No evidence recorded

• Execute your real feature on signed remote runners

  cloudMaturity not recorded in contract

  Register an executable target for a repository commit and entrypoint, serve it from your machine or CI, dispatch signed jobs, stream evidence with the served model, usage and cost provenance, recover leases, and accept one exact completion.

  Web: /remote-runners · 2 CLI commands · 0 governed operations

  Execute your real feature on signed remote runners evidence: No evidence recorded

• Roll out and roll back exact assets

  cloudMaturity not recorded in contract

  Approve exact versions, invoke them, compare canary health, promote, pause, and atomically roll back.

  Web: /releases · 1 CLI commands · 0 governed operations

  Roll out and roll back exact assets evidence: No evidence recorded

• Inspect workflow governance

  cloudMaturity not recorded in contract

  Inspect integrity, policy controls, retention, and the state of the governed evaluation loop.

  Web: /eval-loop · 2 CLI commands · 0 governed operations

  Inspect workflow governance evidence: No evidence recorded

• Ship and verify evidence

  hybridMaturity not recorded in contract

  Synchronize reviewed state, make an evidence-backed release decision, and verify the promoted boundary.

  Web: /releases · 2 CLI commands · 1 governed operations

  Ship and verify evidence evidence: No evidence recorded

• OpenInference over OTLP/HTTP

  Offline fixture and contract qualifiedNot live-qualifiedReleased

  openinference-otlp v1.0.0 · otlp_http · 3 offline fixture test files

  • Needs a live export from an openinference-instrumentation-* package against a staging organization.

  Live qualification evidence

  OpenInference over OTLP/HTTP live qualification evidence: No evidence recorded

• OpenTelemetry GenAI semantic conventions over OTLP/HTTP

  Offline fixture and contract qualifiedNot live-qualifiedReleased

  otel-genai-otlp v1.0.0 · otlp_http · 2 offline fixture test files

  • Needs a live GenAI-instrumented export against a staging organization.

  Live qualification evidence

  OpenTelemetry GenAI semantic conventions over OTLP/HTTP live qualification evidence: No evidence recorded

• LangChain structural tracing wrapper

  Offline fixture and contract qualifiedNot live-qualifiedReleased

  langchain-sdk-wrapper v3.14.1 · sdk_in_process · 2 offline fixture test files

  • Fake-framework tests only; no live LangChain run recorded.

  Live qualification evidence

  LangChain structural tracing wrapper live qualification evidence: No evidence recorded

• CrewAI kickoff tracing wrapper

  Offline fixture and contract qualifiedNot live-qualifiedReleased

  crewai-sdk-wrapper v3.14.1 · sdk_in_process · 1 offline fixture test files

  • Fake-framework tests only; a live crew run needs a model provider key.

  Live qualification evidence

  CrewAI kickoff tracing wrapper live qualification evidence: No evidence recorded

• AutoGen AgentChat and initiate_chat tracing wrapper

  Offline fixture and contract qualifiedNot live-qualifiedReleased

  autogen-sdk-wrapper v3.14.1 · sdk_in_process · 2 offline fixture test files

  • AgentChat support is unreleased until the next SDK release.
  • Tested against the real autogen-agentchat 0.7.5 runtime with a scripted ReplayChatCompletionClient only; no live model provider run recorded.

  Live qualification evidence

  AutoGen AgentChat and initiate_chat tracing wrapper live qualification evidence: No evidence recorded

• Vercel AI SDK Language Model v3 wrapper

  Offline fixture and contract qualifiedNot live-qualifiedReleased

  vercel-ai-model-wrapper v3.14.1 · sdk_in_process · 1 offline fixture test files

  • No live provider run recorded.

  Live qualification evidence

  Vercel AI SDK Language Model v3 wrapper live qualification evidence: No evidence recorded

• Native coding-agent metrics exports (Codex, Claude Code, Cursor, CoWork, generic JSONL)

  Offline fixture and contract qualifiedNot live-qualifiedReleased

  native-harness-exports v3.14.1 · file_import · 1 offline fixture test files

  • No client writes the export natively yet.

  Live qualification evidence

  Native coding-agent metrics exports (Codex, Claude Code, Cursor, CoWork, generic JSONL) live qualification evidence: No evidence recorded

• Harness² run importer

  Offline fixture and contract qualifiedNot live-qualifiedUnreleased

  harness2-runs v0.1.0 · file_import · 3 offline fixture test files

  • Run a Harness² sweep (Vertex AI or Codex credentials) and import its namespace into a staging organization.

  Live qualification evidence

  Harness² run importer live qualification evidence: No evidence recorded

• GEPA optimize_anything evaluator and candidate recorder

  Offline fixture and contract qualifiedNot live-qualifiedUnreleased

  gepa-optimize-anything v0.1.0 · sdk_in_process · 1 offline fixture test files

  • Run optimize_anything with a reflection LM key against a staging organization.

  Live qualification evidence

  GEPA optimize_anything evaluator and candidate recorder live qualification evidence: No evidence recorded

• DSPy GEPA feedback metric

  Offline fixture and contract qualifiedNot live-qualifiedUnreleased

  dspy-gepa-metric v0.1.0 · sdk_in_process · 1 offline fixture test files

  • Run dspy.GEPA with an LM key against a staging organization.

  Live qualification evidence

  DSPy GEPA feedback metric live qualification evidence: No evidence recorded

• review

  Shipped8 tools

  Released read-only review surface; actual calls require organization authentication and scopes.

  project.inspect, project.plan, evaluations.list, runs.list, run.explain, baseline.lineage, docs.search, connector.health

  MCP review evidence: No evidence recorded

• connections

  Shipped9 tools

  Read tools need eval:read; every connection mutation stays admin-only by role and admin:workspace scope.

  connections.list, connections.catalog, connections.create_root (write), connections.roots, connections.repair_root (write), connections.set_root_state (write), connections.graph, connections.backfill (write), connections.select_resources (write)

  MCP connections evidence: No evidence recorded

• portable-quality

  Not publicly released11 tools

  Backend, authorization and hosted workflow acceptance required before MCP release.

  quality.contexts.list, quality.context.create (write), quality.resources.list, quality.plans.list, quality.plan.create (write), quality.check.submit (write), quality.check.get, quality.check.attest_evidence (write), quality.standards.resolve, quality.standard.propose (write), quality.grant.create (write)

  MCP portable-quality evidence: No evidence recorded

• authoring

  Shipped4 tools

  Write tools replay the caller credential through the REST routes that own each operation, so quotas, locks, idempotency, rate limits and audit apply unchanged.

  evaluations.create (write), test_cases.add (write), runs.start (write), traces.create (write)

  MCP authoring evidence: No evidence recorded