> ## Documentation Index
> Fetch the complete documentation index at: https://evalgate.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Annotations

# Annotations API — human labeling and review

> Create annotation tasks, assign traces for human review, and submit labels to build the golden dataset used for measuring LLM judge credibility.

Human annotations are the foundation of judge credibility in EvalGate. When you label a set of traces as pass or fail, those labels become the ground truth that the [LLM Judge alignment endpoint](/docs/api/llm-judge) compares against automated judge scores. A judge with high alignment against a well-labeled dataset is one you can trust to gate your CI pipeline.

## GET /api/annotations/tasks — list annotation tasks

Returns annotation tasks for the authenticated organization.

```bash theme={null} theme={null}
curl https://evalgate.com/api/annotations/tasks \
  -H "Authorization: Bearer YOUR_API_KEY"
```

### Response

```json theme={null} theme={null}
{
  "tasks": [
    {
      "id": 12,
      "name": "Support quality review — March",
      "status": "in_progress",
      "itemCount": 120,
      "completedCount": 87,
      "organizationId": "00000000-0000-4000-8000-000000000001",
      "createdAt": "2026-03-01T09:00:00.000Z"
    }
  ]
}
```

<ResponseField name="tasks" type="array">
  <Expandable title="Task fields">
    <ResponseField name="id" type="integer">Unique task ID.</ResponseField>
    <ResponseField name="name" type="string">Display name for the task.</ResponseField>
    <ResponseField name="status" type="string">Task status: `draft`, `in_progress`, or `completed`.</ResponseField>
    <ResponseField name="itemCount" type="integer">Total number of items (traces) assigned to this task.</ResponseField>
    <ResponseField name="completedCount" type="integer">Number of items that have received a label.</ResponseField>
    <ResponseField name="organizationId" type="string">Owning organization UUID.</ResponseField>
    <ResponseField name="createdAt" type="string">ISO 8601 creation timestamp.</ResponseField>
  </Expandable>
</ResponseField>

***

## POST /api/annotations/tasks — create an annotation task

Creates a new task and assigns a set of traces for labeling.

```bash theme={null} theme={null}
curl https://evalgate.com/api/annotations/tasks \
  -X POST \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Support quality review — March",
    "traceIds": [42, 43, 44, 45]
  }'
```

### Request body

<ParamField body="name" type="string" required>
  Display name for this annotation task.
</ParamField>

<ParamField body="traceIds" type="array" required>
  Array of numeric trace IDs to include in this task. Each trace will become one annotation item.
</ParamField>

<ParamField body="instructions" type="string">
  Optional guidance text shown to annotators when they open the task.
</ParamField>

<ParamField body="labelOptions" type="array">
  Optional array of label strings annotators can choose from. Defaults to `["pass", "fail"]` when not specified.
</ParamField>

### Response (201)

```json theme={null} theme={null}
{
  "id": 13,
  "name": "Support quality review — March",
  "status": "draft",
  "itemCount": 4,
  "completedCount": 0,
  "organizationId": "00000000-0000-4000-8000-000000000001",
  "createdAt": "2026-03-15T11:30:00.000Z"
}
```

***

## GET /api/annotations/tasks/{id} — get task details

Returns a single annotation task with its items.

```bash theme={null} theme={null}
curl https://evalgate.com/api/annotations/tasks/12 \
  -H "Authorization: Bearer YOUR_API_KEY"
```

### Path parameters

<ParamField path="id" type="integer" required>
  Numeric ID of the annotation task.
</ParamField>

### Response

```json theme={null} theme={null}
{
  "id": 12,
  "name": "Support quality review — March",
  "status": "in_progress",
  "items": [
    {
      "id": 201,
      "traceId": 42,
      "label": "pass",
      "notes": "Clear and complete response",
      "labeledAt": "2026-03-10T14:22:00.000Z",
      "labeledBy": "user@example.com"
    },
    {
      "id": 202,
      "traceId": 43,
      "label": null,
      "notes": null,
      "labeledAt": null,
      "labeledBy": null
    }
  ]
}
```

<ResponseField name="items" type="array">
  <Expandable title="Item fields">
    <ResponseField name="id" type="integer">Unique item ID.</ResponseField>
    <ResponseField name="traceId" type="integer">ID of the trace this item references.</ResponseField>
    <ResponseField name="label" type="string | null">The label assigned by the annotator. `null` if not yet labeled.</ResponseField>
    <ResponseField name="notes" type="string | null">Optional free-text notes from the annotator.</ResponseField>
    <ResponseField name="labeledAt" type="string | null">ISO 8601 timestamp when the label was submitted. `null` if not yet labeled.</ResponseField>
    <ResponseField name="labeledBy" type="string | null">Email or identifier of the annotator who submitted the label.</ResponseField>
  </Expandable>
</ResponseField>

***

## POST /api/annotations/tasks/{id}/items — submit an annotation

Submits a label for a single annotation item within a task.

```bash theme={null} theme={null}
curl https://evalgate.com/api/annotations/tasks/12/items \
  -X POST \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "itemId": 202,
    "label": "fail",
    "notes": "Response did not address the user'\''s core question"
  }'
```

### Path parameters

<ParamField path="id" type="integer" required>
  Numeric ID of the annotation task.
</ParamField>

### Request body

<ParamField body="itemId" type="integer" required>
  Numeric ID of the annotation item to label.
</ParamField>

<ParamField body="label" type="string" required>
  The label to assign. Must be one of the task's configured `labelOptions`, or `pass` / `fail` by default.
</ParamField>

<ParamField body="notes" type="string">
  Optional free-text notes explaining the label decision. These are stored alongside the label for audit and inter-rater review.
</ParamField>

### Response

```json theme={null} theme={null}
{
  "id": 202,
  "traceId": 43,
  "label": "fail",
  "notes": "Response did not address the user's core question",
  "labeledAt": "2026-03-15T12:05:00.000Z",
  "labeledBy": "user@example.com"
}
```

<Note>
  Once a task has enough labels, run the [LLM Judge alignment check](/docs/api/llm-judge) to measure how well your automated judge agrees with your team's ground truth. A high-alignment judge is safe to use as an automated CI gate.
</Note>