Synthetic evaluation setEvidence
Each scenario is fictional or intentionally generic. Together they show a supported answer and the three ways this concept declines to give one — invalid input, abstention, and escalation to human review. No live request is sent.
Supported answerDraft a reviewable implementation note from approved fictional evidence
Demonstrates drafting from an approved evidence set with an explicit review gate.
Request
{
"example_id": "controlled-drafting-agent-happy-path",
"requirements": [
{
"id": "REQ-1",
"text": "Describe the planned review step."
}
],
"approved_evidence": [
{
"evidence_id": "SYN-EVID-001",
"text": "A named reviewer checks each draft before external release.",
"source_label": "Example Workflow Note"
}
],
"drafting_instructions": {
"format": "short implementation note",
"tone": "plain language"
},
"prohibited_claim_rules": [
"Do not claim production availability.",
"Do not add commitments not present in approved evidence."
]
}
Response
{
"example_id": "controlled-drafting-agent-happy-path",
"status": "draft_ready_for_review",
"draft_sections": [
{
"section_id": "SEC-1",
"text": "Each draft is checked by a named reviewer before it is released externally.",
"evidence_links": [
"SYN-EVID-001"
]
}
],
"unsupported_claim_flags": [],
"unresolved_requirements": [],
"approval_state": "review_pending",
"external_export_allowed": false
}
AbstainsFlag a service-level claim that is not in the approved evidence
Demonstrates abstention on unsupported commercial commitments.
Request
{
"example_id": "controlled-drafting-agent-abstention",
"requirements": [
{
"id": "REQ-1",
"text": "State the promised response time."
}
],
"approved_evidence": [
{
"evidence_id": "SYN-EVID-002",
"text": "The team discusses support expectations during pilot scoping.",
"source_label": "Example Pilot Note"
}
],
"drafting_instructions": {
"format": "one sentence"
},
"prohibited_claim_rules": [
"Do not invent an SLA or response time."
]
}
Response
{
"example_id": "controlled-drafting-agent-abstention",
"status": "completed_with_unresolved_requirements",
"draft_sections": [],
"unsupported_claim_flags": [
{
"requirement_id": "REQ-1",
"reason": "The approved evidence does not state a response-time commitment."
}
],
"unresolved_requirements": [
"Confirm whether an approved response-time commitment exists."
],
"approval_state": "blocked"
}
Human reviewBlock external release until a named approver reviews the draft
Shows the approval point that remains outside automation.
Request
{
"example_id": "controlled-drafting-agent-human-review",
"requirements": [
{
"id": "REQ-1",
"text": "Prepare a client-facing delivery statement."
}
],
"approved_evidence": [
{
"evidence_id": "SYN-EVID-003",
"text": "The proposed pilot scope will be confirmed after technical and privacy review.",
"source_label": "Example Scope Note"
}
],
"drafting_instructions": {
"format": "client-facing statement"
},
"prohibited_claim_rules": [
"Do not represent proposed scope as approved."
]
}
Response
{
"example_id": "controlled-drafting-agent-human-review",
"status": "review_required",
"draft_sections": [
{
"section_id": "SEC-1",
"text": "The proposed pilot scope remains subject to technical and privacy review.",
"evidence_links": [
"SYN-EVID-003"
]
}
],
"unsupported_claim_flags": [],
"unresolved_requirements": [
"Assign the named approver."
],
"approval_state": "named_approver_required",
"external_export_allowed": false
}
What this record does not yet state
These facts have no value yet. They are listed rather than hidden, so the gap is visible before anyone relies on the record.
| Fact | Group |
|---|
| Named owner | Lifecycle |
| Operational contract review | Lifecycle |
| Last review date | Lifecycle |
| Storage | Trust |
| Retention | Trust |
| Logging | Trust |
| Deletion | Trust |
| Allowed tools | Technical |
| Memory and persistence | Technical |