Plans and Recipes¶

Plans are concrete test specifications — they tell AAT exactly what to execute, what values to use, and what to assert. AAT supports two plan formats: recipes (compact, workflow-based) and full plans (explicit step-by-step). Both use the same aat run plan command.

Overview¶

Recipes are the recommended authoring format. A recipe names a workflow, selects slot options, and provides only the values and assertions that differ from the workflow defaults. AAT composes the full execution plan at load time. Most inputs don't need values in the recipe at all — graph nodes declare default value pools that provide varied, realistic data automatically on each run.

Full plans spell out every step, value, and assertion explicitly. They are used for complex tests that don't fit a workflow pattern.

Both formats are YAML files. AAT auto-detects which format to use based on the presence (recipe) or absence (full plan) of a kind: recipe field.

Recipes¶

What a Recipe Contains¶

A recipe has four top-level sections:

kind: recipe
metadata:
  created: 2026-02-22T10:35:58Z
  prompt: "place an order for express delivery to New York"
  graphVersion: "1.0.0"
selection:
  workflow: Create Order
  description: "Order with express shipping to New York"
  choices:
    payment: Credit Card
    shipping: Express
overrides:
  values:
    confirmOrder.shippingCity: "New York"

Section	Required	Purpose
`kind`	yes	Must be `"recipe"`
`metadata`	no	Provenance (creation time, originating prompt, graph version)
`selection`	yes	Workflow name, description, slot choices, addons, layers
`overrides`	no	Value, selection, and assertion overrides

Here is a recipe from an airline booking test suite:

kind: recipe
metadata:
  created: 2026-02-22T10:35:58Z
  prompt: book a round trip flight from nashville to seattle
  graphVersion: "1.0.0"
selection:
  workflow: Booking
  description: "Round-trip flight BNA→SEA with default cash payment"
  choices:
    payment: Cash
    trip-search: Round-Trip
overrides:
  values:
    searchRoundTrip.leg1Destination: SEA
    searchRoundTrip.leg1Origin: BNA
    searchRoundTrip.leg2Destination: BNA
    searchRoundTrip.leg2Origin: SEA

Workflow and Choices¶

The selection.workflow field names the base workflow to use. It must match a workflow declared in the graph.

selection.choices is a map of slot name to option name. Each entry selects which slot option fills the corresponding slot in the base workflow:

selection:
  workflow: Booking
  choices:
    trip-search: Round-Trip    # fills the trip-search slot
    payment: Card              # fills the payment slot

Omitted slots use the slot's default option (defined in the graph). See Workflows: Slots for how slot filling works.

Adding Addons¶

selection.addons lists addon workflow names to compose into the base workflow:

selection:
  workflow: Booking
  choices:
    payment: Cash
    trip-search: Round-Trip
  addons:
    - Document Overrides
    - Seat Selection
    - Retrieve Booking
    - Cancel Booking

Addons are composed in priority order as declared in the graph. See Workflows: Addons for composition mechanics.

Repeated Steps¶

A recipe has no repeat count. When a test needs several calls to the same node (two travelers, three cart items), the workflow offers them as slot options: each option's template lists the steps it needs under distinct ids, and an inject map on the option carries a count such as passengers: 2 to the other steps. The recipe then picks the option:

selection:
  workflow: Booking
  choices:
    travelers: Two Travelers

See Workflows: Repeated Steps. In a full plan, write the steps out with their own ids (see Step Identity).

Value Overrides¶

overrides.values provides literal values for specific step inputs, using stepId.inputName as the key:

overrides:
  values:
    searchProducts.category: "electronics"
    searchProducts.maxPrice: 500
    confirmOrder.shippingAddress: "123 Main St"

Override values replace whatever the workflow template or graph provides for that input: a literal, a pool, an injected value, or a reference. They are applied after composition, so they take effect on the final composed plan. An override on an input that the template wires with from, fromSelection, or fromInput (including a resolved AUTOWIRE) removes that wiring, so the step sends the override's value. Keys name the composed plan's step IDs, and addon steps carry an inc0_-style prefix (see Workflows: Step ID Prefixing). An override (value, selection, or assertion) for a step that is not in the composed plan is an error that lists the plan's steps.

Selection Overrides¶

overrides.selections overrides the selection strategy for a named selection in the composed plan:

overrides:
  selections:
    searchProducts.bestDeal:
      strategy: min
      sortField: price
      filter: "inStock == true"

Available fields:

Field	Description
`strategy`	Selection strategy (`first`, `last`, `index`, `random`, `min`, `max`, `match`)
`filter`	Predicate expression to narrow the array before selection
`sortField`	Field name for `min`/`max` comparison: a number, or a string that holds one
`index`	Element index for `index` strategy
`onTie`	For `min`/`max`, what a tie does: `first` takes the first of the elements that share the value without a warning, and `fail` fails the step

Assertion Overrides¶

overrides.assertions adds mechanical assertions to specific steps. These are added on top of any assertions the workflow template already defines:

overrides:
  assertions:
    confirmOrder:
      - type: status
        expect: 200
      - type: fieldExists
        path: "orderId"

Creating Recipes¶

Three paths to creating a recipe:

Write by hand, or have an AI coding tool write it — create a YAML file with kind: recipe and the selection/overrides you need. A tool connected to the MCP server can list the workflows, then validate and save the recipe for you.
Copy and modify — duplicate an existing recipe and change the parts that differ
aat prompt --save — describe what you want in natural language; the configured LLM drafts a recipe (see LLM-Assisted Planning)
```
aat prompt --save plans/my-test.yaml "order electronics with express shipping"
```

Running Recipes¶

Recipes use the same command as full plans:

aat run plan plans/my-test.yaml

AAT detects the kind: recipe field, runs the composition pipeline, and executes the result. You can also pass recipes to aat run batch for batch execution.

When to Use Recipes vs Full Plans¶

Aspect	Recipe	Full Plan
Size	10-30 lines	50-200+ lines
Authoring	Write overrides only	Write every step and value
Composition	Automatic via workflow	Manual — you define everything
Creation	Hand-write, copy and modify, an AI assistant through the MCP server, or `aat prompt --save`	Hand-write, or an AI assistant through the MCP server
Variation	Change choices/overrides, add layers	Duplicate and edit
Best for	Tests that follow a known pattern	Custom or generated test sequences

Start with recipes. Use full plans when your test doesn't fit any workflow pattern or when you need step-level control that recipes don't expose.

Full Plans¶

Plan Structure¶

A full plan has these top-level sections:

metadata:
  created: 2026-02-22T10:00:00Z
  prompt: "original prompt text"
  graphVersion: "1.0.0"

auth:
  type: bearer
  credentials:
    token: {source: env, var: API_TOKEN}

headers:
  X-Custom-Header: "test-run"

intent:
  goal: confirmOrder
  description: "Order a product, apply a discount code, and verify the final price"

execution:
  steps:
    # ... step definitions ...
  verification:
    # ... verification steps ...
  cleanup:
    # ... cleanup steps ...

Metadata¶

Metadata records provenance — when the plan was created and why:

Field	Type	Description
`created`	datetime	When the plan was generated or written
`prompt`	string	The natural language prompt that produced this plan (if generated by `aat prompt`)
`graphVersion`	string	Graph version at creation time

Intent¶

The intent section describes what the plan is trying to test. It does not change execution; the plan display, the web UI, and MCP tools show it:

Field	Type	Description
`goal`	string	Step ID of the plan's goal step. `aat validate plan` rejects a goal that names no step, or one that differs from the step marked `isGoal: true`
`description`	string	What the plan verifies, in prose; used as the plan's title

Steps¶

Steps are the core of a plan. Each step targets a graph node and specifies how to resolve its inputs.

Step Identity¶

steps:
  - node: searchProducts
    description: "Search for electronics"

  - id: searchProducts_2
    node: searchProducts
    description: "Second search with different criteria"

Field	Description
`node`	Graph node name (required, unless `slot` is set)
`id`	Optional unique identifier; defaults to `node` if omitted
`slot`	Slot marker name (exclusive with `node` — only used in workflow templates)
`description`	Human-readable step description
`isGoal`	Marks this step as the primary goal of the plan (at most one step; when `intent.goal` is set it must name this step)

When multiple steps target the same graph node, use id to give each a unique identifier. All references (dependsOn, from) use the step ID.

Dependencies¶

dependsOn lists step IDs that must complete before this step runs:

  - node: confirmOrder
    dependsOn: [addToCart, applyDiscount, addPayment]

A step that a value reads with from or fromInput, or that a named selection reads with from, joins dependsOn when the plan is instantiated, and so does a step a graph default's from reads. List only the ordering the data doesn't show. A reference that closes a cycle is reported with the value or selection that implies the dependency.

Values¶

The values map assigns inputs for the step. See Value Resolution for the full resolution chain. Here are the common forms:

Bare scalar — a literal value:

values:
  category: "electronics"
  maxPrice: 500
  departureDate: "{{today + 7 days}}"

Step output reference — pull a value from a previous step's output:

values:
  productId: {from: searchProducts.productId}

Reference with array selection — select from an array output:

values:
  productId:
    from: searchProducts.products
    select:
      strategy: min
      field: productId
      sortField: price
      filter: "inStock == true"

Named selection reference — reference a pre-resolved selection:

values:
  productId: {fromSelection: bestProduct.productId}
  productName: {fromSelection: bestProduct.name}

Intra-step reference — reference an input resolved earlier in the same step:

values:
  origin: "JFK"
  returnOrigin: {fromResolved: destination}
  returnDestination: {fromResolved: origin}

Cross-step input reference — reuse a resolved input from a previous step:

values:
  origin: {fromInput: searchFlights.origin}
  destination: {fromInput: searchFlights.destination}

This references the input value that was sent to the previous step, not its response output. Useful when multiple steps need the same input values (e.g., same origin city for search and booking).

Explicit absence — suppress auto-wiring for an optional input:

values:
  returnDate: {}

An empty map {} (or null) marks the input as explicitly absent: auto-wiring doesn't fill it, and an optional input is left out of the request even when a graph default or a layer sets it. Use it on optional inputs.

What {} does to a required input depends on its default: the graph default, with any layers applied: - A plain value is used, with its expressions evaluated, so a default of {{env.postalCode}} sends the variable's value. A layer that sets the input to a plain value sends that value. - A default with a pool, from, select, or a constraint isn't used, and the input is left out, as an optional one is. The template must send it inside a conditional block such as {{?couponCode}}…{{/couponCode}}, or the request fails on the unresolved placeholder. This suits a field sent only in some requests, such as a payment for an order paid now rather than later. - With no default, the step fails with required input has no value (empty step value).

Inputs you don't need to specify — graph nodes can declare default value pools on their inputs. When a plan or recipe doesn't provide a value for an input, the engine uses the graph default. For example, if the graph declares:

nodes:
  searchProducts:
    inputs:
      - name: category
        type: string
        default: ["electronics", "clothing", "books", "home"]
      - name: departureDate
        type: date
        default: ["{{today + 7 days}}", "{{today + 14 days}}", "{{today + 30 days}}"]

Then a plan step targeting searchProducts doesn't need to specify category or departureDate at all — the engine picks a random value from the pool on each run. This keeps plans small and produces varied test data automatically.

Plan values always take priority over graph defaults. When you need a specific value for a test, provide it in the plan; otherwise, let the pool handle it. See Value Resolution: Fallback Pools for details.

Named Selections¶

The selections block defines named array selections that multiple values can reference:

  - node: priceOffer
    dependsOn: [searchProducts]
    selections:
      bestProduct:
        from: searchProducts.products
        strategy: min
        sortField: price
        filter: "inStock == true"
    values:
      productId: {fromSelection: bestProduct.productId}
      productName: {fromSelection: bestProduct.name}
      productPrice: {fromSelection: bestProduct.price}

Named selections ensure coordinated multi-field extraction — all three values come from the same element, not from potentially different elements.

Field	Type	Required	Description
`from`	string	yes	`stepId.outputName` — the array to select from
`strategy`	string	no	Selection strategy (default: `first`)
`filter`	string	no	Predicate expression to narrow the array
`sortField`	string	no	Field for `min`/`max` comparison: a number, or a string that holds one
`index`	int	no	Element index for `index` strategy
`onTie`	string	no	For `min`/`max`, what a tie does: `first` takes the first tied element without a warning, and `fail` fails the step. Without it, the first is taken and the step warns

See Value Resolution: Array Selection for strategy details.

Assertions¶

Assertions validate the step's response. They are listed under mechanical (a bare list is accepted too) and evaluated by the engine:

Type	Fields	Description
`status`	`expect` (int or class)	HTTP status equals the code (`201`) or falls in the class (`2xx`, `4xx`)
`fieldExists`	`path` (string)	The path exists and is not null
`fieldEquals`	`path` (string), `value` (any)	The value at the path equals `value`
`predicate`	`expr` (string)	The predicate expression evaluates to true
`schema`	—	Validate the response body against the node's OAS response schema. Requires OAS specs wired into the graph; otherwise the assertion is reported as `skipped`.

Every mechanical assertion also accepts raw: true (see below).

assertions:
  mechanical:
    - type: status
      expect: 200
    - type: fieldExists
      path: "order.orderId"
    - type: fieldEquals
      path: "order.status"
      value: "confirmed"
    - type: schema
    - type: predicate
      expr: "order.totalPrice > 0 && order.totalPrice < 10000"

What path and expr see. By default, fieldExists, fieldEquals, and predicate are evaluated against the step's extracted outputs — the values the node's template pulled out of the response, keyed by output name — not the raw HTTP body. That keeps assertions stable when the API's envelope changes, and it means a Lua-transformed output is asserted in its transformed shape. Only on a response with status 400 or above, where nothing is extracted, does the check fall back to the raw body; a 2xx step whose template extracts nothing is checked against an empty object.

Set raw: true on an assertion to evaluate it against the raw response body instead. Use it for fields the template does not extract, or to assert on the envelope itself:

assertions:
  mechanical:
    - type: fieldEquals            # against extracted outputs
      path: "orderId"
      value: "ord-123"
    - type: fieldExists            # against the raw body
      path: "meta.requestId"
      raw: true
    - type: fieldEquals            # a gjson query into an array of the raw body
      path: 'data.items.#(sku=="ABC").price'
      value: 99.5
      raw: true

path is a gjson path, so fieldExists and fieldEquals can index arrays (items.0.id) and query them (items.#(sku=="ABC")); a leading $. and [0] bracket indexes are accepted too. A predicate expr is simpler: it reads dotted field names only, with no array indexes or queries.

Expressions in assertions. A fieldEquals value, and a quoted string in a predicate expr, can hold {{…}} expressions, such as value: "{{today + 3 days}}" or expr: 'quantity == "{{quantity}}"'. - They are evaluated when the step's assertions run, and they can name the step's inputs. today, now, and unixtime read the time the inputs were resolved, so a retried step's {{today}} is the date it resent. - A quoted expression that evaluates to a number or a boolean compares as one. - An expression that can't be evaluated fails its assertion. - aat validate checks their syntax. - Selection filters and cleanup when conditions don't evaluate expressions.

status and schema are unaffected by raw — they always look at the HTTP status and the full response body respectively.

Default status assertion. Steps composed from workflow templates (recipes, aat prompt) that declare no status assertion get status: 2xx, so APIs that answer 201 Created or 204 No Content pass. Steps with expectFailure get no default.

Status under expectFailure. When a step has expectFailure — declared in the plan or added by an overlay — its expectFailure.status list is the status check. A status assertion that expects success (an exact code below 400, or a 1xx–3xx class such as a composed 2xx default) can never hold there: aat validate plan rejects a plan that declares both, and when an overlay adds expectFailure at run time the assertion is reported as skipped. One that agrees with the expected failure, such as 409 or 4xx, is evaluated and can fail the step — useful to pin one code out of a broader expectFailure list. Other assertions still run against the error response.

Retry¶

Steps can configure retry behavior:

  - node: searchProducts
    retry:
      max: 3
      on: [transient, 503]
      failOn: [auth, 400]

Field	Description
`retry.max`	Maximum retry attempts
`retry.on`	Rules that trigger a retry — error category names and/or HTTP status codes
`retry.failOn`	Rules that cause immediate failure with no retry; checked before `on`

Each entry in on and failOn is either an error category name or an HTTP status code written as an integer. The two can be mixed freely: on: [503, transient] retries on any transient failure and on a bare 503. AAT classifies every failure into exactly one category:

Category	Covers
`transient`	HTTP 429, 502, 503, 504; connection refused or reset
`client`	Any other 4xx
`auth`	HTTP 401, 403
`server`	HTTP 500, 501, and any other 5xx not listed under `transient`
`timeout`	No response within aat's 30-second request timeout, or a context deadline exceeded
`network`	DNS and other connection-level errors
`adapter`	Template rendering, input resolution, or output extraction errors
`response_error`	A 2xx response whose body matched the graph's `errorDetection` rules

When on is omitted, the default retries transient, timeout, and server failures. A failOn match always wins, so failOn: [auth] stops the step on the first 401 even if on would otherwise retry it. Status codes must be in the range 100–599; aat validate plan rejects unknown category names and out-of-range codes rather than letting a typo silently disable retries.

Waiting between attempts. A retry waits an exponential backoff: about 500 ms before the first retry, doubling each time up to 10 seconds, with ±25% jitter.

The server can ask for longer. A retry waits at least as long as the failed response asks, in seconds or until an HTTP date. The request comes from the Retry-After header, or on a 429 without one, the RateLimit-Reset header.
Past 60 seconds, the retries stop. The step fails with the action failed_fast, and the error classification's detail says how long the server asked for.
Every wait counts toward the step's duration, and Ctrl+C interrupts it.

Every attempt sends the same inputs. A step's values are resolved once, before its first attempt, so a retry resends the same request: a pool pick, a date, or an overlay value doesn't change between attempts. A plan-level --retries rerun starts the plan again and resolves them anew.

To keep a rate-limited API from answering 429 in the first place, set settings.minRequestInterval.

Negative Testing (expectFailure)¶

Steps can declare that failure is the expected outcome:

  - node: createOrder
    description: "Attempt to create an order with invalid data"
    expectFailure:
      status: [400, 422]
      description: "Invalid order should be rejected"
    values:
      productId: "INVALID-ID"

When expectFailure is set:

The step passes if the response status matches one of the listed codes
The step fails if the response returns a success status (2xx)
Retries are skipped — the first response determines the outcome
Cleanup still runs normally

Mutations: codified negative suites¶

A mutations: block on a step expands at plan instantiation into one sibling step per mutation. Each sibling shares the parent's dependsOn, selections, and values, then applies its own set overrides and declares its own expectStatus. The parent step remains in place as the happy-path run, so a single block produces both the positive case and its negative variants.

- id: happy
  node: createOrder
  dependsOn: [setup]
  values: { productId: "PROD-001", quantity: 1 }
  assertions:
    mechanical:
      - type: status
        expect: 201
      - type: schema                   # validate response body shape
  mutations:
    - name: empty-productId
      set: { productId: "" }
      expectStatus: [400]
    - name: zero-quantity
      set: { quantity: 0 }
      expectStatus: [400, 422]
      description: "Server must reject zero-quantity orders"
    - name: malformed-body
      rawBody: '{"broken json'         # bypasses template substitution
      expectStatus: [400]

At runtime this produces four archive entries: happy, happy--empty-productId, happy--zero-quantity, and happy--malformed-body. The prereq chain (setup here) runs once; each mutation is an independent sibling step.

Mutation field	Required	Description
`name`	yes	Unique within the step. Becomes the sibling's id suffix.
`description`	no	Carried into the sibling's `expectFailure.description`.
`set`	no*	Map of input name → value. Overrides the parent's values for this sibling.
`rawBody`	no*	Raw request body string. Overwrites the adapter-built body after template substitution. Use for malformed payloads that can't be expressed via `set`.
`expectStatus`	yes	List of failure status codes (each `>= 400`).

*Each mutation must declare at least one of set or rawBody.

Expansion rules:

Siblings inherit the parent's dependsOn — prereqs run once, all siblings reference their outputs.
The parent's assertions are stripped on siblings; expectFailure is installed from expectStatus/description.
Sibling ids take the form <parentId>--<mutationName>.
Happy-path assertions are unchanged on the parent.

Shared vs. isolated prereqs¶

By default mutations share the parent's prereq chain: the prereqs run once and every sibling references the same outputs. That's correct for the common case where the server rejects a malformed request before touching state.

For stateful APIs, shared prereqs cause false failures: single-use tokens get consumed by the happy path, inventory is depleted, or the happy path commits a resource the mutations then collide with. Opt into isolation with mutationScope: isolated on the parent step:

- id: addItem
  node: addItem
  dependsOn: [createCart]
  mutationScope: isolated            # each mutation gets fresh prereqs
  values:
    cartId: { from: createCart.cartId }
    productId: "P1"
  mutations:
    - name: empty-productId
      set: { productId: "" }
      expectStatus: [400]
    - name: unknown-productId
      set: { productId: "NO-SUCH" }
      expectStatus: [404]

In isolated scope, AAT deep-clones the entire transitive prereq closure (login, createCart, anything they depend on) once per mutation. Cloned step ids are <origId>__<mutationName>; the mutation sibling's dependsOn and from refs are rewritten to point at the clones, so each sibling uses its own fresh cart, token, or other stateful resource.

Graph-level cleanup (node.cleanup: deleteX) runs automatically for every cloned step, so each isolated mutation cleans up the resources it created.

Scope	When to use	Cost
`shared` (default)	Input-validation errors; prereqs produce reusable state	Cheap — one prereq chain per plan
`isolated`	Stateful APIs, single-use tokens, consumable resources, duplicate-detection tests that would false-trigger on the happy path's state	Prereq chain runs once per mutation

A step may not declare mutationScope without mutations:, and an unknown scope value ("shared"/"isolated" only) is rejected at validation.

Smoke-test mode: skip mutations entirely¶

Pass --no-mutations to aat run plan or aat run batch to strip the mutations: blocks from plans before execution. The run exercises only the happy-path steps and their prereq chain — useful for quick CI smoke tests, local iteration, or confirming the plan itself is well-formed before paying the cost of the full negative suite (especially with mutationScope: isolated where each mutation re-runs the prereq chain).

aat run plan plans/createOrder-with-errors.yaml --no-mutations
aat run batch --no-mutations

The flag doesn't modify plan files on disk; it's applied in memory per run.

Raw request bodies¶

For one-off malformed-payload tests without a mutations: block, set rawBody directly on a step. When non-empty, the adapter-built body is overwritten at execution time, bypassing template placeholder substitution:

- node: createOrder
  rawBody: '{"oops": '
  expectFailure:
    status: [400]

Verification Steps¶

Verification steps run after the main execution flow to confirm side effects without modifying state:

execution:
  verification:
    - node: getOrder
      purpose: "Verify the order was created correctly"
      assertions:
        mechanical:
          - type: fieldEquals
            path: "order.status"
            value: "confirmed"

Field	Description
`node`	Graph node to execute
`purpose`	Human-readable description of what is being verified
`assertions`	Same assertion structure as main steps
`values`	Input values, as a main step takes them. A `from` or `fromInput` names a main step; a verification step has no selections

An input without a value takes its graph default. A default's from: node.output reads the last main step on that node that isn't expected to fail, so a check after a refused retry reads the request that succeeded. To read another step, set the value:

  verification:
    - node: getCharge
      values:
        charge: {from: firstRefund.charge}

Cleanup Steps¶

Cleanup steps run after the main steps finish — whether the plan passed, failed, errored, or was interrupted with Ctrl+C. The one exception is a --stop-after checkpoint, which deliberately skips cleanup so the created resources stay alive.

execution:
  cleanup:
    - node: cancelOrder
      runOn: always
    - node: sendNotification
      runOn: failure

`runOn` value	When cleanup runs
`always`	After every execution (default if omitted)
`success`	Only if the plan passed
`failure`	If the plan failed, errored, or was aborted

A cleanup step has just two fields: node and runOn. There is no values: block — inputs are filled by output-name matching: for each input the node declares, AAT looks for an output of the same name, first on the step that registered the resource, then on the most recent executed step that has one. cancelOrder with an orderId input picks up orderId from the createOrder step. Name your graph outputs to match the inputs of their teardown nodes and this needs no wiring at all.

Ordering. A graph-level cleanup: pairing (see API Graphs: Cleanup) runs once for each step that created a resource, from a last-in-first-out stack, so the most recently created resource is released first and nothing is sent for a resource that was never created. Listing a paired node here, as recipes and aat prompt plans do, does not run it a second time or change that order; the listed step's runOn decides whether those cleanups run. If the plan lists that node more than once, they run when any listing's runOn matches the outcome. Cleanup steps for other nodes, such as sendNotification above, run first, in declaration order. A node reached through a cleanup chain counts as a pairing too: listing it does not run it on its own, and its runOn decides whether the chain continues to it.

Skipped pairings. After runOn, a registered pairing is skipped when it's no longer needed: a later main step already released its resource, such as an explicit cancelOrder step for the order the pairing would cancel, or the pairing's when condition is false. See API Graphs: Cleanup.

Cleanup results are recorded in the archive and in the cleanup array of --json output, and appear under a cleanup: block in the console. Skipped pairings are recorded in the archive's cleanupSkipped and in cleanup_skipped in --json, and appear as skipped: lines under cleanup:. A cleanup failure never changes the run outcome. See Running Tests: Cleanup for the execution-time details.

Plan-Level Auth and Headers¶

Plans can override the environment's auth and headers:

auth:
  type: bearer
  credentials:
    token:
      source: env
      var: SPECIAL_TOKEN

headers:
  X-Api-Version: "v2"
  X-Test-Correlation-Id: "test-123"

Plan auth takes the same form as an environment's auth section: the types are oauth2, apikey, bearer, and none, and each credential is a secret reference (source: env with var, or source: literal with value), not a string. See Environments: Authentication.

Precedence¶

Plan auth replaces the environment's auth for the whole run, including overlay auth, and override entries that declare no auth of their own inherit it. Auth priority, lowest to highest: env.yaml auth, .aat-overrides.yaml auth, the --overlay file's auth, plan auth.

Headers merge in this order, later sources replacing earlier ones with the same name (in any case):

Environment headers
Plan headers
Template request.headers
The auth credential (Authorization, or the API key header)
.aat-overrides.yaml headers
--overlay file headers

So neither a plan header nor a template header can replace the credential, and an overlay header replaces everything before it. See Environments: Custom Headers for nodes routed by an override.

Layers¶

What Layers Are¶

Layers are data overlays that provide alternate input values without duplicating the entire plan. They are ideal for parameterized testing — the same plan logic with different data sets.

Layer File Format¶

A layer is a YAML file with a name, description, and an inputs map, kept in the directory that the manifest's layers: field names. Commands and recipes refer to a layer by its name, not by its file path. Keys are either nodeName.inputName (qualified) or bare inputName (applies to all nodes with that input):

# layers/european.yaml
name: european
description: European airport codes for intercontinental routes
inputs:
  searchFlights.origin: [CDG, LHR, FRA, AMS, FCO, MAD, BCN]
  searchFlights.destination: [CDG, LHR, FRA, AMS, FCO, MAD, BCN]

Layer input values use the same syntax as graph input defaults — bare scalars, pool lists, or rich objects with pools and references:

# layers/near-term.yaml
name: near-term
description: Near-term travel dates (2-5 days out)
inputs:
  departureDate:
    pool: ["{{today + 2 days}}", "{{today + 3 days}}", "{{today + 5 days}}"]
  returnDate:
    pool: ["{{today + 9 days}}", "{{today + 10 days}}", "{{today + 12 days}}"]

# layers/test-cards.yaml
name: test-cards
description: Test credit card numbers
inputs:
  addPayment.cardNumber:
    pool: ["4111111111111111", "5500000000000004", "340000000000009"]
  addPayment.cardCode: VISA

When the API expects an array parameter, use the explicit value: form to pass a literal array rather than a pool:

# layers/premium.yaml
name: premium
description: Constrain flight searches to premium cabins
inputs:
  cabinPreference:
    value: [Business, First]

Note the distinction: bare [A, B] is a pool (random pick at runtime), while value: [A, B] is a literal array value passed through as-is. The array is JSON-serialized at template rendering time (e.g., ["Business","First"]).

Applying Multiple Layers¶

When multiple layers are applied, they stack in order — later layers override earlier ones:

aat run plan my-test.yaml --layer european --layer near-term

Within a single layer, qualified entries (searchFlights.origin) take priority over bare entries (origin) for the same node.

Layer Groups and Permutations¶

Layer groups define sets of layers that expand into permutations during batch execution. Each --layer-group flag provides a comma-separated list of layer names. Each group also has a "none" choice, and multiple groups create a cartesian product:

aat run batch \
  --layer-group european,international \
  --layer-group near-term,far-out

This runs each plan nine times, (2 + 1) × (2 + 1): with no layer, with each of the four layers alone, and with european+near-term, european+far-out, international+near-term, and international+far-out. Permutations that produce the same execution are skipped as duplicates; see Matrix Testing with Layer Groups.

Using Layers in Recipes¶

Recipes can declare layers in the selection:

selection:
  workflow: Booking
  choices:
    trip-search: Round-Trip
    payment: Card
  layers:
    - european
    - near-term

Layers from the recipe are combined with any --layer flags from the command line.

Common Patterns¶

Minimal Recipe¶

The simplest recipe — just a workflow name:

kind: recipe
selection:
  workflow: Booking

All slots use defaults, no overrides, no addons.

Recipe with Overrides¶

Override specific inputs while keeping the rest of the workflow's defaults:

kind: recipe
selection:
  workflow: Create Order
  choices:
    payment: Credit Card
overrides:
  values:
    applyDiscount.discountCode: "SAVE10"
    confirmOrder.shippingCity: "Boston"
  assertions:
    confirmOrder:
      - type: status
        expect: 200

Full Plan with Negative Testing¶

Test that an invalid request is properly rejected:

execution:
  steps:
    - node: createOrder
      description: "Attempt order with invalid product ID"
      expectFailure:
        status: [400, 422]
        description: "Server should reject invalid product reference"
      values:
        productId: "NONEXISTENT"
        quantity: 1

Full Plan with a Mutations Suite¶

Exercise happy-path and negative variants of the same endpoint in one plan. The happy-path step runs first; each mutation expands into a sibling that shares the prereq chain:

execution:
  steps:
    - id: setup
      node: createBooking

    - id: happy
      node: addTraveler
      dependsOn: [setup]
      values:
        surname: "Smith"
        givenName: "Jane"
        age: 30
      assertions:
        mechanical:
          - type: status
            expect: 200
          - type: schema
      mutations:
        - name: empty-surname
          set: { surname: "" }
          expectStatus: [400]
        - name: negative-age
          set: { age: -1 }
          expectStatus: [400, 422]
        - name: malformed-body
          rawBody: '{"oops":'
          expectStatus: [400]

Overlay-Driven Depth Tests¶

For ad-hoc negative testing without editing a plan, use an overlay file to override individual input values and declare expected failure on matched nodes. The overlay applies at run time, so an existing happy-path plan can be rerun as a negative test just by layering in the overlay:

# .aat-overrides.yaml
overrides:
  - match: createOrder
    values:
      productId: ""
    expectFailure:
      status: [400]

See Environments: Overlay Files for the full schema, and Local Development for auto-discovery behaviour.

Layer Parameterization¶

Run the same test with different data sets:

# Single layer
aat run plan plans/booking.yaml --layer european

# Multiple layers stacked
aat run plan plans/booking.yaml --layer european --layer near-term

# Batch with layer group permutations: none, european, international
aat run batch --layer-group european,international

Validation¶

aat validate plan checks a plan (or recipe) for structural correctness:

All step node references exist in the graph
Required inputs have plan values
dependsOn references valid step IDs
from references point to valid step outputs
dependsOn has no cycle, counting the dependencies that references imply
Verification values name inputs of their node, and their references name main steps
Named selection from references are array types
Selection strategies are valid (first, last, index, random, min, max, match)
Filter and predicate expressions parse correctly
fromSelection references valid named selections
expectFailure status codes are 400+
Mutation names are unique within a step
Each mutation declares at least one of set or rawBody
Mutation expectStatus is non-empty and every entry is >= 400
mutationScope is either "shared" (default) or "isolated"
mutationScope is only set on steps that declare mutations:
Isolated-mutation clone ids don't collide with existing step ids
Cleanup runOn values are valid (always, success, failure)
Retry on/failOn entries are known categories or HTTP status codes (100–599)
Plan auth has a valid type and its required fields
intent.goal names a step, at most one step sets isGoal, and the two agree
No duplicate step IDs

aat validate plan --plan plans/my-test.yaml

When a manifest is discoverable, --graph can be omitted — the graph path resolves from aat-project.yaml. See Validation for the complete validation reference.

Recipe Schema Reference¶

Complete annotated recipe YAML:

# Required — identifies this file as a recipe
kind: recipe

# Optional — provenance tracking
metadata:
  created: 2026-02-22T10:00:00Z    # when this recipe was created
  prompt: "original prompt text"     # the prompt that generated it (if any)
  graphVersion: "1.0.0"             # graph version at creation time

# Required — workflow selection and composition inputs
selection:
  workflow: WorkflowName             # base workflow name (required)
  description: "What this test does" # human-readable description
  layers:                            # layer names to apply
    - european
    - near-term
  choices:                           # slot name → option name
    slotName: OptionName
  addons:                            # addon workflow names to compose
    - Addon One
    - Addon Two

# Optional — overrides applied after composition
overrides:
  values:                            # stepId.inputName → literal value
    stepId.inputName: value          #   replaces the input's wiring; stepId must be a step of the composed plan
  selections:                        # stepId.selectionName → strategy override
    stepId.selectionName:
      strategy: min                  # first, last, index, random, min, max, match
      filter: "predicate expression" # narrows array before strategy
      sortField: fieldName           # for min/max comparison
      index: 0                       # for index strategy
  assertions:                        # stepId → additional assertions
    stepId:
      - type: status
        expect: 200
      - type: fieldExists
        path: "response.path"
      - type: fieldEquals
        path: "response.path"
        value: expectedValue
      - type: predicate
        expr: "field > 0"

Full Plan Schema Reference¶

Complete annotated full plan YAML:

# Optional — provenance tracking
metadata:
  created: 2026-02-22T10:00:00Z
  prompt: "original prompt text"
  graphVersion: "1.0.0"

# Optional — override environment auth for this plan
auth:
  type: bearer                       # oauth2, apikey, bearer, none (same fields as env.yaml auth)
  credentials:
    token:                           # a secret reference, not a string
      source: env                    #   env (with var) or literal (with value)
      var: API_TOKEN

# Optional — additional headers for all steps
headers:
  X-Custom-Header: "value"

# Optional — test intent metadata
intent:
  goal: uniqueStepId                 # step ID of the goal step (the one with isGoal: true)
  description: "What this plan verifies"

# Required — the execution specification
execution:
  steps:
    - id: uniqueStepId               # optional; defaults to node name
      node: graphNodeName             # required (or slot: for templates)
      description: "What this step does"
      isGoal: true                    # marks as primary goal (matches intent.goal)
      dependsOn: [stepA, stepB]       # steps that must complete first; a step that from,
                                      #   fromInput, or a selection reads is added

      # Named selections from array outputs
      selections:
        selectionName:
          from: stepId.arrayOutput    # required — array source
          strategy: first             # first, last, index, random, min, max, match
          filter: "predicate"         # narrows array before strategy
          sortField: fieldName        # for min/max
          index: 0                    # for index strategy

      # Input values
      values:
        literalInput: "value"                           # bare scalar
        refInput: {from: stepId.outputName}             # step output reference
        selectedInput: {fromSelection: selName.field}   # named selection field
        resolvedInput: {fromResolved: otherInput}       # intra-step reference
        inputRef: {fromInput: stepId.inputName}         # cross-step input reference
        absentInput: {}                                 # explicitly skip
        exprInput: "{{today + 7 days}}"                 # dynamic expression
        poolInput:                                      # value with pool fallback
          default: "preferred"                          # used unless it fails the constraint
          pool: [alt1, alt2, alt3]                      # tried when there is no default, or it fails
          poolStrategy: random                          # random (default) or sequential
          constraint: "value != literalInput"           # predicate over value and earlier inputs

      # Post-step validation
      assertions:
        mechanical:
          - type: status
            expect: 200
          - type: fieldExists
            path: "response.field"
          - type: fieldEquals
            path: "response.field"
            value: "expected"
          - type: predicate
            expr: "field > 0 && field < 100"

      # Retry configuration
      retry:
        max: 3                        # maximum retry attempts
        on: [transient, server, 503]  # categories and/or HTTP status codes that trigger retry
        failOn: [auth, client]        # categories and/or status codes that fail immediately

      # Negative testing
      expectFailure:
        status: [400, 422]            # expected failure status codes
        description: "Why this should fail"

      # Optional — raw request body override, bypasses template substitution
      rawBody: '{"oops":'             # use for malformed-payload tests

      # Optional — mutation siblings for codified negative suites
      mutationScope: shared           # "shared" (default) or "isolated";
                                      #   isolated clones the prereq chain per mutation
      mutations:
        - name: empty-input           # required; unique within the step
          description: "why this variant should fail"
          set:                        # map of input → value; overrides parent values
            someInput: ""
          expectStatus: [400]         # required; each entry must be >= 400
        - name: malformed-body
          rawBody: '{"oops":'         # alternative to set; replaces the body entirely
          expectStatus: [400]

  # Optional — post-execution read-only checks
  verification:
    - node: graphNodeName
      purpose: "Verify side effects"
      values:                         # optional; as a step's, with references to main steps
        inputName: {from: stepId.outputName}
      assertions:
        mechanical:
          - type: fieldEquals
            path: "status"
            value: "confirmed"
          - type: fieldExists
            path: "meta.requestId"
            raw: true                 # evaluate against the raw body, not extracted outputs

  # Optional — cleanup runs after execution (inputs matched by output name; no values block)
  cleanup:
    - node: graphNodeName
      runOn: always                   # always, success, failure

Source: plan types in plan/types.go, recipe types in plan/recipe.go, validation in plan/validate.go, layers in graph/layer.go.