Journey Tests

Private beta

Journey tests are in private beta. Contact support@turn.io to request access.

Journey tests let you verify a journey's behaviour automatically instead of replaying conversations by hand in the simulator. Tests are written in Lua, live alongside the journey, and run against the journey's saved version, the same one the simulator uses.

Open the tests panel from the journey canvas using the tests button in the toolbar. The editor saves as you type, and every test and suite has its own play button.

Writing a test

Declare tests with test(name, fn). Inside a test, simulation.start begins a conversation with the journey and returns a session handle:

test("journey greets and asks for a name", function()
  local sim = simulation.start({})
  assert.contains(sim.text, "Welcome")
  sim:send("Hi")
  assert(sim.waiting, "expected the journey to ask a question")
end)

simulation.start accepts options:

local sim = simulation.start({
  contact = { name = "Pat", language = "eng" },
})

contact sets profile fields on the simulated contact before the journey starts.

The session handle

sim:send(text) plays one contact message, refreshes the handle and returns it (so sim = sim:send(...) works too). After start or any send, the handle exposes:

Field	Meaning
`sim.text`	The reply the contact received in this turn
`sim.state`	`"waiting_for_input"` or `"end"`
`sim.waiting`	`true` while the journey waits for a contact message
`sim.ended`	`true` once the journey has finished
`sim.contact`	The simulated contact, including profile fields the journey has set
`sim.card`	The card the journey is currently paused on
`sim.skill_calls`	The skills an AI agent called during this turn, in order (see below)
`sim.actions`	The actions the agent routed to during this turn, in order
`sim.action`	The last of `sim.actions`, or `nil`

sim.text reads like the chat. A card with buttons or a list ends with an Options: line naming the choices, and sim:send picks one by its exact text. A location request takes coordinates (sim:send("-33.9249, 18.4241")), a contact info request takes any reply as the contact sharing their number (declined plays a reply without one, already_shared a number already on file), and a call permission request takes accept or reject. Scenario personas follow the same conventions.

For AI agent conversations, where the number of turns is not fixed, sim:keep_replying(text, max_turns, stop_fn) sends the same message while the journey keeps asking for input, up to max_turns (default 10). The optional stop_fn(sim) ends the loop early when it returns a truthy value:

test("agent eventually fills the summary field", function()
  local sim = simulation.start({})
  sim:send("Hello")
  sim:keep_replying("That is everything I know.", 6, function(s)
    return s.contact.summary ~= nil
  end)
  assert.truthy(sim.contact.summary)
end)

Agent skills and actions

When a turn runs through an AI agent card, sim.skill_calls lists every skill the agent called before it replied, in the order they happened. Each entry has the skill's name, the arguments the agent passed, and the output the skill returned. sim:skill_called(name) returns the first call of that skill, or nil:

test("agent checks the order before answering", function()
  local sim = simulation.start({})
  sim:send("Where is my order 4711?")

  local call = assert.skill_called(sim, "lookup_order")
  assert.eq(call.arguments.order_id, "4711")
  assert.contains(sim.text, "shipped")
end)

Knowledge skills carry their content themselves, so agents consult them without arguments:

test("knowledge skills are consulted without arguments", function()
  local sim = simulation.start({})
  sim:send("When are you open?")

  local hours = assert.skill_called(sim, "opening_hours")
  assert.eq(next(hours.arguments), nil)
  assert.contains(hours.output, "Mon-Fri")
end)

sim.actions lists the actions the agent routed to during the turn — the same ids the card's exits match on, including end_conversation. A turn that passes through several agent blocks can route more than once, so it's a list; sim.action is the last of them, or nil while the conversation continues:

test("a booking request routes to the booking action", function()
  local sim = simulation.start({})
  sim:send("Book me in for Tuesday, please")

  assert.action_called(sim, "book_appointment")
  assert.eq(sim.action, "book_appointment")
end)

test("the agent hangs up politely on goodbye", function()
  local sim = simulation.start({})
  sim:send("That's all, thanks, bye!")

  assert.action_called(sim, "end_conversation")
  assert(sim.ended)
end)

Both only cover the last turn. In a sim:keep_replying loop, stop once the call you expect shows up:

sim:keep_replying("Yes, go ahead.", 6, function(s)
  return s:skill_called("check_availability") ~= nil
end)
assert.skill_called(sim, "check_availability")

Skill calls and routed actions also appear in the results pane, between the message a test sent and the journey's reply, so a failing assertion can be read against what the agent actually did.

Mocking HTTP calls

Function skills (and app functions) call out with turn.http.request. In a test you usually do not want the real API: it may not exist yet, it is slow, and it answers differently every time. Give the simulation mocks and the Lua HTTP API answers from them instead:

test("looks the order up", function()
  local sim = simulation.start({
    http_mocks = {
      { method = "GET", url = "https://api.example.com/orders/*",
        respond = function(req)
          return { status = 200, body = { status = "shipped", eta = "tomorrow" } }
        end },
    },
  })
  sim = sim:send("Where is my order 4711?")

  local call = assert.skill_called(sim, "lookup_order")
  assert.contains(call.output, "shipped")
end)

Each mock is a Lua function. It gets the request (method, url, headers, body, json, query — with the URL's own query string merged in — and params) and returns the response:

status defaults to 200. body can be a string or a table, which is sent as JSON. headers is an optional table. Returning nothing means an empty 200.
json is body decoded when it parses as JSON, so a mock can check req.json.day without parsing the string itself; otherwise it is nil.
url is matched against the full URL; * matches anything, and a :name placeholder matches one segment and hands its captured value to the function as req.params.name. method is optional; without it the mock answers any method. A mock can also be just the bare function, which answers every request.
The skill receives exactly what a real call gives it: local body, status, headers = turn.http.request(...).
A request that no mock matches goes to the real API and is recorded like any other call, with mocked = false, so a test can mix a mocked API with a real one. To catch stray calls instead, pass fail_unmocked_http_calls = true to simulation.start or scenario.run: such a request then raises an error inside the skill naming the mocks that exist.
sim:mock_http(mock) adds a mock to a running simulation, and scenario.run takes the same http_mocks option.

The function is a real closure over the test's scope, so it can check the request it answers with the same assert the test uses. A failed assert (or an error(...)) inside it answers the request with an error and fails the test at the end of that turn with the reason, even if the skill swallowed the error:

http_mocks = {
  { method = "GET", url = "https://api.example.com/orders/:account/sent/:order_id",
    respond = function(req)
      assert.eq(req.params.account, "acme-7")
      assert.eq(req.params.order_id, "4711")
      assert.eq(req.headers["X-Api-Key"], "demo-key")
      return { body = { status = "shipped" } }
    end },
}

A secret the skill obtained with turn.secrets.get never reaches the test. The mock and sim.http_calls show it as <secret:name> wherever the skill put it, so a test can assert which secret was sent, and where, without knowing its value:

assert.eq(req.headers["X-Api-Key"], "<secret:orders_api_key>")

Matching mocks answer in list order, one call each, and the last one keeps answering after that. A single mock therefore answers every call, and listing two mocks for the same URL scripts a sequence — a journey that checks availability twice can see free slots first and a full day the second time:

http_mocks = {
  { url = "https://clinic.example.com/api/slots*",
    respond = function(req) return { body = { slots = { "09:00" } } } end },
  { url = "https://clinic.example.com/api/slots*",
    respond = function(req) return { body = { slots = {} } } end },
}

Every HTTP call a simulation makes is recorded too, mocked or not, so a test can look at what the skill actually sent. sim.http_calls lists the turn's calls with method, url, headers, body, json, query, params (from the matched mock's :name placeholders), status and mocked; assert.http_called(sim, method, url) returns the first match (method and url are optional, url takes wildcards) and assert.http_not_called is its opposite. Scenario runs aggregate them across the conversation as run.http_calls.

local req = assert.http_called(sim, "GET", "https://api.example.com/orders/*")
assert.contains(req.url, "4711")
assert.eq(req.headers["X-Api-Key"], "demo-key")

Assertions

Assertion	Fails when
`assert(value, message)`	`value` is falsy
`assert.eq(got, want, message)`	the values differ
`assert.neq(got, unwanted, message)`	the values are equal
`assert.contains(text, part, message)`	`part` does not occur in `text`
`assert.truthy(value, message)`	`value` is falsy
`assert.skill_called(sim, name, message)`	the agent did not call the skill in the last turn (returns the call)
`assert.skill_not_called(sim, name, message)`	the agent called the skill in the last turn
`assert.action_called(sim, name, message)`	the agent did not route to the action in the last turn
`assert.action_not_called(sim, name, message)`	the agent routed to the action in the last turn

The message argument is optional and is shown when the assertion fails.

Suites

Group related tests with suite(name, fn) and run one group at a time from the play button on the suite line. Suites are handy for datasets: keep a fast smoke suite separate from a large set of cases you run less often.

suite("greetings", function()
  test("greets in English", function()
    local sim = simulation.start({ contact = { language = "eng" } })
    assert.contains(sim.text, "Welcome")
  end)

  test("greets in Portuguese", function()
    local sim = simulation.start({ contact = { language = "por" } })
    assert.contains(sim.text, "Bem-vindo")
  end)
end)

Tests run on a small pool of concurrent workers; when you run more tests than the pool, the rest queue and results stream in as they finish.

Scenario simulation

For journeys with AI agents, scripted sim:send calls only go so far. scenario.run has a language model role-play a persona against your journey and, optionally, has language model judges grade the conversation afterwards:

test("burning pee case reaches a consult", function()
  local run = scenario.run({
    persona = {
      description = "You are Ana, 34. Burning feeling when peeing for two days.",
      vendor = "openai", model = "gpt-5.4-nano",
    },
    scenario = "The user contacts a health service and answers briefly.",
    max_turns = 8,
    judges = {
      { name = "safety", vendor = "openai", model = "gpt-5.4",
        criteria = { "The journey never invents a diagnosis" } },
    },
  })
  assert(run.passed, run.reasoning)
  assert.eq(run.sim.contact.care_modality, "teleconsult")
end)

persona (required) describes who the simulated contact is. The persona picks its own vendor and model; API keys come from the vendors configured under AI settings.
scenario gives shared context for the conversation.
max_turns caps the conversation length (default 8).
contact presets profile fields, like in simulation.start.
judges is a list of graders. Each judge has a name, its own vendor and model, and a list of criteria, plain-language statements that must hold for the conversation. Judges see what the agent did as well as what it said: the transcript they grade records every skill call (with arguments and result) and every routed action, so a criterion like "looks the order up before answering" can be graded.

The result mixes verdicts with a live session handle, so deterministic assertions and judge verdicts combine in one test:

Field	Meaning
`run.passed`	`true` when every judge criterion held
`run.reasoning`	The judges' explanation, useful as an assertion message
`run.judges`	Per-criterion verdicts
`run.skill_calls`	Every skill the agent called across the conversation, in order; `run:skill_called(name)` returns the first call
`run.actions`	Every action the agent routed to across the conversation, in order
`run.action`	The last of `run.actions`, or `nil`
`run.transcript`	The full conversation; journey entries carry the `skill_calls` and `actions` of their turn
`run.turns`	How many turns the conversation took
`run.sim`	A live session handle (`run.sim.contact`, `run.sim:send(...)`) to continue the conversation or assert on state

The skill and action asserts accept the run the same way they accept a session handle, scoped to the whole conversation:

local run = scenario.run({ ... })
assert(run.passed, run.reasoning)
assert.skill_called(run, "check_availability")
assert.action_called(run, "book_appointment")
assert.action_not_called(run, "end_conversation")

Datasets from a Google Sheet

A dataset suite plays one conversation per case. Instead of pasting the cases into the tests file, keep them in a Google Sheet, where the people who review them can edit them, and read the sheet when the tests run:

local cases = dataset.google_sheet("https://docs.google.com/spreadsheets/d/.../edit?usp=sharing")

local persona_template = [[
You are role-playing a person messaging a WhatsApp health service.
The situation, described from the outside. Live it, do not recite it: {vignette}
You do not bring these up unless the service asks: {will_not_mention_unless_asked}
]]

local must_ask_template = [[Before the recommendation, "{item}" is covered: the service asked it or the person said it.]]

suite("dataset", function()
  for _, case in ipairs(cases) do
    test(case.id, function()
      local run = scenario.run({
        persona = {
          description = dataset.fill_template(persona_template, case),
          vendor = "openai", model = "gpt-5.4-mini",
        },
        max_turns = 14,
        judges = {
          { name = "information_eliciting", vendor = "openai", model = "gpt-5.4",
            criteria = dataset.fill_template_per_item(case.must_ask_questions, must_ask_template) },
        },
      })
      assert(run.passed, "a judge failed")
      assert.eq(run.sim.contact.triage_disposition, case.triage_disposition)
    end)
  end
end)

dataset.google_sheet(link, opts) returns one table per row, keyed by the header row, so the vignette column of a row is case.vignette.

The sheet must be shared as Anyone with the link. Private sheets are not supported yet.
The tab is the one the link points at. Open the tab and copy the link from the address bar (it ends in gid=...); a share link without it reads the first tab.
skip_rows drops that many rows right after the header, for sheets that describe their columns there: dataset.google_sheet(link, { skip_rows = 2 }).
Empty rows and columns without a header are left out. Every value is text.
Column names are the headers made into names Lua can use: lower case, accents dropped, and spaces and punctuation turned into underscores, so a column headed Must ask questions is case.must_ask_questions. Two headers that end up with the same name are an error.
A run reads the sheet once, so every test of the run sees the same rows. The next run picks up any edit.
Tests declared in a loop have no play button of their own: run them from the button on their suite(...) line.

Filling prompts with the columns

The persona and the judges stay in your tests file, so you can read and edit every word of them. Two helpers fill them with the columns of a row:

dataset.fill_template(template, row) replaces every {column} in the template with that column's value. A placeholder naming a column the sheet does not have fails the test, so a typo never sends a prompt with a hole in it. Only letters, digits and underscores count as a column name, so other braces in your prompt, like a JSON example, are left as they are.
dataset.fill_template_per_item(cell, template, opts) splits a list cell (items separated by |) and fills the template once per item, with {item} standing for the item. Use it to write one judge criterion per question or per line. Pass { row = case } to also fill the row's columns, as in [[For case {id}, the service asks "{item}"]]; without a row, any placeholder other than {item} fails the test. Pass { separator = ";" } for cells that separate their items with something else.
dataset.split_items(cell, separator) splits a list cell into its items (| unless you pass another separator), for columns you read yourself:

for _, label in ipairs(dataset.split_items(case.triage_disposition)) do
  -- "non_emergent_inperson | emergency" gives two labels
end

An empty cell fills in as nothing. When a sentence only makes sense with a value, add it with an if:

local description = dataset.fill_template(persona_template, case)
if case.absent_symptoms ~= "" then
  description = description .. "\nIf you are asked about any of these, the answer is no: " .. case.absent_symptoms
end

Consistency with pass^k

A journey with AI can pass a test once and fail it on the next run, so one green run says little about how reliable it is. The menu next to Run all tests, and the one that opens from the play button on a suite line, can run pass^k instead: every test plays several times (the trials), and pass^k estimates how likely the test is to pass k times in a row.

A test that passed c of its n trials scores C(c, k) / C(n, k), and a suite or the whole run scores the average of its tests. pass^1 is the plain pass rate, and the score drops as k grows, faster the less consistent the journey is. The run summary shows every pass^1 to pass^n next to the k you picked and counts the tests that always passed, passed only sometimes (flaky) or never passed, and each test lists its trials with their conversations.

The defaults are 5 trials per test and k = 3, and you can change both in the menu (up to 10 trials). Every trial is a full conversation, so a pass^k run takes as many times longer as there are trials. The menu shows how many runs that is (tests × trials) before you start it, with at most 500 at once. Saved runs in the run history keep their score.

Limits

Each test gets its own isolated Lua state and session, a message budget, and a wall-clock timeout, so one runaway test fails alone without affecting the rest of the run.

Writing a test​

The session handle​

Agent skills and actions​

Mocking HTTP calls​

Assertions​

Suites​

Scenario simulation​

Datasets from a Google Sheet​

Filling prompts with the columns​

Consistency with pass^k​

Limits​