Skip to content
LiteTraffic

verdicts

Four verdicts. None of them a guess.

A load test counts responses. LiteTraffic checks the effects those responses were supposed to have, and only calls a run a pass when the evidence is complete.

observe · verdict

interactive comparison · checkout --wrong-duplicate

A fast response can still be a failed checkout.

The demo server's faulty mode charges twice when a payment is retried. Every request still succeeds. Switch views to see what each tool reports.

verify · checkoutillustrative timing
one_effect_per_paymentFail
expected1
actual2

2 charges for 1 payment.

{"actual": 2, "expected": 1,
 "logical_key": "<run_id>-traffic-0",
 "sequence": 2}

Each failing sample keeps what the scenario expected and what the app did, so the report points at the retry, not at a log.

examples · deliberate faults

Each included scenario ships with a working demo server and a faulty mode. The same scenario must pass one and fail the other.

  • CheckoutDuplicate charge on retry
  • InventoryOverselling
  • ReportingPartial response or wrong final ledger
  • Cached searchPermanently stale cache
  • Tenant APIData leak or deny-all shortcut

What each verdict means

VerdictExitMeaning
pass0Every declared assertion passed, with complete expected evidence and no reported run limitation.
fail1At least one declared business assertion definitely failed.
inconclusive2Evidence or progress was incomplete without a definite failure, such as a timeout. An unproven run is not a pass.
error3The engine, lifecycle, a budget, a fixture or an unreachable target prevented a clean evaluation.

litetraffic verify exits with the verdict, so CI needs no parsing to pass or fail. Ctrl+C exits 130 and still keeps the evidence collected so far.

Lifecycle is a separate field

Every run also records how it ended: finished, timed_out, cancelled or crashed. The two are independent. A timed-out run can keep a definite failing assertion; it can never pass. A k6 threshold breach (exit status 99) is not a crash: the run stays finished and fails. While a run is in progress, run.json records its stage: preparing, running, observing, then finalizing.

What keeps a run from passing

  • A missing sample, a sample without a journey key, or the same key reported twice makes that assertion unknown.
  • k6 dropping iterations (under-delivered load), or a journey's min_overlap not being reached, adds a limitation.
  • When checks pass but no journey reached its final declared stage, the run is inconclusive: a green run where nothing finished is a silent failure.
  • An observer that cannot respond yields unknown evidence.
  • A target that never answers is an error, and failing rows become unknown. A run that gets any HTTP response, even a 500, is judged normally.
  • Exceeding max_requests, max_write_attempts, max_in_flight or max_artifact_bytes, or a failed fixture cleanup, makes the verdict error.

Every result also carries notes on what the preview never measures, such as per-arrival lateness. Notes never change the verdict.

Failing samples point at the cause

Each failing assertion in result.json keeps its first three failing samples, each with its sequence number, journey key, expected, actual and detail. report.html shows them as a table and the local dashboard expands them in place, so a failure points at the journey that broke, not at a log.

Comparing two runs

litetraffic diff compares a baseline and a candidate run. They are comparable only with the same bundle digest, seed, engine version and resolved schedule; otherwise the comparison is inconclusive.

  • It fails when the candidate's verdict is fail, or when the optional p95 gate reports a regression.
  • It is inconclusive when the candidate errored or was inconclusive, or delivered less work than the baseline.
  • The p95 gate needs a positive baseline p95 and at least 200 duration samples in each run. Without a gate, timing differences are reported but not graded.

Details in results and reports and running in CI.