How we test our recipes
Every number on this page is computed from our published test results file, generated Oct 28, 2026 (2026-10-28T17:17:33+01:00). This page describes what we ran, where it ran, and what broke. It isn't a performance claim: a passing test means the workflow did what we expected against our test setup, not that it will work unchanged in your business.
The testbed
- Workflow engine: every workflow is imported into n8n 2.45.4, self-hosted on our own test machine, and run end to end through its webhooks and schedules.
- Outside services: a local mock server accepts the same request shapes as each service's documented API and records every call, so tests can check exactly what would have been sent. Mocks follow the docs; they can't catch an API that behaves differently from its documentation.
- AI steps: a small local model, Qwen2.5-3B-Instruct (4-bit), runs on the same machine. We haven't tested hosted AI models.
- Speech to text: Whisper base.en, running locally.
- Data: invented customers, jobs, invoices and messages. No real customer data.
- Runs: results dated Oct 27, 2026 to Oct 28, 2026. Each recipe's page shows the date of its own run.
The numbers
| Trade | Recipes | Scenarios passed | Against live Stripe test mode | Known limitations listed |
|---|---|---|---|---|
| HVAC & Plumbing | 5 | 55 / 55 | 8 | 16 |
| Law Firms | 5 | 38 / 38 | 1 | 23 |
| Dental Practices | 5 | 29 / 29 | 0 | 24 |
| Real Estate Agents | 5 | 36 / 36 | 0 | 21 |
| Accounting & Bookkeeping | 5 | 26 / 26 | 0 | 22 |
| Roofing Contractors | 5 | 27 / 27 | 0 | 27 |
| All | 30 | 211 / 211 | 9 | 133 |
Source: data/recipes.json, generated 2026-10-28T17:17:33+01:00, built from each recipe's latest published result file. Earlier failed runs aren't in these totals; they're listed below. A shared hardening suite (forged-webhook rejection, Twilio signatures, Clio and Jobber token refresh, Clio rate-limit retries) also runs but isn't counted here.
Failure modes we found
The most useful output of testing is the list of things that go wrong. Each recipe page has a “Known limitations” list; together they hold 133 items. The recurring kinds:
- The AI step is the least reliable part. The small local model sometimes drops or misreads line items, which is why the voice-memo recipe writes a draft note for a person to approve, not invoice line items.
- Speech-to-text slips carry through. A misheard word (“flashed” for “flushed”) becomes an unmatched item, and a skipped sentence becomes a missing one.
- Setup mismatches fail quietly. For example, a webhook signature check rejects every real callback if the public URL you configure doesn't match exactly.
- Mocks follow the docs. Real accounts can have rate limits, permissions, plan limits and API version changes our mocks don't reproduce.
- Message length costs money. Some texts run to 2 SMS segments, which costs twice as much as one.
Flaky and failed runs
These are from our run logs. The published totals above use each recipe's latest result file, so a recipe can show a pass there and still appear here.
| When (CET) | What | Result | What happened |
|---|---|---|---|
| Oct 28, 2026, about 02:27 | voice-memo-to-invoice | 4 of 5 | The local AI model listed “flashed tank” (a speech-to-text slip for “flushed the tank”) as an unpriced extra item instead of matching it to the water-heater flush in the price book. The next run passed 5 of 5. |
| Oct 28, 2026, about 17:30 | voice-memo-to-invoice | 3 of 5 | The local AI model left expected line items out of two draft invoices (memos M2 and M3). An earlier run the same day passed 5 of 5. |
| Oct 28, 2026, 18:08 | voice-memo-to-invoice | 4 of 5 | For the kitchen-sink memo (M2), the speech-to-text output skipped part of what the technician said, and the draft invoice left out the P-trap and the labor hours. The test caught it because it checks every expected item and the hours. |
| Oct 28, 2026, about 01:55 | real-estate-past-client-anniversary | test bug | Our test grabbed the unsubscribe link from whichever email arrived first, and the order isn’t fixed. We fixed the test to pick the right recipient’s link; the workflow itself was fine. |
| Oct 28, 2026, about 01:55 | HVAC suites using the AI model | failed | The local AI model server had stopped, so every AI step failed. We restarted it and reran. |
| Oct 28, 2026 | whole suite | false failures | Two test runs overlapped. They share one n8n, one mock server and one local AI model, and the second run reset the mock log under the first, which caused 3-of-5 flakes. We added a lock so only one full run happens at a time. |
| Oct 28, 2026, about 17:45 | whole suite | timeout | One full run hit our 600-second limit during the final phase; 30 of 31 suites had passed. |
The voice-memo recipe is the flakiest one we publish: it chains speech-to-text and a small AI model, and it doesn't pass every run. The workflow writes a draft note for a person to approve before billing, not an invoice.
Download the test log
One row per scenario: recipe, trade, scenario, result, run date, mock or live, result file, and that recipe's known limitations.
What we don't test
- Your real accounts, numbers, phone carriers and message deliverability.
- Hosted AI models, or long-running use over weeks.
- Whether a message is legal to send. That depends on consent and your trade's rules; see texting consent rules by trade (not legal advice).
Found something wrong? Send a correction. More about us: About and Matron.