The TestSprite suite

TestSprite is a testing service that drives a real browser. For every pull request to Penny it opens that pull request's own preview and runs 24 tests against it. GitHub will not merge the pull request until TestSprite's check is green. A second TestSprite project, the API tests, calls Penny's API on the live site after every merge.

The 24 testsCopy Anchor Link

The names are TestSprite's own. The first 21 are listed in its comment on pull request #10; the last three were added after the planted-bug study.

That is 7 tests on groups and members, 5 on splitting, 2 on currency, 2 on correcting expenses, 5 on balances and settling, and 3 added after the study.

How the tests are writtenCopy Anchor Link

• Planned from the spec, then cut by hand. TestSprite proposed 57 tests from the product spec. 12 were kept at the start, after dropping duplicates and tests for features not yet built, and each was rewritten with fixed data and exact expected text before TestSprite ran it. The suite grew with the product, to 14, 16 and then 21 tests as features landed, and to 24 after the study.
• Fixed data. Names, amounts, currencies and rates are chosen in advance, so the right answer is known before the test runs.
• Exact expected text. A test checks what is written on the page, not just that the page loaded. Examples: "33.34" for the leftover cent, "155.74 EUR" for a balance, and "3 payments settle 7 expenses." for the sample trip.
• Each test makes its own group. Every test starts from the home page and creates a fresh group, so tests never share state and a failure points at one behaviour. No test depends on a fixed production group, because anyone can edit a group whose link appears in a public test file or report.
• No back door. Production has no seed, reset or bypass endpoint, flag or header for tests. TestSprite's browsers get exactly what an anonymous visitor gets.

The 16 API testsCopy Anchor Link

A second TestSprite project calls Penny's API directly, with no browser. A backend test carries a fixed address, so these run against production, not a preview. After each merge to main, the api-suite workflow waits until the live site serves the merged commit, then runs every test in the project. A failure fails that workflow run on the merge commit.

• Whole answers. A test compares the full balance list, the full settle-up plan or the full error body, for example {"error": {"code": "stale_version"}}, never a single field or just a status.
• Hand-worked numbers. Each expected amount was worked out from the money rules, with the working in a comment beside it, never copied from what the server returned.
• Checked to fail. Each test ran twice against production, then once with one expected value changed, and that copy failed.
• Plain files. The tests are Python files in testsprite/api. PENNY_BASE_URL points one at another server, such as a local copy.

A full run makes about 20 groups named "API suite: ..." on production. They hold made-up data and open only with their own links.

What happens on a pull requestCopy Anchor Link

1. A commit is pushed to a pull request branch.
2. Vercel builds a preview. The preview uses its own Neon database branch, copied from the preview database, which holds synthetic data only.
3. TestSprite tests that preview and posts a commit status named TestSprite V3 on the commit. It is pending while the tests run, then success or failure. The status text reads "All 24 tests passed" or, for a failure, something like "23/24 passed, 1 failed".
4. When the pull request closes, the neon-cleanup workflow deletes its database branch.

The whole sequence is drawn on the self-fixing loop page.

The check holds the mergeCopy Anchor Link

The branch rule on main requires a pull request, applies to administrators, and blocks force pushes and deletion. Two commit statuses must be green before GitHub allows a merge.

Each check is pinned to its source. A job that borrows either name cannot satisfy the rule. The ci workflow (typecheck, lint, unit tests) also runs on every pull request, but it is not a required check.

The checks pageCopy Anchor Link

The checks page, headed "How Penny is tested", shows the same facts from inside the product, read live from GitHub.

• "Live right now" says which commit this site runs, for example "This site runs commit f18a91e from pull request #14.", which it said on 20 October 2026 after that pull request merged. Under it is that pull request's own TestSprite line, read from its head commit by the same rules as the list, however old the pull request is.
• "Recent pull requests" lists the 12 newest pull requests opened by the maintainer. Each shows its number and title, its state, the TestSprite line, and links to "Pull request" and "Test report (TestSprite sign-in)". The report opens on TestSprite's site, which asks you to sign in. The TestSprite line reads like "TestSprite: passed, 24 of 24" or "TestSprite: failed, 23 of 24 passed, 1 failed".
• A line at the bottom, such as "Read from GitHub at 05:31 UTC.", says when the data was fetched. If GitHub did not answer, the page keeps the last good read and says so.

The verdict is one of passed, failed, blocked, running, not run or unknown. The page is strict about "passed":

• It shows "passed" only when TestSprite's GitHub App posted the status, on the pull request's current head commit, as exactly success. A same-named status from anyone else does not count.
• If the counts cannot be read from the status text, the verdict is unknown and the counts are blank. They are never shown as zero.
• A success with no passed test and some blocked tests shows as blocked, because nothing was shown to work. Blocked tests are always counted in the line.
• The report link appears only if it is an https address on TestSprite's own report host.

Planted-bug studyCopy Anchor Link

Eight pull requests titled "Study" each plant one deliberate bug. TestSprite tests each one, and each is closed unmerged.

TestSprite caught 5 of 8. The unit tests caught b1 to b7; only the browser suite caught b8.

Three tests were then added to the suite, one aimed at each miss, making 24:

• "Convert at a rate that lands on half a cent": 10.05 USD at 0.5 must show "= 5.03 EUR at 0.5".
• "Settle up in the fewest payments when the simple plan needs more": a five-person trip that settles in exactly 3 payments, where the greedy plan needs 4.
• "Reset link: the old link opens nothing and the new one opens the group".

The three missed bugs were tested again with all 24 tests (#16, #17 and #21, reopened with a fresh preview, so their latest check now reads "23/24 passed, 1 failed"). Each time the one failure was the new test aimed at that bug. Each new test passed on the other two planted bugs and on production, so none of them fails on correct code.

The suite now catches all 8 planted bugs.

What the suite does not coverCopy Anchor Link

A green check means TestSprite's tests passed, nothing more. The 24 tests cover groups, splitting, currency, corrections and settling in a browser, against a preview with synthetic data. They say nothing about how the home page looks or animates. The API tests check error codes and the 16 KiB body cap on production, but never trip a rate limit. Message wording for every server error code, rate limiting and the database rules are covered by the unit and database tests on Unit and property tests.