Build notes / Process · Testing

Why every feature is tested in the running app

Automated tests check what we thought to test, and some of them drive the page in a browser. They do not replace a separate pass that uses the feature the way a person would and checks it against what it promised. So before a feature is merged, it is used in the running app, and the result is written on its pull request.

Sheet
N · Build note
Subject
Process · Testing
Where
Simulator, emulator, device or browser
Verdicts
Pass · fail · blocked
Recorded
On the pull request

R-01 What tests miss

Passing tests are not the same as a working feature

Automated checks run on every pull request, and they catch a great deal. Unit tests check pieces of code, and browser tests can drive a page. But each one checks only what its author thought to check. None of them replaces using the feature for what it is for and comparing the result with what it promised.

Some problems only show up that way: a button the keyboard covers, text cut off on a narrow screen, a step that works alone but not after the one before it. So passing tests alone never count as acceptance.

R-02 The check

What the check writes down

01 · The build

The exact version

The feature is tested on the exact version in the pull request, not on an earlier one.

02 · The oracle

Expected first

What should happen is written down from the feature's scope, separately from what the screen shows.

03 · The tester

Not the builder

A separate test pass runs the check. Whoever built the feature does not test it.

04 · The verdict

On the record

Pass, fail or blocked, with the cases behind it, posted on the pull request.

What was tried and what was observed also go into the pull request description, so anyone reading it can see what happened.

R-03 One feature at a time

Each feature gets its own test

Every feature has a stable name in a map of the product, along with the files it runs on. A change lists exactly which features it touches, and each of those features gets its own test. Two features never share one.

If a test fails, the findings go back to the same branch. The fix runs the automated checks again, and the same feature is tested again on the new version. A feature that already passed, and that the fix does not touch, keeps its result.

R-04 In a client build

The same path for your app

A client build follows the same steps as our own products: a written scope, one pull request per feature, automated tests and review, then testing in the running app before the release path agreed for the project.

Start a project

Tell us what you want to build and which platforms you need, or send the scope you drew. We reply by email.

hi@cr8veworks.com