Guides·5 min read

Proving a Game Is Finished, Rather Than Hoping

A build can compile, open, look perfect and be impossible to finish. Ours are tested by a machine that plays them.

The level card from Tape Loader reading Level 1 of 3, The Bedroom, 1982, the machine arrives.

Here is a way to deliver a broken game while every signal says it is fine.

The file assembles without error. It opens in a browser. The title screen appears, the music plays, the character moves. You send it. Somewhere in the middle of level two there is a gap that cannot be jumped, and the person it was made for discovers that on their birthday.

Nothing in a normal build process catches that, because "it built" only means the file was put together.

The failure that taught us this

Our own engine ships with an example build made from real production artwork. It compiled. It opened. The character walked, jumped and animated correctly.

The level could not be completed at its intended difficulty. Reaching the goal needed precise vertical platforming, which is exactly wrong for the setting aimed at a four-year-old. Everything technical was correct and the thing did not work.

That is a design error surviving a technical check, and it is the category that gets delivered. So the checks changed to test the design.

The seven checks

Every build has to pass all of these before anyone sees it. A real browser opens the actual file and drives it.

One: it is within budget. A hard size limit, because the single-file promise stops being kind past a certain point. This is the check we recently failed deliberately, and it is why one of our builds is not playable on this site.

Two: every image actually decodes. Not "the file contains an image", but the browser successfully turned it into pixels of non-zero size. A sprite that silently fails to decode leaves an invisible character, and the game plays perfectly around a hero nobody can see.

Three: the level can be completed. The important one. A machine plays it: holds right, jumps on a cadence, and the build fails if the goal is never reached. Not an inspection of the level data, an actual playthrough at the target difficulty.

Four: the collectibles are reachable. A game with a counter reading 13 of 14 that can never say 14 is worse than one with no counter, because it tells the player they have missed something that does not exist.

Five: nothing touches the network. Zero requests during play. If the game reaches for anything outside itself, the build fails. This is what makes the promise that it will still work in five years an actual property rather than an aspiration.

Six: no errors in the console. Games are forgiving of small errors in a way that hides them: a handler throws, the frame carries on, and the visible symptom appears an hour later in a saved score.

Seven: the title is set. Trivial, and it is what shows in the browser tab and on the link when it is shared with family. Getting a game called "Untitled Document" undercuts everything else about it.

The level card from Tape Loader reading Level 1 of 3, The Bedroom, 1982, the machine arrives.
Three levels means three completion checks, at the difficulty the build is actually set to.

Playing deterministically

One technical detail worth explaining, because it is what makes the completion check trustworthy.

The obvious approach is to open the game, send keystrokes and see what happens after a while. That is unreliable in a way that is worse than no test: it depends on how busy the machine is, so it passes on a fast day and fails on a slow one. A check that fails randomly gets ignored, and an ignored check is worse than an absent one.

So the game exposes its own physics to the test. The test does not wait for real seconds to pass, it steps the game forward frame by frame and drives the controls between steps. The same test on any machine produces exactly the same playthrough. It either completes the level or it does not, and the answer means something.

What testing does not catch

Worth being straight about the limits.

It does not tell you the game is good. Nothing automated does. It cannot tell you the joke landed, the likeness is right, or that the pacing works. That is what the approval stage with a real person is for.

It does not catch everything technical either. Two of the defects we found most recently came from real devices, not from the checks: touch controls that overlapped each other on any screen under about 700 pixels, and a game that sat blank because it was waiting on a font from a third-party server. Both were found by opening the thing on a phone.

So the honest position is that automated checks catch the failures that are boring, repetitive and catastrophic, and a human catches the rest. Neither replaces the other.

Why this matters commercially

Most software can be fixed after release. A birthday game largely cannot.

It is opened once, in front of people, on a specific day. If it fails then, the moment does not come round again, and a patch on Tuesday does not undo it. The asymmetry is the whole argument for testing something this small this thoroughly.

It is also why we would rather find a level uncompletable a fortnight early than have the person it was made for find it. The checks exist to move the discovery earlier, which is all testing has ever done.

What to ask any studio

Ask how they know the game can be finished. If the answer is "we played it", ask who, at which difficulty, and how recently relative to the last change.

Ask what happens if their server goes away. A game that depends on an outside service is borrowed rather than owned.

Ask to see it on a phone, before delivery, on a real one. It is where most of what we have found actually turned up.

Difficulty settings explains why completion has to be tested at the level it is aimed at, and the process guide covers where testing sits between brief and delivery. Both builds in the arcade have been through all seven.

◆ Questions

Common questions

How do you test a custom video game before delivery?+

Automatically, on every build. A machine opens the game in a real browser and checks seven things, including that the level can actually be completed by playing it through rather than by inspecting the code.

What does it mean for a game to build but not work?+

A build only proves the file was assembled. It does not prove the artwork decoded, that the goal is reachable, or that nothing errors during play. All three have happened to us and all three would have looked fine on delivery.

Can a game be impossible to finish without anyone noticing?+

Easily. Our own example level once required precise vertical platforming to reach the goal, which was wrong for its intended difficulty. The build was green and the game ran. Only playing it revealed the problem, which is why a machine now plays every build.

Do your games work offline?+

Yes, and it is tested. One of the checks fails the build if the game makes any network request at all during play, because a commission that quietly depends on an outside server is one that can stop working later.

What happens if a test fails?+

The build does not ship. The check that most often catches something real is completion, because it tests the level design rather than the code.

  • process
  • behind the scenes
  • craft
  • quality
◆ Keep reading

Related from the journal.