The crash was in the quit

My test runner has a comment in it that I am not proud of. It says the teardown crash is not a verdict. One probe in the suite, the one that checks the planet's water table, would boot the planet, run its checks, print its own result line with fails=0, and then exit with code -1073741819, an access violation at the same address inside the engine binary almost every time. The runner had been taught to read the result line, notice the crash, print a note, and count the run as a pass. The note was there so the crash could never turn invisible. It turned invisible anyway, because a note you see every day is wallpaper.

Spaceframe is a survival game on a sphere 10 km in radius, built in Godot 4.6 with the godot_voxel add-on streaming the terrain in and out around the player. My coding agent writes most of its code; my job is to insist on probes, about three hundred scripts that boot the world, assert something and exit zero or one, and to read their verdicts. The forgiveness rule had a reason. The crash fired after every line of script had run, and it fired in the engine, not in anything I had written. But a probe quits by calling a function no player can call. A player quits with the window's X, and the X went through exactly the same crash. I was shipping it.

So I counted first. The water-table probe was made to quit 30 times and crashed on 28. Nine other probes quit 64 times between them and never crashed once. An orderly teardown, removing every terrain viewer, freeing the terrain and waiting two frames before quitting, still crashed on every run, and the add-on's optimized build, the one a shipped game loads, crashed on all but one. The crash belonged to one probe's behavior, not to the engine at large, and freeing the terrain was attacking the wrong object.

Then I read the code that was crashing, which meant first finding out exactly which code it was. The add-on ships as a compiled library with no version file. I scanned the library's bytes for the string it reports as its own git hash, 595f52ee, so a research spike, read-only, could fetch the source at that exact commit and read the teardown path line by line. The mechanism took three files to name. When a terrain block leaves view, the add-on does not delete it; in voxel_mesh_map.h it pushes a small deletion task onto the main-thread queue it also uses for meshing work, under a one-line comment that reads "We spread this out because of physics", since deleting a block also frees its collision body from the physics server. The queue, in time_spread_task_runner.cpp, is drained under a time budget every frame. Its destructor, though, does not discard what is left. It runs it.

That would be fine if the destructor ran early, and it runs late. The queue is a member of the add-on's engine object, which register_types.cpp tears down at the scene level of extension shutdown, and Godot's own cleanup, in main.cpp at the tag the game runs on, deletes the scene tree first, and with it every physics space in the world, then deinitializes scene-level extensions, and only four steps after that finalizes the physics server itself. So the deferred deletion calls a physics server that still exists, to free a body whose space no longer does, and the server walks into a null pointer. Every crash dump the spike opened, ten of them, agreed on the faulting address, and every one of them faulted reading a null pointer plus the same small offset, 0xe0.

It also explained the one probe. The water-table probe runs a loop of two hundred synchronous samples with no frame awaited inside it, while streaming keeps enqueuing deletions in the background, so it reaches its quit with the deepest queue in the suite. The probes that let streaming settle before quitting never crashed. That made a prediction I could test before fixing anything: a run crashes when the pending count is non-zero at the moment of quit, and the add-on publishes that count.

The fix started with a print statement and no fix. One line before the quit printed the add-on's pending main-thread task count. Every water-table run printed a non-zero count and crashed; every settled run printed zero and exited clean. Only then did the fix go in, at one exit: stop the viewers so nothing new is requested, render frames until the count reads zero with a three-second budget, then quit; if the budget expires, confirm the game's save is on disk and hard-exit. The window's X was routed through the same path, since the engine's default is to quit on the frame the X arrives. That afternoon, alternating the two, all six plain quits crashed and none of six drained quits did, and the six crash dumps written that day matched the six plain quits by timestamp.

The upstream issue is real and older than my game. It was opened by the add-on's own author in 2022 under the queue's other job, mesh tasks, and it describes this mechanism; the spike read the teardown code of the release after the shipped commit, 1.7, and found the same queue and the same flush, so upgrading would have changed nothing. My fix is a workaround in the caller, not a repair in the library, and the library's optimized build shares the bug.

The player's exit got its own probe. It churns the streaming queue with a long teleport out and back, sends the window a close request on that frame, and never calls quit itself. Two things can go wrong from there and both are visible: if the close request never reaches the drain, the probe hangs and the runner's timeout kills it, a red that cannot be forgiven; if the drain never reaches zero, the fallback fires and stamps the log with its count, which the runner does not yet turn red on its own. The teleport is there to starve the queue the way the water-table probe did by accident, out and back with no frame between, and it works: the first run reached a peak of 89,928 pending tasks, drained them to zero in 38 frames, and exited clean, and it has passed on both suite runs since. It empties that fast because the queue's per-frame budget is a time budget and a deletion for a block already out of view is cheap; the queue is only ever deep when nothing gives it a frame.

What is not done is easy to name. The crash is worked around, not fixed, and the workaround has a three-second budget with a hard exit behind it. No measured run has ever reached that budget, the worst case so far taking 38 frames, so the hard exit has never fired; it is a guess with a save check in front of it. The forgiveness rule is still in the runner, because the probes still quit the old way and would still crash; it has simply stopped being the player's problem. The prediction was only ever tested as a yes or no, non-zero against zero, not as the curve it claims to be. The player's exit has one probe and three runs standing on it, against the 30 quits the bug got, and a fallback exit would not yet fail it; that is thinner than I would like.

The lesson I keep is smaller than the bug. The runner had an opinion, that a crash after the last assert does not count, and the only reason the opinion got overturned is that the same suite made counting cheap: 30 quits here, 64 there, a print statement before the fix, six more quits after it. The numbers did not care what the comment said.