Tales of extreme debugging

Sometimes debugging, at least through the rose-tinted goggles of hindsight can be quite amusing. Today I’m going to share with you two of the better ones I’ve experienced myself to brighten up your day. I don’t really know if there’s any wisdom to be drawn from them, but they’re true and fun.

It’s too cold outside to debug

Back in late -04 early -05 I was working for Gizmondo programming games. This wasn’t exactly by choice, since our studio was bought by them we just sort of ended up doing games for this unit that was supposed to be the next big thing. Turned out it became a big thing for all the wrong reasons. So, I was one of the precious few people who actually got to develop on this machine, and while it might have been a miserable failure in some points it was groundbreaking in others. One thing was that this little device actually included a GPS chip. Now kids, this is way before the first iPhone so this was quite a novelty. I had this stint with some talented guys to make the client side of this multiplayer game Colors, which was to be the first GPS multiplayer game in the world. The gist of it was that you could control neighborhoods by actually going there and challenging the current ruler. So, being the client side guy I set out on a task to implement this.

If there is one thing you should know about GPS is that it is an extremely sensitive technology. And I don’t mean sensitive in the sense that it measures accurately. No, what I’m talking about here is that the satellite signals are very weak and it doesn’t take much to make the GPS very confused. So, when I started up this device and went to it’s dashboard it took an awfully long time to detect the signals, but eventually it did – provided it was reasonably close to the window. However, when I tried to do this while rendering anything at all on the screen the most I got out of it was two satellites and a position about halfway to Copenhagen. Turns out that no-one had considered the possibility of the graphic chip actually inducing interference on the GPS chip causing it to go haywire. There was no time to rectify this in the current hardware, so I had to make due with what I had. Which meant finding out at exactly what treshold it actually went bananas.

The only way to do this was to go outside. And this is Sweden in February. Each iteration of this debugging session entailed going outside and walking to the city park in Helsingborg and sitting down at a park bench staring at the device. Waiting for about 20 minutes to see if I got enough satellites and walking back again, usually dissappointed. To compound this, I didn’t have a laptop so there was no way to actually step through the code. And if I did printf style debugging to write to the screen it might disturb the GPS. Since then, I’ve always at least been grateful that no matter what problem I’m having, at least I’m warm.

Mysterious pauses and flying cars

Game development leads to visually funnier bugs than other branches of development. It was at least back then more prone to heisenbugs, bugs that defy detection and only appear if a debugger isn’t attached. This is a tale of a bug that fits both these criteria. During the beta testing period of the development of Richard Burns Rally, we had a bug report filed that sometimes after running races against the ghost car the game would freeze up on the PS2. The ghost car was a fully platform generic component of the game, so the report itself puzzled us some. We found out that it was indeed true, there was a one to two second pause when the ghost car finished the race, but only for prerecorded ghost cars. And only for prerecorded ghost cars driven by one of our speed daemon artists that consistently set the record for the stages.

So, the question came up, what the hell is the ghost car doing when it crosses the finish line? And how can we find out? There was only one answer in reach, and that was to drive as fast as or faster than the ghost car and look at it when it was supposed to stop after the finish line. This took a few men and many tries but eventually we managed to get to the finish line and we observed what happened. Once the race finished the ghost car teleported two feet up, and turned slighly to the left and up towards the sky. It then continued to drive upwards at an amazing speed and stopped. We just looked in amazement at this display.

So, it turns out that this was an error with the prerecorded ghost cars. More specifically it was missing four bytes at the end. Those four bytes terminated the sequence of prerecorded moves and were supposed to be 0x00000000. Since we never wrote those terminators the ghost car continued to happily drive it’s way through random memory, or other fragments of ghost cars of the past since we continually re-used the buffer for ghost cars. On the XBOX and PC it just so happened that the array were zeroed out properly by memory management. On the PS2, since we had our own memory management routines, it never was. And because of those memory management routines it was hard to get an illegal memory access violation. So it just happily looped through quite a lot of the memory until it randomly stopped.

We never re-recorded those ghost cars. We modified the file loading to append four more bytes, tossing the problem under the rug and released without either flying cars or mysterious stops.

Tagged , , , , ,

2 thoughts on “Tales of extreme debugging

  1. Sebastian says:

    The gizmondo/colors project was interesting in so many ways. Thanks for sharing a small piece of it that I had actually forgotten.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: