there is a small pokédex here

Hey, I am Eevee and this is veekun and it's a Pokédex. You probably want to type into that box in the top right, or maybe just start browsing.

Other stuff of interest:

Updates


The Pokémon TV app is now available for Apple TV.
A new trailer for Pokémon Sun and Moon has been released, which features seven new Pokémon.
Codes for Shiny Xerneas will be available at Hyper Japan 2016 in London, England from July 15 to 17, 2016.
The Sinnoh Classic Wi-Fi tournament has been announced. Participants will receive Cynthia's Garchomp, which was used in the Generation IV games.

June’s theme is clearing my plate.

  • SLADE: I fixed yet another obscure geometry editing bug. I feel a little more cleansed every time I do. Also I found out that GZDoomBuilder based its own geometry code on SLADE’s and almost immediately borrowed my SLADE fix.

  • art: Still doing daily Pokémon. The half hour time limit is surprisingly harsh, but I think I’m making some progress, gradually. Also finished an old art trade.

  • runed awakening: I built out most of a whole puzzle sequence that had languished unfinished for a while, and drew a WIP or two of some of the rooms and objects involved. It doesn’t sound like a lot, and I wish I’d done more, but I made some good progress.

  • zdoom: On a total whim, I did an extremely simple proof of concept of Lua in ZDoom. There’s no approval or planned goals or anything for this; I just felt like trying it out. May or may not continue it. Ideally it would be able to replace ACS and at least massively enhance DECORATE.

  • blog: I wrote about video games again.

I think I ran out of small things on my plate, and now I’m finding all the large things a little daunting. And here I’ve invented another one. Argh. Still, keep moving forward.

I’m almost 30, so I have to start practicing being crotchety.

Okay, maybe not all video games, but something curious has definitely happened here. Please bear with me for a moment.

Discovering Doom

Surprise! This is about Doom again.

Last month, I sat down and played through the first episode of Doom 1 for the first time. Yep, the first time. I’ve mentioned before that I was introduced to Doom a bit late, and mostly via Doom 2. I’m familiar with a decent bit of Doom 1, but I’d never gotten around to actually playing through any of it.

I might be almost unique in playing Doom 1 for the first time decades after it came out, while already being familiar with the series overall. I didn’t experience Doom 1 only in contrast to modern games, but in contrast to later games using the same engine.

It was very interesting to experience Romero’s design sense in one big chunk, rather than sprinkled around as it is in Doom 2. Come to think of it, Doom 1’s first episode is the only contiguous block of official Doom maps to have any serious consistency: it sticks to a single dominant theme and expands gradually in complexity as you play through it. Episodes 2 and 3, as well of most of Doom 2, are dominated by Sandy Petersen’s more haphazard and bizarre style. Episode 4 and Final Doom, if you care to count them, are effectively just map packs.

It was also painfully obvious just how new this kind of game was. I’ve heard Romero stress the importance of contrast in floor height (among other things) so many times, and yet Doom 1 is almost comically flat. There’s the occasional lift or staircase, sure, but the spaces generally feel like they’re focused around a single floor height with the occasional variation. Remember, floor height was a new thing — id had just finished making Wolfenstein 3D, where the floor and ceiling were completely flat and untextured.

The game was also clearly designed for people who had never played this kind of game. There was much more ammo than I could possibly carry; I left multiple shell boxes behind on every map. The levels were almost comically easy, even on UV, and I’m not particularly good at shooters. It was a very stark contrast to when I played partway through The Plutonia Experiment a few years ago and had to rely heavily on quicksaving.

Seeing Doom 1 from a Doom 2 perspective got me thinking about how design sensibilities in shooters have morphed over time. And then I realized something: I haven’t enjoyed an FPS since Quake 2.

Or… hang on. That’s not true. I enjoy Splatoon (except when I lose). I loved the Metroid Prime series. I played Team Fortress 2 for quite a while.

On the other hand, I found Half-Life 2 a little boring, I lost interest in Doom 3 before even reaching Hell, and I bailed on Quake 4 right around the extremely hammy spoiler plot thing. I loved Fallout, but I couldn’t stand Fallout 3. Uncharted is pretty to watch, but looks incredibly tedious to play. I never cared about Halo. I don’t understand the appeal of Counterstrike or Call of Duty.

If I made a collage of screenshots of these two sets of games, you’d probably spot the pattern pretty quickly. It seems I can’t stand games with realistic graphics.

I have a theory about this.

The rise of realism

Quake introduced the world to “true” 3D — an environment made out of arbitrary shapes, not just floors and walls. (I’m sure there were other true-3D games before it, but I challenge you to name one off the top of your head.)

Before Quake, games couldn’t even simulate a two-story building, which ruled out most realistic architecture. Walls that slid sideways were virtually unique to Hexen (and, for some reason, the much earlier Wolfenstein 3D). So level designers built slightly more abstract spaces instead. Consider this iconic room from the very beginning of Doom’s E1M1.

What is this room? This is supposed to be a base of some kind, but who would build this room just to store a single armored vest? Up a flight of stairs, on a dedicated platform, and framed by glowing pillars? This is completely ridiculous.

But nobody thinks like that, and even the people who do, don’t really care too much. It’s a room with a clear design idea and a clear gameplay purpose: to house the green armor. It doesn’t matter that this would never be a real part of a base. The game exists in its own universe, and it establishes early on that these are the rules of that universe. Sometimes a fancy room exists just to give the player a thing.

At the same time, the room still resembles a base. I can take for granted, in the back of my head, that someone deliberately placed this armor here for storage. It’s off the critical path, too, so it doesn’t quite feel like it was left specifically for me to pick up. The world is designed for the player, but it doesn’t feel that way — the environment implies, however vaguely, that other stuff is going on here.


Fast forward twenty years. Graphics and physics technology have vastly improved, to the point that we can now roughly approximate a realistic aesthetic in real-time. A great many games thus strive to do exactly that.

And that… seems like a shame. The better a game emulates reality, the less of a style it has. I can’t even tell Call of Duty and Battlefield apart.

That’s fine, though, right? It’s just an aesthetic thing. It doesn’t really affect the game.

It totally affects the game

Everything looks the same

Realism” generally means “ludicrous amounts of detail” — even moreso if the environments are already partially-destroyed, which is a fairly common trope I’ll touch on again.

When everything is highly-detailed, screenshots may look very good, but gameplay suffers because the player can no longer tell what’s important. The tendency for everything to have a thick coating of sepia certainly doesn’t help.

Look at that Call of Duty screenshot again. What in this screenshot is actually important? What here matters to you as a player? As far as I can tell, the only critical objects are:

  • Your current weapon

That’s it. The rocks and grass and billboards and vehicles and Hollywood sign might look very nice (by which I mean, “look like those things look”), but they aren’t important to the game at all. This might as well be a completely empty hallway.

To be fair, I haven’t played the game, so for all I know there’s a compelling reason to collect traffic cones. Otherwise, this screenshot is 100% noise. Everything in it serves only to emphasize that you’re in a realistic environment.

Don’t get me wrong, setting the scene is important, but something has been missed here. Detail catches the eye, and this screenshot is nothing but detail. None of it is relevant. If there were ammo lying around, would you even be able to find it?

Ah, but then, modern realistic games either do away with ammo pickups entirely or make them glow so you can tell they’re there. You know, for the realism.

(Speaking of glowing: something I always found ridiculous was how utterly bland the imp fireballs look in Doom 3 and 4. We have these amazing lighting engines, and the best we can do for a fireball is a solid pale orange circle? How do modern fireballs look less interesting than a Doom 1 fireball sprite?)

Even Fallout 2 bugged me a little with this; the world was full of shelves and containers, but it seemed almost all of them were completely empty. Fallout 1 had tons of loot waiting to be swiped from shelves, but someone must’ve decided that was a little silly and cut down on it in Fallout 2. So then, what’s the point of having so many shelves? They encourage the player to explore, then offer no reward whatsoever most of the time.

Environments are boring and static

Fallout 3 went right off the rails, filling the world with tons of (gray) detail, none of which I could interact with. I was barely finished with the first settlement before I gave up on the game because of how empty it felt. Everywhere was detailed as though it were equally important, but most of it was static decorations. From what I’ve seen, Fallout 4 is even worse.

Our graphical capabilities have improved much faster than our ability to actually simulate all the junk we’re putting on the screen. Hey, there’s a car! Can I get in it? Can I drive it? No, I can only bump into an awkwardly-shaped collision box drawn around it. So what’s the point of having a car, an object that — in the real world — I’m accustomed to being able to use?

And yet… a game that has nothing to do with driving a car doesn’t need you to be able to drive a car. Games are games, not perfect simulations of reality. They have rules, a goal, and a set of things the player is able to do. There’s no reason to make the player able to do everything if it has no bearing on what the game’s about.

This puts “realistic” games in an awkward position. How do they solve it?

One good example that comes to mind is Portal, which was rendered realistically, but managed to develop a style from the limited palette it used in the actual play areas. It didn’t matter that you couldn’t interact with the world in any way other than portaling walls and lifting cubes, because for the vast majority of the game, you only encountered walls and cubes! Even the “behind the scenes” parts at the end were mostly architecture, not objects, and I’m not particularly bothered that I can’t interact with a large rusty pipe.

The standouts were the handful of offices you managed to finagle your way into, which were of course full of files and computers and other desktop detritus. Everything in an office is — necessarily! — something a human can meaningfully interact with, but the most you can do in Portal is drop a coffee cup on the floor. It’s all the more infuriating if you consider that the plot might have been explained by the information in those files or on those computers. Portal 2 was in fact a little worse about this, as you spent much more time outside of the controlled test areas.

I think Left 4 Dead may have also avoided this problem by forcing the players to be moving constantly — you don’t notice that you can’t get in a car if you’re running for your life. The only time the players can really rest is in a safe house, which are generally full of objects the players can pick up and use.

Progression feels linear and prescripted

Portal dodged a huge bullet by not trying to emulate real-world environments, instead setting the player in deliberately-constructed test chambers. Games like Half-Life 2, which takes place in a realistic city, have a problem on their hands: realistic manmade spaces are designed for efficiency and utility, not interesting exploration. A very common trick is to just have the world be partially-destroyed. Apartment buildings are not terribly exciting to explore, so let’s jam half the doors and put a few upturned cars in the street.

This sucks. The design serves the realistic aesthetic, not the player’s experience, and has to be awkwardly contorted to fit into the game. The thing I love about Doom’s “abstract” design is that it can serve the gameplay first and be fit to aesthetics second. Metroid Prime did an excellent job of this as well, and I’d even call its design abstract — it has a lot more detail than Doom, sure, but the rooms are more like handwaved concepts than slices of a space that would reasonably exist. (Who would fill a cave with powered doors, anyway? Who cares?)

But returning to Portal, its main draw is one of my favorite properties of games: you could manipulate the environment itself. It’s the whole point of the game, even. And it seems to be conspicuously missing from many modern “realistic” games, partly because real environments are just static, but also in large part because… of the graphics!

Rendering a very complex scene is hard, so modern map formats do a whole lot of computing stuff ahead of time. (For similar reasons, albeit more primitive ones, vanilla Doom can’t move walls sideways.) Having any of the environment actually move or change is thus harder, so it tends to be reserved for fancy cutscenes when you press the button that lets you progress. And because grandiose environmental changes aren’t very realistic, that button often just opens a door or blows something up.

It feels hamfisted, like someone carefully set it all up just for me. Obviously someone did, but the last thing I want is to be reminded of that. I’m reminded very strongly of Half-Life 2, which felt like one very long corridor punctuated by the occasional overt physics puzzle. Contrast with Doom, where there are buttons all over the place and they just do things without drawing any particular attention to the results. Mystery switches are sometimes a problem, but for better or worse, Doom’s switches always feel like something I’m doing to the game, rather than the game waiting for me to come along so it can do some preordained song and dance.

I miss switches. Real switches, not touchscreens. Big chunky switches that take up half a wall.

It’s not just the switches, though. Several of Romero’s maps from episode 1 are shaped like a “horseshoe”, which more or less means that you can see the exit from the beginning (across some open plaza). More importantly, the enemies at the exit can see you, and will be shooting at you for much of the level.

That gives you choices, even within the limited vocabulary of Doom. Do you risk wasting ammo trying to take them out from a distance, or do you just dodge their shots all throughout the level? It’s up to you! You get to decide how to play the game, naturally, without choosing from a How Do You Want To Play The Game menu. Hell, Doom has entire speedrun categories focused around combat — Tyson for only using the fist and pistol, pacifist for never attacking a monster at all.

You don’t see a lot of that any more. Rendering an entire large area in a polygon-obsessed game is, of course, probably not going to happen — whereas the Doom engine can handle it just fine. I’ll also hazard a guess and say that having too much enemy AI going at once and/or rendering too many highly-detailed enemies at once is too intensive. Or perhaps balancing and testing multiple paths is too complicated.

Or it might be the same tendency I see in modding scenes: the instinct to obsessively control the player’s experience, to come up with a perfectly-crafted gameplay concept and then force the player to go through it exactly as it was conceived. Even Doom 4, from what I can see, has a shocking amount of “oh no the doors are locked, kill all the monsters to unlock them!” nonsense. Why do you feel the need to force the player to shoot the monsters? Isn’t that the whole point of the game? Either the player wants to do it and the railroading is pointless, or the player doesn’t want to do it and you’re making the game actively worse for them!

Something that struck me in Doom’s E1M7 was that, at a certain point, you run back across half the level and there are just straggler monsters all over the place. They all came out of closets when you picked up something, of course, but they also milled around while waiting for you to find them. They weren’t carefully scripted to teleport around you in a fixed pattern when you showed up; they were allowed to behave however they want, following the rules of the game.

Whatever the cause, something has been lost. The entire point of games is that they’re an interactive medium — the player has some input, too.

Exploration is discouraged

I haven’t played through too many recent single-player shooters, but I get the feeling that branching paths (true nonlinearity) and sprawling secrets have become less popular too. I’ve seen a good few people specifically praise Doom 4 for having them, so I assume the status quo is to… not.

That’s particularly sad off the back of Doom episode 1, which has sprawling secrets that often feel like an entire hidden part of the base. In several levels, merely getting outside qualifies as a secret. There are secrets within secrets. There are locked doors inside secrets. It’s great.

And these are real secrets, not three hidden coins in a level and you need to find so many of them to unlock more levels. The rewards are heaps of resources, not a fixed list of easter eggs to collect. Sometimes they’re not telegraphed at all; sometimes you need to do something strange to open them. Doom has a secret you open by walking up to one of two pillars with a heart on it. Doom 2 has a secret you open by run-jumping onto a light fixture, and another you open by “using” a torch and shooting some eyes in the wall.

I miss these, too. Finding one can be a serious advantage, and you can feel genuinely clever for figuring them out, yet at the same time you’re not permanently missing out on anything if you don’t find them all.

I can imagine why these might not be so common any more. If decorating an area is expensive and complicated, you’re not going to want to build large areas off the critical path. In Doom, though, you can make a little closet containing a powerup in about twenty seconds.

More crucially, many of the Doom secrets require the player to notice a detail that’s out of place — and that’s much easier to set up in a simple world like Doom. In a realistic world where every square inch is filled with clutter, how could anyone possibly notice a detail out of place? How can a designer lay any subtle hints at all, when even the core gameplay elements have to glow for anyone to pick them out from background noise?

This might be the biggest drawback to extreme detail: it ultimately teaches the player to ignore the detail, because very little of it is ever worth exploring. After running into enough invisible walls, you’re going to give up on straying from the beaten path.

We wind up with a world where players are trained to look for whatever glows, and completely ignore everything else. At which point… why are we even bothering?

There are no surprises

Realistic” graphics tend to come along with a “realistic” world, and let’s face it, the real world can be a little dull. That’s why we invented video games, right?

Doom has a very clear design vocabulary. Here are some demons. They throw stuff at you; don’t get hit by it. Here are some guns, which you can all hold at once, because those are the rules. Also here’s a glowing floating sphere that gives you a lot of health.

What is a megasphere, anyway? Does it matter? It’s a thing in the game with very clearly-defined rules. It’s good; pick it up.

You can’t do that in a “realistic” game. (Or maybe you can, but we seem to be trying to avoid it.) You can’t just pick up a pair of stereoscopic glasses to inexplicably get night vision for 30 seconds; you need to have some night-vision goggles with batteries and it’s a whole thing. You can’t pick up health kits that heal you; you have to be wearing regenerative power armor and pick up energy cells. You can’t carry eight different weapons; you have to constantly switch between two. Even Doom 4 seems to be uncomfortable leaving brightly flashing keycards lying around — instead you retrieve them from the corpses of people wearing correspondingly-colored armor.

Everything needs an explanation, which vastly reduces the chances of finding anything too surprising or new.

I’m told that Call of Duty is the most popular vidya among the millenials, so I went to look at its weapons:

  • Gun
  • Fast gun
  • Long gun
  • Different gun

How exciting! If you click through each of those gun categories, you can even see the list of unintelligible gun model numbers, which are exactly what gets me excited about a game.

I wonder if those model numbers are real or not. I’m not sure which would be worse.

Get off my lawn

So my problem is that striving for realism is incredibly boring and counter-productive. I don’t even understand the appeal; if I wanted reality, I could look out my window.

Realism” actively sabotages games. I can judge Doom or Mario or Metroid or whatever as independent universes with their own rules, because that’s what they are. A game that’s trying to mirror reality, I can only compare to reality — and it’ll be a very pale imitation.

It comes down to internal consistency. Doom and Team Fortress 2 and Portal and Splatoon and whatever else are pretty upfront about what they’re offering: you have a gun, you can shoot it, also you can run around and maybe press some buttons if you’re lucky. That’s exactly what you get. It’s right there on the box, even.

Then I load Fallout 3, and it tries to look like the real world, and it does a big song and dance asking me for my stats “in-world”, and it tries to imply I can roam this world and do anything I want and forge my own destiny. Then I get into the game, and it turns out I can pretty much just shoot, pick from dialogue trees, and make the occasional hamfisted moral choice. The gameplay doesn’t live up to what the environment tried to promise. The controls don’t even live up to what the environment tried to promise.

The great irony is that “realism” is harshly limiting, even as it grows ever more expensive and elaborate. I’m reminded of the Fat Man in Fallout 3, the gun that launches “mini nukes”. If that weapon had been in Fallout 1 or 2, I probably wouldn’t think twice about it. But in the attempted “realistic” world of Fallout 3, I have to judge it as though it were trying to be a real thing — because it is! — and that makes it sound completely ridiculous.

(It may sound like I’m picking on Fallout 3 a lot here, but to its credit, it actually had enough stuff going on that it stands out to me. I barely remember anything about Doom 3 or Quake 4, and when I think of Half-Life 2 I mostly imagine indistinct crumbling hallways or a grungy river that never ends.)

I’ve never felt this way about series that ignored realism and went for their own art style. Pikmin 3 looks very nice, but I never once felt that I ought to be able to do anything other than direct Pikmin around. Metroid Prime looks great too and has some “realistic” touches, but it still has a very distinct aesthetic, and it manages to do everything important with a relatively small vocabulary — even plentiful secrets.

I just don’t understand the game industry (and game culture)’s fanatical obsession with realistic graphics. They make games worse. It’s entirely possible to have an art style other than “get a lot of unpaid interns to model photos of rocks”, even for a mind-numbingly bland army man simulator. Please feel free to experiment a little more. I would love to see more weird and abstract worlds that follow their own rules and drag you down the rabbit hole with them.

June’s theme is clearing my plate, a concept that becomes increasingly nebulous as time goes by.

I accidentally went nocturnal over the past week, which always leaves me completely fried for a few days due to losing a day. So not the best week, but not the worst either.

  • art: I drew a very quick happy Minccino to try to cheer everyone up last weekend. Their tail fluffs are upside-down. Sorry.

    I drew and colored two old friends on a whim, trying to apply some things Mel had told me about color harmony. I think it came out as possibly one of the nicest things I’ve ever drawn, so, that’s nice, whoa.

    I started drawing daily Pokémon (in a predetermined random order), where the only rule is a time limit of 30 minutes. Here are Wobbuffet and western Shellos; I’ll be dumping them all in a Tumblr tag too.

    While I was out at lunch, I drew on paper for the first time in a while. I should probably do it more, but the results aren’t too bad.

  • Runed Awakening: Aha! I pixelled an NPC or two, possibly marking the first real character design I’ve ever done, as well as an item and an attempt at a room illustration.

    A couple friends played through the current state of the game, which led to a couple days of chasing down extremely obtuse minor bugs. It’s a little frustrating to have spent a lot of time and have so little to show for it, but one of the bugs was something that’s plauged me for over a year, so I guess it at least fits with the theme of getting rid of looming things. From here I should be able to get back to building things.

  • spline: I finally granted Mel the ability to create a new folder by themselves. So far I’ve always done it manually, which has gotten increasingly painful since the folders use the nested set model. There’s still no UI for rearranging them, but this removes a huge source of… requests for manual intervention.

  • veekun: I started reabsorbing the current state of the new YAML schema and thinking about how to get it where I want. Didn’t make any actual progress, though.

    Working on getting the remaining images veekun is missing, too.

Probably more of the same this next week. I want to make huge inroads on Runed Awakening, fix the other category of things I have to do manually for spline, and get some work done on veekun one way or another.

June’s theme is clearing my plate! I will finally stop spreading these vegetables around to make it look like I ate some of them, and just sneak them to the dog under the table.

  • art: I drew a beautiful manga, which is based on true events. I drew a new header image for here, replacing the old poor cutout of a JPEG exported from Pokémon Art Academy, and adjusted the colorscheme of the whole site to match it. And I did a lot of doodling. A loooot of doodling.

  • doom: I did a lot of work on my factory map for DUMP 3, but I don’t think I’m going to make the deadline. I’m okay with that. I did a mad scramble for DUMP 2 in April, then a longer mad scramble for Under Construction in May; I’m alright with not doing a mad scramble for a third month in a row. I can still make maps outside of a particular challenge.

  • twitter: I made @perlin_noise, a Twitter bot that tweets Perlin noise in various forms.

  • pyscss: I’d had a ton of GitHub mail marked unread for ages, so I finally spent a day fixing a lot of easy bugs and released 1.3.5. I don’t plan to spend significant time on this in the future — especially when there’s now a libsass in C and Python bindings to it — but it’s nice to see some obvious problems fixed.

  • blog: It’s been a year since I quit now!

It doesn’t sound like a lot, but honestly most of the time went into the Doom map and artwork, both of which are things I wanted to do this month. I’m not sure how I’ll schedule the mapping if I’m giving up on DUMP 3; maybe I’ll hop on Runed Awakening for a bit and look at the map again later with fresh eyes.

2016-06-12 07:00:00

A year ago today was my last day working a tech job.

What I didn't do

I think I spent the first few months in a bit of a daze. I have a bad habit of expecting worst case scenarios, so I was in a constant state of mild panic over whether I could really earn enough to support myself. Not particularly conducive to doing things.

There was also a very striking change in… people scenery? Working for a tech company, even remotely, meant that I spent much of my time talking to a large group of tech-minded people who knew the context behind things I was working on. Even if they weren’t the things I wanted to be working on, I could at least complain about an obscure problem and expect to find someone who understood it.

Suddenly, that was gone. I know some tech people, of course, and have some tech followers on Twitter, but those groups are much more heterogenous than a few dozen people all working on the same website. It was a little jarring.

And yet, looking back, I suspect that feeling had been fading for some time. I’d been working on increasingly obscure projects for Yelp, which limited how much I could really talk to anyone about them. Towards the end I was put on a particularly thorny problem just because I was the only person who knew anything about it at all. I spent a few weeks hammering away at this thing that zero other people understood, that I barely understood myself, that I didn’t much enjoy doing, and that would ultimately just speed deployments up by a few minutes.

Hm.

When I left, I had a lot of ideas for the kinds of things I wanted to do with all this newfound free time. Most of them were “pure” programming ideas: design and implement a programming language, build a new kind of parser, build a replacement for IRC, or at least build a little IRC bot framework.

I ended up doing… none of those! With more time to do things, rather than daydream restlessly about doing things, I discovered that building libraries and infrastructure is incredibly tedious and unrewarding. (For me, I mean. If that’s your jam, well, I’m glad it’s someone’s.)

I drifted for a little while as I came to terms with this, trying to force myself to work on these grandiose dreams. Ultimately, I realized that I most enjoy programming when it’s a means to an end, when there’s a goal beyond “write some code to do this”. Hence my recent tilt towards game development, where the code is just one part of a larger whole.

And, crucially, that larger whole is something that everyone can potentially enjoy. The difference has been night and day. I can tweet a screenshot of a text adventure and catch several people’s interest. On the other hand, a Python library for resizing images? Who cares? It’s not a complete thing; it’s a building block, a tool. At worst, no one ever uses it, and I have nothing to show for the time. Even at best, well… let’s just say the way programmers react to technical work is very different from the way everyone else reacts to creative work.

I do still like building libraries on occasion, but my sights are much smaller now. I may pick up sanpera or dywypi again, for instance, but I think that’s largely because other people are already using them to do things. I don’t have much interest in devoting months to designing and building a programming language that only a handful of PLT nerds will even look at, when I could instead spend a day and a half making a Twitter bot that posts random noise and immediately have multiple people tell me it’s relaxing or interesting.

In short, I’ve learned a lot about what’s important to me!

Ah, yes, I also thought I would’ve written a book by now. I, uh, haven’t. Writing a book apparently takes a lot more long-term focus than I tend to have available. It also requires enough confidence in a single idea to write tens of thousands of words about it, and that doesn’t come easily either. I’ve taken a lot of notes, written a couple short drafts, and picked up a bit of TeX, so it’s still on the table, but I don’t expect any particular timeframe.

What I did do

Argh, this is going to overlap with my birthday posts. But:

I wrote a whopping 43 blog posts, totalling just over 160,000 words. That’s two or three novels! Along the way, my Patreon has more than tripled to a level that’s, well, more reassuring. Thank you so much, everyone who’s contributed — I can’t imagine a better compliment than discovering that people are willing to directly pay me to keep writing and making whatever little strange things I want.

I drew a hell of a lot. My progress has been documented elsewhere, but suffice to say, I’ve come a long way. I also expanded into a few new media over this past year: watercolors, pixel art, and even a teeny bit of animation.

I made some games. The release of Mario Maker was a really nice start — I could play around with level design ideas inside a world with established gameplay and let other people play them fairly easily. Less seriously, I made Don’t Eat the Cactus, which was microscopic but ended up entertaining a surprising number of people — that’s made me rethink my notions of what a game even needs to be. I made a Doom level, and released it, for the first time. Most recently, of course, Mel and I made Under Construction, a fully-fledged little pixel game. I’ve really enjoyed this so far, and I have several more small things going at the moment.

The elephant in the room is perhaps Runed Awakening, the text adventure I started almost two years ago. It was supposed to be a small first game, but it’s spiraled a little bit out of hand. Perhaps I underestimated text adventures. A year ago, I wasn’t really sure where the game was going, and the ending was vague and unsatisfying; now there’s a clear ending, a rough flow through the game, and most importantly enough ideas to see it through from here. I’ve rearchitected the entire world, added a few major NPCs, added core mechanics, added scoring, added a little reward for replaying, added several major areas, implemented some significant puzzles, and even made an effort to illustrate it. There’s still quite a lot of work left, but I enjoy working on it and I’m excited about the prospect of releasing it.

I did more work on SLADE while messing around with Doom modding, most notably adding support for ZDoom’s myriad kinds of slopes. I tracked down and fixed a lot of bugs with editing geometry, which is a really interesting exercise and a challenging problem, and I’ve fixed dozens of little papercuts. I’ve got a few major things in progress still: support for 3D floors is maybe 70% done, support for lock types is about 70% done. Oh, yes, and I started on a static analyzer for scripts, which is a fantastic intersection of “pure programming” and “something practical that people could make use of”. That’s maybe 10% done and will take a hell of a lot of work, but boy would it be great to see.

I improved spline (the software powering Floraverse) more than I’d realized: arbitrarily-nested folders, multiple media per “page”, and the revamped archives were all done this past year. I used the same library to make Mel a simple site, too. It’s still not something I would advise other people run, but I did put a modicum of effort into documenting it and cleaning up some general weirdness, and I made my own life easier by migrating everything to runit.

veekun has languished for a while, but fear not, I’m still working on it. I wrote brand new code to dump (most of) RBY from scratch, using a YAML schema instead of a relational database, which has grown increasingly awkward to fit all of Pokémon’s special cases into. I still hope to revamp the site based on this idea in time for Sun and Moon. I also spent a little time modernizing the pokedex library itself, most notably making it work with Python 3.

I wrote some other code, too. Camel was an idea I’d had for a while, and I just sat down and wrote it over the course of a couple days, and I’m glad I did. I rewrote PARTYMODE. I did another round of heteroglot. I fixed some bugs in ZDoom. I sped Quixe (a JavaScript interpreter for some text adventures) up by 10% across the board. I wrote some weird Twitter bots. I wrote a lot of one-off stuff for various practical purposes, some of it abandoned, some of it used once and thrown away.

Is that a lot? It doesn’t even feel like a lot. I want to do just as much again by the end of the year. I guess we’ll see how that goes.

Some things people said

Not long after my original post made the rounds, I was contacted by a Vox editor who asked if I’d like to expand my post into an article. A paid article! I thought that sounded fantastic, and could even open the door to more paid writing. I spent most of a week on it.

It went up with the title “I’m 28, I just quit my tech job, and I never want another job again” and a hero image of fists slamming a keyboard. I hadn’t been asked or told about either, and only found out by seeing the live page. I’d even given my own title; no idea what happened to that, or to the byline I wrote.

I can’t imagine a more effective way to make me sound like a complete asshole. I barely remember how the article itself was phrased; I could swear I tried to adapt to a broader and less personal audience, but I guess I didn’t do a very good job, and I’m too embarrassed to go look at it now.

I found out very quickly, via some heated Twitter responses, that it looks even worse without the context of “I wrote this in my blog and Vox approached me to publish it”. It hadn’t even occurred to me that people would assume writing an article for a news website had been my idea, but of course they would. Whoops. In the ensuing year, I’ve encountered one or two friends of friends who proactively blocked me just over that article. Hell, I’d block me too.

I don’t think I want to do any more writing where I don’t have final editorial control.

I bring this up because there have been some wildly differing reactions to what I wrote, and Vox had the most drastic divide. A lot of people were snarky or angry. But just as many people contacted me, often privately, to say they feel the same way and are hoping to quit their jobs in the future and wish me luck.

It’s the money, right? You’re not supposed to talk about money, but I’m an idiot and keep doing it anyway.

I don’t want anyone to feel bad. I tried, actively, not to say anything wildly insensitive, in both the original post and the Vox article. I know a lot of people hate their jobs, and I know most people can’t afford to quit. I wish everyone could. I’d love to see a world where everyone could do or learn or explore or make all the things they wanted. Unfortunately, my wishes have no bearing on how the system works.

I suspect… people have expectations. The American Dream™ is to get a bunch of money, at which point you win and can be happy forever.

I had a cushy well-paying job, and I wasn’t happy. That’s not how it’s supposed to work. Yet if anything, the money made me more unhappy, by keeping me around longer.

People like to quip that money can’t buy happiness. I think that’s missing the point. Money can remove sadness, but only if that sadness is related to not having enough money. My problem was not having enough time.

I was tremendously lucky to have stock options and to be able to pay off the house, but those things cancelled each other out. The money was finite, and I spent it all at once. Now it’s gone, and I still have bills, albeit fewer of them. I still need to earn income, or I’ll run out of money for buying food and internets.

I make considerably less now. I’m also much, much happier.


I don’t know why I feel the need to delve so deeply into this. The original post happened to hit lobste.rs a few days ago, and there were a couple “what a rich asshole” comments, which reminded me of all this. They were subtly weird to read, as though they were about an article from a slightly different parallel universe. I was reminded that many of the similar comments from a year ago had a similar feel to them.

If you think I’m an asshole because I’ve acted like an asshole, well, that’s okay. I try not to, and I’ll try to be better next time, but sometimes I fuck up.

If you think I’m an asshole because I pitched a whiny article to Vox about how one of the diamond lightbulbs in my Scrooge McDuck vault went out, damn. It bugs me a little to be judged as a caricature with little relation to what I’ve actually done.

To the people who ask me for advice

Here’s a more good comment:

The first week was relaxing, productive, glorious. Then I passed the midpoint and saw the end of my freedom looming on the horizon. Gloom descended once more.

I thought I was the only one, who felt like this. I see myself in everything [they] describe. I just don’t have the guts to try and sell my very own software as a full time thing.

I like to liberally license everything I do, and I fucking hate advertising and will never put it on anything I control

It’s almost as if that [person] is me, with a different name, and cuter website graphics.

First of all, thank you! I have further increased the cuteness of my website graphics since this comment. I hope you enjoy.

I’ve heard a lot of this over the past year. A lot. There are a shocking number of people in tech who hate being in tech, even though we all get paid in chests full of gold doubloons.

A decent number of them also asked for my input. What should they do? Should they also quit? Should they switch careers?

I would like to answer everyone, once and for all, by stressing that I have no idea what I’m doing. I don’t know anything. I’m not a renowned expert in job-quitting or anything.

I left because, ultimately, I had to. I was utterly, utterly exhausted. I’d been agonizing over it for almost a year prior, but had stayed because I didn’t think I could pull it off. I was terrified of failure. Even after deciding to quit, I’d wanted to stay another six months and finish out the year. I left when I did because I was deteriorating.

I hoped I could make it work, Mel told me I could make it work, and I had some four layers of backup plans. I still might’ve failed, and every backup plan might’ve failed. I didn’t. But I could’ve.

I can’t tell you whether it’s a good decision to quit your job to backpack through Europe or write that screenplay you’ve always wanted to write. I could barely tell myself whether this was a good idea. I’m not sure I’d admit to it even now. I can’t decide your future for you.

On the other hand…

On the other hand, if you’re just looking for someone to tell you what you want to hear, what you’ve already decided…

Well, let’s just say you’d know better than I would.

June’s theme is, ah, clearing my plate! Yes, we’ll try that again; I have a lot of minor (and not-so-minor) todos that have been hanging over my head for a long time, and I’d like to clear some of them out. I also want to do DUMP 3 and make a significant dent in Runed Awakening, so, busy busy.

  • blog: I published a very fancy explanation of Perlin noise, using a lot of diagrams and interactive things I’d spent half the previous week making, but it came out pretty cool! I also wrote about how I extracted our game’s soundtrack from the PICO-8. And I edited and published a two-year-old post about how I switched Yelp from tabs to spaces! I am on a roll and maybe won’t have to write three posts in the same week at the end of the month this time.

    I did a bit of work on the site itself, too. I linked my Mario Maker levels on the projects page. I fixed a PARTYMODE incompatibility with Edge, because I want my DHTML confetti to annoy as many people as possible. I fixed a silly slowdown with my build process. And at long last, I fixed the   cruft in the titles of all my Disqus threads.

  • gamedev: I wrote a hecka bunch of notes for Mel for a… thing… that… we may or may not end up doing… but that would be pretty cool if we did.

  • patreon: I finally got my Pokémon Showdown adapter working well enough to write a very bad proof of concept battle bot for Sketch, which you can peruse if you really want to. I had some fun asking people to fight the bot, which just chooses moves at complete random and doesn’t understand anything about the actual game. It hasn’t won a single time. Except against me, when I was first writing it, and also choosing moves at complete random.

    I rewrote my Patreon bio, too; now it’s a bit more concrete and (ahem) better typeset.

  • doom: I started on three separate ideas for DUMP 3 maps, though I’m now leaning heavily in favor of just one of them. (I’d like to continue the other two some other time, though.) I did a few hours of work each day on it, and while I’m still in the creative quagmire of “what the heck do I do with all this space”, it’s coming along. I streamed some of the mapping, which I’ve never done before, and which the three people still awake at 3am seemed to enjoy.

  • SLADE: I can’t do any Doom mapping without itching to add things to SLADE. I laid some groundwork for supporting multiple tags per sector, but that got kinda boring, so I rebased my old 3D floors branch and spruced that up a lot. Fixed a heckton of bugs in it and added support for some more features. Still a ways off, but it’s definitely getting there.

  • art: I drew a June avatar! “Drew” might be a strong word, since I clearly modified it from my April/May avatar, but this time I put a lot of effort (and a lot of bugging Mel for advice) into redoing the colors from scratch, and I think it looks considerably better.

  • spring cleaning: Sorted through some photos (i.e. tagged which cats were in them), closed a few hundred browser tabs, and the like.

Wow, that’s a lot of things! I’m pretty happy about that; here’s to more things!

This post is about the thing in the title.

I used to work for Yelp. For historical reasons — probably “the initial developers preferred it” — their mostly-Python codebase had always been indented with tabs. That’s in stark contrast to the vast majority of the Python ecosystem, which generally uses the standard library’s style guide recommendation of four spaces. The presence of tabs caused occasional minor headaches and grumbles among the Python developers, who now numbered in the dozens and were generally used to spaces.

At the end of 2013, I bestowed Yelp with a Christmas gift: I converted their entire primary codebase from tabs to four spaces. On the off chance anyone else ever wants to do the same, here’s how I did it. Probably. I mean, it’s been two and a half years, but I wrote most of this at the time, so it should be correct.

Please note: I do not care what you think about tabs versus spaces. That’s for a different post! I no longer work for Yelp, anyway — so as compelling as your argument may be, I can no longer undo what I have done.

Prerequisites

First, be absolutely sure you’re never going to change your mind. If you as an organization are ambivalent about your whitespace needs, or if you have influential coworkers who will use this post for evil to switch everything back to tabs as soon as you leave, you may wish to reconsider.

Fix mixed indentation

If you’re using a whitespace-sensitive language, you must fix any inconsistent indentation. (You might want to do this anyway, or your code will look like nonsense.) By “inconsistent”, I mean any code that will change relative indentation levels if the width of a tab changes. Consider:

1
2
....if foo:
------->print("true")

If a tab is considered to be eight cells wide, as the good Lord intended, this is fine. But if you’re about to replace all your tabs with four spaces, the extra indentation level vanishes, and suddenly this is invalid Python.

You would not believe how many cases of this I found. I most fondly remember the file that was for some reason indented with n tabs plus a single space, except where it wasn’t. I have no idea how that happened. (Incidentally, the need to micromanage variable-width invisible characters is one of the reasons I wanted to get rid of tabs.) (Please don’t leave comments about this.) (Also consider set shiftround if you use vim, which pretty well eliminates this problem but is tragically under-used.)

On the other hand, if you are some kind of monster and want to replace every tab with eight spaces, then you don’t need to worry about this.

You could just scan your codebase for leading spaces, but if you have a mix of tabbed files and spaced files, you’ll get a ton of false positives and it’ll be a huge pain in the ass. A somewhat more robust approach for Python specifically is:

1
python -tt -m compileall . | grep Sorry

-tt tells the interpreter to treat inconsistent indentation as a SyntaxError. The compileall module searches recursively for .py files and produces .pyc bytecode, which requires parsing each file, which will trigger the SyntaxError. And any errors encountered while compiling modules produce a line starting with Sorry, along with the filename, line number, and column number.

Now you can spend an afternoon fixing those all by hand and trying to figure out why this one file seems to have been written with a tabstop of 3.

Distribute a Git filter definition

The actual process uses a Git filter to enforce that no new tabs find their way into the repository and fix any tabs on in-flight branches. The configuration for which filters to run on which files is stored as part of the repository, but unfortunately, the configuration for what each filter does is not.

One way or another, you must get this block in your devs’ Git configuration — anyone doing regular development who doesn’t have the filter definition will be utterly confused. This is probably the hardest part of the process. Thankfully, Yelp mostly does work on beefy shared dev machines, so I only had to bug an ops person to stick this incantation in /etc/gitconfig and wait for Puppet. YMMV.

1
2
3
4
5
6
[filter "spabs"]
    clean = expand --initial -t 4
    smudge = expand --initial -t 4
    required
[merge]
    renormalize = true

I’ll explain what this all actually does later. Oh, and it might help to actually have expand installed. Most Unix-likes should have it already; if you have developers using Windows, expand is one of the utilities in the unixutils project. Also, BSD (i.e. OS X) expand apparently doesn’t have the --initial argument, but unless you’re in the habit of sprinkling tab characters inside string literals, you can safely leave it off.

Doing it

Here’s the good part.

If at all possible, get all your collaborators to stop doing work for a day while you get this sorted. Christmas works pretty well! I did this on December 26, when we were so short-staffed that there weren’t even any deploys scheduled.

Invoking Git's nuclear option

First, create or amend .gitattributes in the root of your repository with the following:

1
*.py    filter=spabs

You can add as many source-like filetypes as you want by adding more lines with different extensions. I converted everything I could find that we’d used in any repository, including but not limited to: .css, .scss, .js, .html, .xml, .xsl, .txt, .md, .sh, etc. (I left .c and .h alone. It seemed somehow inappropriate to change tabbed C code.)

Here’s a brief explanation. .gitattributes is a magical file that tells Git how to handle the contents of files. The most common use is probably line-ending conversion behavior for projects edited on both Windows and Unix; I’ve also seen some use of it to define language-aware diffing for certain files (i.e. for each hunk, figure out what function it lives in, and put the function name in the hunk’s header line).

What I’ve done here is add a custom filter, i.e. a program run on checkin and checkout. The actual program, expand, was listed in the Git configuration you (hopefully) distributed to everyone. When Git sticks a file in the repository (via add, commit, whatever), it runs the clean filter; when it updates a file on disk based on the repository, it runs the smudge filter. In this case I want to be extra sure there are never any tabs anywhere, so I made both filters do the same thing: convert all leading tabs to four spaces. (The required line from the config will cause Git to complain if expand doesn’t exit with 0 — that means something has gone really wrong.)

This isn’t perfect, as we’ll see later, but it’s some gentle protection against letting tabs sneak into your codebase. I hope we can all agree that mixing tabbed lines with spaced lines is far worse than either tabs or spaces.

If you like, you can commit .gitattributes separately. If you do, DO NOT PUSH YET.

Performing the conversion

I’m paranoid, and Yelp’s codebase was colossal, so I wrote a whole script that inspected every single text-like file in the codebase and manually ran expand on it and very carefully and idempotently adjusted .gitattributes. The nice thing about this was that anyone else could then run the script against one of Yelp’s myriad smaller repositories, without having to understand any of this Git witchcraft. (Gitchcraft?) Unfortunately, I quit and don’t have it any more.

The much faster way to do this is:

1
git checkout HEAD -- "$(git rev-parse --show-toplevel)"

This asks git checkout to re-checkout every single file in your whole repository. As a side effect, the smudge command will be run, converting all your tabs to spaces. You will end up with a whole lot of whitespace changes.

You may want to run your test suite right about now.

Then, commit! As per Yelp tradition when rewriting every single file in the whole codebase, I attributed the commit to Yelp’s lovable mascot Darwin. It stands out better in git blame, and it preserved the extremely critical integrity of my commit stats.

Push to master and you’re done. More or less.

Effects on Git

I think somewhere in the neighborhood of two million lines were affected, and Git handled it surprisingly well. The commit was basically instant, and there weren’t any noticeable performance problems going forward, save for a couple minor wrinkles explained later.

The impact on Git workflow is fairly minimal. Most of the complications will happen to people who are performing casual wizardry anyway and thus probably know a little bit about what they’re doing. Developers who just commit and merge shouldn’t have any problems.

  • A fresh checkout of the repository (or master, at least) will contain spaces, of course, because the checked-in files contain spaces.

  • Any inflight branches will contain tabs, because they haven’t seen the .gitattributes file or the mass conversion commit.

  • Merging an inflight branch with master (in either direction) will transparently convert all the tabs on the branch to spaces before merging. The developer shouldn’t even notice that anything special happened at all.

    This is the magical thing the merge.renormalize setting does. renormalize is an option for the default merge strategy (recursive) that applies filters before performing the merge; the merge.renormalize setting turns it on by default for git merge. Since the merged .gitattributes contains the filter, it gets applied to both sides. I think.

    Note: I don’t know if renormalize works with more exotic merge strategies. I also don’t know what happens if there are merge conflicts within .gitattributes itself.

    Note 2: renormalize does not apply to new files created in the branch — they only exist on one side, so there’s no need to merge them. See below.

  • Rebasing an inflight branch will not work, or more precisely will produce a zillion merge conflicts. merge.renormalize doesn’t apply to git rebase, and there is no rebase.renormalize setting.

    Luckily, you can do the same thing manually with -X. git rebase -Xrenormalize origin/master should work fine.

    -X is supported by all Git commands that do anything resembling a merge, so the same applies to e.g. git cherry-pick or git pull --rebase. You can use it with git merge, as well, but the setting makes it unnecessary.

  • Old stashes probably won’t apply cleanly, and git stash apply tragically ignores -X. I know of two workarounds:

    1. Convert the stash to a branch with git stash branch, then merge or rebase or whatever as above.

    2. Apply the stash manually with, e.g., git cherry-pick 'stash@{0}' -n -m 1 -Xrenormalize. You need the -m 1 (“use diff against parent 1”) because under the hood, a stash is a merge between several distinct commits that hold different parts of the stash, and cherry-pick needs to know which parent to diff against to create a patch. -n just prevents committing, so your stash description of “wip: this doesn’t fucking work” isn’t automatically turned into a commit message.

  • Blame is not, in fact, permanently ruined. git blame -w ignores whitespace-only changes.

  • The total size of your repository will increase, but not by nearly as much as you’d think. Git ultimately stores compressed binary patches, and a patch that contains mostly the same two characters compresses really well. I want to say Yelp’s repository only grew by 1% or so. (The increase may be larger short-term, but git gc will eventually compress it all away.)

Possible fallout

Relatively minor, considering the magnitude of the change. Some short-term, some persisting for the life of your project, sorry.

Old branches that introduced new tabbed files

About a week after the conversion, as developers trickled back from being on vacation, there was a sudden surge of confusion about a phantom file listed in git status. It would be marked as modified, and no amount of git checkout or git reset would make it go away. Everyone with this problem saw the same file marked as modified, but nobody had touched it.

It turned out that someone had had an inflight branch with a newly-created file, indented with tabs. This branch had been merged into master about a week after the conversion, and developers were seeing the new file show up as phantom-modified after their next interactions with master.

The problem was that git dutifully applied the smudge filter when checking this file out, converting it to spaces on-disk… but the copy in the repository still had tabs, making it appear modified. git checkout didn’t fix this because it had caused the problem in the first place: a checkout would again run the filter and produce a modified file. (I suspect this wouldn’t have happened if our clean and smudge had actually been inverses and the repository had remained tabbed, but we explicitly didn’t want that.)

Fixing this was simple enough: I told everyone to just commit the phantom changes in a separate commit whenever this happened. (If the file had also been modified, git diff -w would show a “clean” diff.) The whitespace change would happen in multiple commits, but they’d all merge cleanly as they hit master, since they all contained the same change. Once the checked-in copy of the file contained spaces, the problem disappeared.

I saw a few instances of this over the first few weeks, but they all sorted themselves out as devs committed Git’s whitespace change. I think it could’ve been prevented with a clever git hook that applies filters to new files during a merge, but that would’ve been much more complicated.

Intermittently slow git status

One or two developers saw git status be preposterously slow, taking a minute or more, rather than less than half a second.

Some strace revealed that expand was being run tens of thousands of times. Whoops!

The developers who ran into this ended up making a fresh clone, which mysteriously resolved the issue. My best guess is that we were accidentally hitting the slow path in git status—the solution to the “racy Git” problem. I had to do some guessing here, because the implications aren’t fully described in that documentation, and very few people seem to have ever run into this.

Essentially, Git cheats a bit to keep “is this file changed?” fast: it compares only file stats, like size and mtime. Git has a file called the “index”, which contains, well, the index: it’s a description of what the next commit will look like if you run a plain git commit. The index also remembers which files on disk were modified, as of the last time it was written. So if a file’s mtime is older than the index’s mtime, it’s safe to assume the index is still correct. But it’s also possible that a file was changed in a way that kept its size the same immediately after the index was last written — so quickly that the mtimes for both are identical.

To fix this, if Git sees a file that it thinks is unmodified, and the mtime is exactly the same (or newer, of course) than the index’s mtime, Git will compare the file’s full contents to whatever’s in the index. Naturally, this takes much longer than just stating a bunch of files.

Now imagine someone switches from a very old branch to master. A great many files are updated in the process, but the machine is fast enough that all the updated files and Git’s index end up with the same mtime.

I only discovered this explanation after all the affected developers had given up and recloned, so I never found out for sure whether it was the true case, and I never saw it happen again. But that seems plausible.

If you find yourself with an index with a slow cache, you just need to do something that updates the index. Read-only commands like git status or git diff won’t do it, but git add will. If you really don’t have anything to add yet, you can force an update manually:

1
git update-index somefile

somefile can be any arbitrary file that’s in the repository. This command forces Git to examine it and write its modifiedness to the index—as a side effect, the index will now be updated.

Final cleanup

Once everything has settled, you may want to remove all the filter stuff and just add a pre-commit hook that rejects tabs outright.

You can also tell your developers that they can finally remove all their .vimrc hacks for switching to tabs specifically in your codebase. (Maybe tell them they should’ve been using vim-sleuth.)

Sources: