there is a small pokédex here

Hey, I am Eevee and this is veekun and it's a Pokédex. You probably want to type into that box in the top right, or maybe just start browsing.

Other stuff of interest:

Updates


To promote its debut in the Pokémon Trading Card Game, Zygarde is current being distributed via Nintendo Network for PAL region games.
To promote the debut of Pokémon the Series: XYZ, which was recently added to the Pokémon TV mobile app, Shiny Xerneas, Shiny Yveltal, and Zygarde are being distributed via Nintendo Network for American region games.
The Johto Classic Wi-Fi competition has been announced. Participants will receive a special Pokémon which will be revealed some time before registration for the competition begins.
The Pokémon Jukebox app, a Pokémon music app for Android, will be discontinued over the next two months.

April’s theme was finish Runed Awakening, which I definitely did not do, but I am way further along than I was a month ago. Maybe by the end of May? I don’t want to set a hard deadline when I have no real way to estimate how much work is left, but I don’t want to lose this momentum, either. I’ll keep y’all posted. Hopefully by the end of this week I’ll have made a significant dent in some combination of Runed Awakening and May blogging.

  • Runed Awakening: Implemented padlocks, woohoo. Did some more high-level planning; getting pretty close to a cohesive whole here, at least in theory. Drew a couple more pixel items. Then, uh, kinda didn’t touch it again for most of the week because I had Patreon obligations to take care of, oops.

  • blog: I wrote about dozenal and using Lua for game development, which somehow turned into a dissection of how to embed Lua and Python.

  • art: I spent a couple days drawing after not really doing so much of it for the last few weeks, resulting in some cat complaints, some cat revenge, and my finishing this ancient art trade. At which point two more people asked to trade, sob.

    Also touched up my avatar for May.

  • doom: So there’s a texture pack (Community Chest 4) that was really popular among DUMP 2 mappers, but it’s some 13MB. A good chunk of it is just translations or color substitutions of vanilla textures, which is something ZDoom can already express with a human-readable text format, so I started writing a script to detect and convert as many of these as possible. It turned out to be harder than I was expecting and I kinda bailed halfway through. This is very fascinating, I’m sure.

    I did also poke at anachrony a bit; still trying to settle on a set of weapons/items/monsters so I can actually go build some maps when I have the urge.

  • tech: A friend lost some data from a web-based shop thing and I magicked it back into existence because I’m actually a witch IRL.

  • gamedev: Having admitted I’d never tried PICO-8 in the Lua post, I bought it on a whim and tinkered with it a bit last night, producing this fine non-game. (I worked on it some more today, so it’s better now, but it’s not last week any more so this doesn’t count.)

Nova Dasterin asks, with money:

How about usage of Lua for game development? Love2d etc. Also http://lexaloffle.com/pico-8.php which I recently heard about.

clarification: thoughts on Lua as a ‘good choice’, also Lua vs Python for that subject (gamedev)

There are a couple ways I can interpret this, so I’ll go with: all of them.

Writing an entire game in Lua or Python

This is entirely possible! Eve Online, for example, is allegedly mostly Python.

You can use pygame or cocos2d (built atop pyglet) or a handful of other things. They’re libraries/frameworks rather than dedicated game development studio apps like Unity, and they leave a lot to be desired at times, but they do exist and are maintained and have little communities around them.

Somewhat more well-known is Ren’Py, a visual novel engine that uses Python mixed with a sort of Pythonesque visual novel macro language. That’s not quite the same, but it’s close.

As for Lua, er, I don’t know. Finding this out is pretty hard, since the Internet is flooded with information on engines with Lua embedded. Maybe someone has written OpenGL or SDL bindings for Lua and sprinkled a skeleton of game stuff on top, but if so, I haven’t found it.

But this is probably not what you meant anyway, since you mentioned LÖVE, which does embed Lua.

Embedding Lua or Python

You have a game engine written in a host language, and you want to write a bunch of behavior in a guest language, because you secretly hate your host language but it’s fast at doing math.

What you really want to do is embed a particular language implementation, of course — embedding Jython is going to be a much different experience from embedding CPython. The host language is usually C or something related to C, what with C being our lingua franca (a phrase that refers to how Italian is the universal spoken language), so I assume this is about embedding liblua or CPython.

Embedding a language is a tricky problem. If it’s high-level enough to need “embedding” (rather than “linking to”), it probably has some kind of runtime, which you’ll also need to embed and babysit. At the very least, embedding a language means you have to:

  • start the guest runtime
  • load some guest code
  • convert host data to guest data
  • call some guest functions
  • convert guest data back to host data

In many cases, you’ll also want to expose host functions to the guest runtime, so that guest code can call back into your engine. You might also want to expose entire new types to the guest language, e.g. a fast vector type. That’s a lot of things to worry about, and the difficulty is entirely dependent on how cooperative the guest is.

I admit I’ve never had the need to embed a language in C myself, so don’t take any of this as gospel truth. I’ve been around people who have, though, and hopefully I’ve osmotically absorbed some of their knowledge.

Embedding Lua

I know much less about embedding Lua, but it also seems more straightforward, so it makes a good reference point.

The Lua book, which serves as its documentation, has a chapter that begins:

Lua is an embedded language.

That’s a pretty good start. Indeed, Lua is so embeddable that you can just paste the entire Lua source code into your project — which I know because SLADE has done exactly that. It’s a flat directory of C files totaling 640K, and nothing else is necessary.

As for actually embedding, the book provides this example of using a Lua script for configuration and reading width and height variables out of it:

1
2
height = 600
width = 800
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
#include <lua.h>
#include <lauxlib.h>
#include <lualib.h>

void load (char *filename, int *width, int *height) {
    lua_State *L = lua_open();
    luaL_openlibs(L);

    if (luaL_loadfile(L, filename) || lua_pcall(L, 0, 0, 0))
        error(L, "cannot run configuration file: %s", lua_tostring(L, -1));

    lua_getglobal(L, "width");
    lua_getglobal(L, "height");
    if (!lua_isnumber(L, -2))
        error(L, "`width' should be a number\n");
    if (!lua_isnumber(L, -1))
        error(L, "`height' should be a number\n");
    *width = (int)lua_tonumber(L, -2);
    *height = (int)lua_tonumber(L, -1);

    lua_close(L);
}

An interesting concept in embedded Lua is the stack, a structure that straddles C-land and Lua-land. Invoking Lua operations from C will leave the results in the stack, where C code can then fetch them. In the above code, lua_getglobal is used to put two global variables in the stack, and lua_tonumber retrieves them as C doubles.

Lua’s own value types don’t seem to be exposed to C at all, and there is no “any Lua value” type. Instead, there are a handful of primitive types (nil, boolean, number, string, table, function, userdata, thread) and separate C functions for getting or putting a value of the equivalent C type from or onto the stack.

This means that the Lua-land garbage collector knows exactly what types are examinable by C — they’re the ones on the stack — and it never has to worry about arbitrary C code holding direct references to Lua values.


I haven’t actually used LÖVE! I probably should sometime. I definitely want to give PICO-8 a spin, now that I’m experimenting with pixel art.

For now, my only real experience with embedded Lua comes from tinkering with modding Starbound, where actor behavior is all defined in Lua and new actors can be dropped in alongside the stock ones.

The API isn’t particularly well-documented, and it changes arbitrarily between releases. Starbound still technically hasn’t been released, so this is a little forgivable, but it sure is inconvenient having to reverse-engineer the API surface to get anything done.

Starbound is closed-source, so I don’t know how exactly they embed Lua. From the point of view of a modder, every object has a list of zero or more Lua scripts attached to it, and those scripts are mashed into a single namespace. The namespace can implement various hooks by defining global functions with specific names. There are self and storage tables made available, but there’s no use of Lua’s OO-ish facilities; they’re globals injected into your scripts, corresponding to the entity whose hooks are being executed right now. The hooks back into the engine are, similarly, exposed via a few global tables of functions.

The result is rather goofy, and not what I imagine “native” Lua would look like. If you want to, say, store the ID of your entity when it’s created:

1
2
3
4
5
function init(virtual)
    if virtual then return end

    self.id = entity.id()
end

self is a plain global table. entity is, likewise, a plain global table, but containing some API functions you can use that are specific to the entity you happen to be working with right now.

If a regular library API were implemented like this, it would seem ludicrous. Why should it be any different just because it’s part of a game engine?

This might be the hardest part of embedding any language: making it actually feel native to people who only see the guest side. If you’re going to embed a language, try writing a dummy plugin/script/whatever — before you design the API — the same way you would expect to write it if the entire engine were written in the guest language. Then embed the guest in such a way that it actually works.

To Lua’s credit, it doesn’t look like this ought to be particularly difficult; there’s a whole chapter on exposing custom C types to Lua, and the salient bits are pretty straightforward.

Hm. I just don’t have that much to say about embedding Lua; its own documentation lays out pretty much everything you might want to know. It’s designed for this very purpose, and it shows.

Python

Python is a pretty nice and featureful language, and it is embeddable, yet embedding it seems relatively uncommon.

I’ve asked around about this before, but it was many months ago and I don’t remember who it was or what they said, which is super helpful. I’ll do my best!

I’ve used the CPython C API before (to embed C in Python), and it contrasts very sharply with the Lua API. Where Lua’s API is clearly designed for embedding, CPython’s API is clearly designed for writing CPython. The exposed constructs are the ones used for implementing the standard library and the C parts of third-party libraries; if you want to embed CPython, you’re using the same wide array of fairly fiddly tools.

Python’s embedding documentation doesn’t have an exact analogue for the Lua config example, but I’ve crafted one myself:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
#include <Python.h>

int load(char* modname, int* width, int* height) {
    Py_Initialize();
    PyObject* py_modname = PyUnicode_DecodeFSDefault(modname);
    if (! py_modname) {
        if (PyErr_Occurred())
            PyErr_Print();
        fprintf(stderr, "Couldn't decode module name \"%s\"\n", modname);
        return 1;
    }

    PyObject* py_module = PyImport_Import(py_modname);
    // This function wraps the commonly-used Py_DECREF macro, ensuring that you
    // get the right behavior even if running on a Python that was configured
    // differently, or something?  I dunno I read that somewhere
    Py_DecRef(py_modname);

    if (! py_module) {
        PyErr_Print();
        fprintf(stderr, "Failed to load \"%s\"\n", modname);
        return 1;
    }

    // TODO PyLong_AsLong can actually raise a (Python-land) exception, if the
    // number is too big to fit in a C long...  and of course a long might be
    // bigger than an int!
    PyObject* py_width = PyObject_GetAttrString(py_module, "width");
    *width = (int) PyLong_AsLong(py_width);
    Py_DecRef(py_width);
    PyObject* py_height = PyObject_GetAttrString(py_module, "height");
    *height = (int) PyLong_AsLong(py_height);
    Py_DecRef(py_height);

    Py_DecRef(py_module);

    Py_Finalize();

    return 0;
}

This is definitely, ah, wordier. (In fairness, I suspect the error() function in the Lua example is assumed to be supplied by you.) Most notably, you have to call Py_DecRef all over the place, and you have to understand what’s going on well enough to do it at the right times, or you’ll get a memory leak that’ll suck to track down. And this was just for reading two globals; calling a function is a bit worse.

On the other hand… from what I know of the CPython API, I can’t think of any reason why you couldn’t emulate something like the Lua stack. Define the stack as a structure that only exists in C; wrap the useful parts of the API in versions that interact with the stack instead; write some accessors for the stack that only expose C types instead of PyObject*s everywhere. Then the stack access functions could worry about refcounting, and C code would never have to think about it.

The result would be a bit more complex than Lua’s API, perhaps; Python has a lot of built-in types and a lot of “shapes” of types and overall just a really big API surface. Still, if you’re going to embed Python, I would definitely recommend wrapping up all the C API stuff in a simpler layer like this. I’m actually kind of surprised such a wrapper API doesn’t already exist.

That aside, there’s a much bigger problem with the above code, which is that it doesn’t actually work. I compiled it in a directory alongside this config.py file, then ran ./a.out config.

1
2
height = 600
width = 800

Result: ImportError: No module named 'config'.

Ah. sys.path is wrong. Regular ol’ Python will automatically add the current directory to sys.path if you do something like python -m config, but it seems the C API will not. And I… can’t figure out how to fix it. I can set the path manually to ., but then Python can’t find any of its own built-in modules any more, and nothing works at all — even decoding a bytestring requires the standard library. I could manually add . to the path after starting the interpreter, but surely that’s not the right way to fix this, right? A workaround is to run it as PYTHONPATH=. ./a.out config, but boy howdy is that ugly.

The problem isn’t so much this particular gotcha, but the wider issue that Python is designed as if it controlled the entire process. Its state is mostly global variables; it reads environment variables; it assumes it can look for modules as real files in standard locations. (Lua’s standard library is baked into its C, for better or worse.)

If you want to embed Python, you’ll need to think about how to map features like module importing — which has a bunch of implicit behavior perfectly suited for a standalone interpreter — to something more appropriate for your application. Where is Python’s standard library? Python can import from a zip file, and my compiled program’s sys.path even begins with /usr/lib/python35.zip (which doesn’t exist?), so this isn’t a terribly difficult problem to solve, but it’s something you have to think about from the get-go that you can pretty much ignore with Lua.

Also, do you want the entire standard library? I wouldn’t recommend eliminating all of it (though that does seem to be possible), as it’s a large part of the appeal of Python, but does embedded code need to be able to fork or start threads or spawn a GUI? Probably not, but now you have the tricky problem of sifting through some 200 modules to figure out what’s necessary and what’s not. If you don’t want embedded code to be able to read or write arbitrary files, you also need to strip out some builtins. Good luck.

Again, there’s nothing really stopping anyone from doing all this work for you and making it available as a wrapper. But no one has. Maybe I’m wrong, and the comments are full of links to projects that do exactly this! I sure hope so. If not, and you decide to build such a thing for your own project, please share it with the world.

Other potential roadbumps that come to mind:

  • You probably don’t want to start and stop the interpreter all the time, since Python has a lot of default state that it would have to create and destroy repeatedly.
  • Having multiple interpreters is possible, using “sub-interpreters”, but care is required — nothing prevents passing a PyObject* from one interpreter into another, for example.
  • I’ve heard that it’s difficult to use Python modules written in C from embedded Python, though I don’t know the details.
  • Python supports (OS) threads, so there are things to be aware of if you call into Python from different threads. I don’t know what those things are, exactly.

Other ways to embed Python

CPython’s API is not the only option. Perhaps one of these will be more helpful.

Boost.Python is a common suggestion, but it’s mostly intended for embedding C++ in CPython. It does have some support for embedding, but the second paragraph mentions “you’ll need to use the Python/C API to fill in the gaps”, which is not a particularly good sign. Still, this example code is much more encouraging than the equivalent using the C API:

1
2
3
4
object main_module = import("__main__");
object main_namespace = main_module.attr("__dict__");
object ignored = exec("result = 5 ** 2", main_namespace);
int five_squared = extract<int>(main_namespace["result"]);

In fact, I can rewrite my C example as something like this, which I have not bothered to test:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
#include <boost/python.hpp>

void load(string modname, int* width, int* height) {
    using namespace boost::python;

    object py_module = import(modname);

    *width = extract<int>(py_module.attr("width"));
    *height = extract<int>(py_module.attr("height"));
}

Certainly much better. No reference counting in sight; the object wrapper type takes care of it. Python exceptions are (partially) mapped to C++ exceptions, so the caller can worry about them. You’re not saved from the runtime concerns like where the stdlib lives, but your actual glue code is much less terrible.


Cython is another project originally intended for writing extension modules — it uses a modified Python syntax that compiles to C code using the Python API, and can operate on both Python and C objects interchangeably.

The good news is, it’s possible to embed the entire CPython interpreter with Cython! The bad news is, the documentation appears to consist solely of a one-page wiki article, a few mailing list threads, and a demo.

I do see a glimmer of real potential here. The part you compile is more or less just Python, which means you can write all your loading code in Python. That reduces the amount of C or C++ glue junk you have to write to nothing, an infinity-percent improvement.

Unfortunately, Cython produces a binary that just directly executes the compiled module. From what’s provided, I don’t see how you’re supposed to produce a C API you can call at will from other C code. So close! If only some project had done the same thing but actually finished and documented it.


CFFI did!

CFFI is a pretty slick library, not for writing extension modules, but for calling C library functions from Python. It’s kinda like ctypes, but with a less atrocious API and the ability to learn the C library’s API from its header files.

I found out yesterday that it now also supports embedding Python. It’s the same general idea as with Cython: you write some Python and it gets built into a shared library. But here, you can declare Python wrappers for C functions and C wrappers for Python functions, giving you an explicit list of interfaces between the environments. Particularly nice about this is that CFFI is already designed for wrapping C libraries, so you have a good shot at creating a Python API that actually feels Python-native.

As an added bonus, CFFI is built into PyPy, the JITted Python implementation, so you can use that instead of CPython if you like.

Alas, this still doesn’t save you from dealing with paring down and locating the stdlib. Someone should really figure that out once and for all, then tell the rest of us.

Conclusions

Lua is certainly the simpler option. You don’t even have to link to Lua; you can just drop the whole damn thing into your codebase and forget about it. The API is already fairly simple to use, which leaves you lots of spare dev time for making your API not be a few bags of injected globals.

On the other hand, if you really like Python, you might be willing to deal with its few extra headaches to get it embedded. It’s not as common as Lua, but it certainly can be done. The best option at the moment looks to be CFFI, which has some pretty clever people behind it who might be able to give you some pointers.

What a letdown. I really wanted to root for Python here. If only… if only there were some other way…

A third option: embedding in Python

If you ask the #python IRC channel about embedding Python in an application, you will almost always get the same response: don’t.

Instead, they’ll tell you, you should embed your application in Python.

Now, this isn’t always practical. If your application is intended to support plugins in multiple different languages, this is perhaps not the best approach. But if you only ever intend to support Python, and if you wish you could be writing Python anyway, this is entirely doable.

Remember that the biggest issue with embedding Python is that it’s designed assuming it controls the entire process. It has global state and tendrils into the system all over the place. In contrast, your application is probably less invasive, and you have complete control over it besides. So why not hand the wheel to Python?

Build your application like a library, then call its entry point from Python using one of the C embedding gizmos, like CFFI. You can pass Python callbacks into your C code with def_extern, or do the bulk of application work in Python and keep the heavy lifting in C, or whatever.

Then you can bundle and deploy the whole thing with any of the existing Python-to-executable tools: cx_Freeze, py2exe, py2app, PyInstaller, etc. (This isn’t a thing I have to do, so I don’t know which is the latest and greatest.) Or, hell, compile all your Python in Cython’s embedded mode.

This is a pretty radical departure from what people tend to mean when they say “embedding”, but it has the major benefit that there’s just far more C-in-Python tooling available than Python-in-C.

Alas! You still have to worry about paring down the standard library and builtins yourself, and that’s really tricky from pure Python. If you want arbitrary mods to be relatively safe for players to use, and you are not an expert on Python internals (I’m not!), you might have to stick with Lua.

A fourth option

Godot took the rather radical approach of creating their own vaguely Pythonesque language, GDScript.

Obviously this is a hell of a lot of work, especially if you’ve never designed a language before. On the other hand, the control over the embedding API and native types was apparently reason enough for the Godot developers. As I’ve previously mentioned, most programming languages are bad at expressing simulation, so a new language can be a chance to design some more appropriate syntax and types. (I can’t speak to whether GDScript has pulled this off.)

It’s not unheard of, either. RPG Maker had its own language before switching to Ruby a few years ago. The original Quake had QuakeC, though that was dropped in favor of native libraries in later engines. Unreal had its own language before… uh… ditching it two years ago. Hmm, maybe there’s a pattern here.

One thing GDScript does have going for it is that it kinda solves the problem of getting scripts to feel “native”, since the language itself is just designed to be native. In particular, scripts are associated with nodes, and everything in a script file is implicitly part of that node’s type (or class, if you must) in a way that plays pretty well with Godot’s concept of instancing.

I definitely wouldn’t advise designing your own language without a really good reason, especially not for just a single game. Language design is hard — most languages are still evolving to fill their gaps decades after they were first conceived.

But hey, if you do go down this road, for the love of god, please don’t base your language on Java or C. That gets you the worst of both worlds: all the introspection of a static compiled language with all the tooling of a dynamic language.

Appendix: the languages themselves

I don’t want to get into a huge language fight here. Lua is pretty good for what it is, which is an embedded dynamic language in 640KB of ultraportable C and nothing else. It has some goofy decisions, like using ~= for inequality and indexing arrays from 1, but whatever. It also has coroutines and metatypes, which is impressive. I only have a couple real beefs with Lua, and they’re mostly grievances I have with JavaScript as well:

  • There’s only one numeric type (number, double-precision floating point) and only one data structure (table). Ehh. You can return multiple values from a function or iterator, but there’s no real tuple type, and misusing a faux tuple will silently discard the unconsumed values.

  • Reading a nonexistent variable is not an error; it produces nil. Likewise (actually for the same reason), reading a nonexistent table entry silently produces nil. Writing to a variable that’s not declared local will silently create a new global.

  • If a function receives too many arguments, the extras are silently ignored. If too few, the missing ones default to nil.

  • There’s nothing like Python’s repr built in, let alone pprint. A table prints as table: 0xa6c300, without even delimiters around it. This is terrible — half the appeal of a dynamic runtime is the ability to inspect the state of your program from within itself, but the only data structure the language even has is opaque.

  • break and return have to be the last statement in a block. There’s a trivial workaround (wrap it in a dummy do ... end) but it’s still kind of annoying. Also there’s no continue.

  • You need the built-in debug library to do a lot of useful things, like print Lua tracebacks, but the Lua book advises disabling it in production builds because it allows Lua code to do naughty things. Indeed, Starbound doesn’t have debug. Trouble is, a production build of Starbound is the development environment for a mod.

These issues are compounded when Lua is embedded in a larger engine with a poorly-documented API, ahem. Mistakes like nils and typoed variable names can persist for several lines or several functions before causing any obvious errors, at which point you have to reload the entity (or as is oftne the case with Starbound, reload all the game assets) for every attempted diagnosis or fix.

Contrast with Python, which is perfect in every way.

Well, er. The biggest drawback with Python is the one I’ve already mentioned: it’s a general-purpose language which exposes a lot of general-purpose system functionality. If you don’t want your scripts to be able to start a web server, you’ll have to trim the language down into something more appropriate. (On the other hand, it looks like people are making use of sockets in Blender…) And what of the other things Python developers take for granted, like imports or the logging module? Do you support third-party modules? How? Do you respect Python’s environment variables? Where does print go? What do you do with all the goodies in the sys module — some useful, some nefarious, some a bit of both? How do you insulate scripts from each other; an entire sub-interpreter for each one? Did you know that CPython lets you construct, modify, and execute arbitrary bytecode? Do you need to protect against that? Or against using ctypes to call the CPython C API from Python code?

One other thought: Python has syntactic async support as of 3.5, which sounds like something that might be pretty useful in a game engine, but I have no idea how you would actually embed code that uses it. I’d sure be interested to hear if you got it to work, though.

Decimal sucks.

Ten is such an awkward number. Its only divisors are two and five. Two is nice, but five? Who cares about five? What about three and four?

I have a simple solution to all of these non-problems and more, which is: we should switch to base twelve.

A brief recap

For ease of writing larger numbers, we use places. The number “123” really means: three ones, two tens, one hundred. “Hundred”, of course, is just what we call ten tens.

This all assumes that ten is a magical number. It’s not. We generally have ten fingers, which makes it convenient. Mathematically, though, ten is not particularly special. We could just as well use a different number, like two (which gives us binary), or seven, or twelve. The number we pick, the number where the ones place gets full and rolls over, is called the number base.

We use decimal, base ten. This post is about duodecimal, base twelve.

It turns out that there are people who advocate for this. I haven’t read any of their writing, instead opting to do my own research. The one tidbit I’ve picked up is that they insist that base twelve be called dozenal, as the name “duodecimal” is still in terms of ten.

So, in dozenal, “123” means: three ones, two dozens, one gross. You know that number as one hundred and seventy-one.

To avoid confusion, all numbers in this post that are written with digits will be in dozenal. Words are independent of base — “thirteen” always means the same thing — so I’ll use those to translate.

Extra digits

Base ten has ten digits, so base twelve needs twelve digits. I need to pick some.

There are in fact already two dozenal digits in Unicode 8.0: ↊ U+218A TURNED DIGIT TWO and ↋ U+218B TURNED DIGIT THREE. Unfortunately, since these are very recent, font support is lacking.

There’s also the consideration of 7-segment LCD displays. An upside-down 3 would work fine and would resemble a capital E, but an upside-down 2 would look the same as a 5.

I mulled this over for a while and came up with:

  • ƌ for ten — resembles “d” for decimal, and displays in 7-segment like a mirrored 6
  • ⅃ for eleven — resembles a rotated 7, which it also rhymes with

No? Fair enough. These are all pretty awkward to type, anyway, even for me. I could use the more self-explanatory ⑩ and ⑪… but perhaps not.

For now, I’ll go with the old standby and use A to mean ten, B to mean eleven.

The numbers from one to twenty are thus written: 1, 2, 3, 4, 5, 6, 7, 8, 9, A, B, 10, 11, 12, 13, 14, 15, 16, 17, 18.

Got it? Super.

Twelve in daily life

We’re already surrounded by twelves. This year is 1200 in dozenal!

There are twelve months in a year. (Leap years are every four years, so in dozenal, a leap year would always be a year ending in 0, 4, or 8.)

There are twelve hours on the clock and sixty minutes in an hour (which is 50 — five dozen). Dozenal would let us save an entire digit on clock faces, since the hour will never be two digits; the moment before midnight is B:4B.

There are 10 (twelve) inches in a foot, 3080 feet in a mile (easily rounded to an even 3000). If metric were based on dozenal, the biggest part of the squabble between metric and imperial units would largely disappear.

Donuts come in dozens. It’s there in plain sight: a donut resembles a zero, and one dozen is written 10.

Teenagedom would begin at 10, and adulthood would be at 16, halfway to 20. No one would ever have lived to 100, though.

We have unique names for numbers from one to twelve; it’s only at thirteen that patterns start to emerge. We also already have a name for twelve twelves.

I had to learn times tables up to twelve in elementary school, which makes much more sense if twelve is written as 10.

Twelve digits would fit nicely in a 3×4 grid. Adding a fifth row for symbols (pound, star, something else?) would make the grid 3×5, which with square buttons is pleasingly close to the golden ratio.

A full circle is three hundred sixty degrees, which is a multiple of twelve. It’s also two and a half gross, i.e. 260.

Numbers would be eight percent shorter on average. (Dozenal can express everything up to one hundred forty-three with only two digits.)

Twitter might have rounded the number of available characters up from B8 to an even 100.

But wait!” I hear you cry. “Twelve might be great, but how do we count to it when we only have ten fingers?”

No problem! Check out your hands. Each of your fingers has two knuckles, which divide it into three segments. Three segments times four fingers makes… twelve! And your thumb is left over for keeping track of where you are. You can actually count higher with one hand in dozenal than with both hands in decimal! If you use your other hand for the dozens place, you can count all the way to 100 (a gross, one hundred forty-four) just on your fingers.

Arithmetic

Is exactly the same, but with two more digits. Carrying and whatnot are the same regardless of base; the only difference is that 9 + 1 is the single-digit A rather than wrapping around to two digits.

You can do long subtraction:

1
2
3
4
5
  23
  3̸4̸2
- 173
-----
  18B

Or long multiplication:

1
2
3
4
5
6
7
8
  342
× 173
-----
  A06
1B52
342
-----
54526

Or long division… which I will not attempt to Unicode-art here.

Fractions

Twelve is what’s called a highly composite number — it’s the smallest number with five divisors. (One, two, three, four, and six.) Ten, on the other hand, has three divisors (one, two, five), making it only as composite as six. What a waste.

The number of divisors is important, because it allows for many more “natural” divisions. Consider that most simple fractions in decimal repeat forever, making them ugly and cumbersome to work with. Well, check out some fractions in dozenal.

  • ½ = 0.6
  • ⅓ = 0.4
  • ⅔ = 0.8
  • ¼ = 0.3
  • ¾ = 0.9
  • ⅙ = 0.2
  • ⅚ = 0.A
  • ⅛ = 0.16
  • ⅜ = 0.46
  • ⅑ = 0.16

True, one-fifth is written as the somewhat less-pleasing 0.2̅4̅9̅7̅. But, honestly, who cares about fifths? Fifths are usually a cheap imitation of sixths, which are the beautiful 0.2 in dozenal.

You can see how nice this scheme really is when you consider it in terms of percentages. Decimal percentages are a little hard to reckon with, since so few of them divide evenly into one hundred — what does thirteen percent mean, exactly? (About one-eighth. You can’t even express a pizza slice as a whole percentage!) On the other hand, all of the above fractions can be expressed as whole dozenal percentages: one third is 40%, one half is 60%, one twelfth is of course 10%. Just look how many dozenal percentages ending in zero are really simple fractions.

Let’s say a percentage is “satisfying” if it can be converted into a fraction where at least two primes cancel out. For example, seventeen percent is just seventeen hundredths, which is unsatisfying; but thirty-two percent is eight twenty-fifths, which is at least partly simplified.

By this definition, only 28 whole decimal percentages are satisfying — that’s 3A%. But a whopping 48 whole dozenal percentages are satisfying, which is of course 48%! 48% is itself even satisfying; it’s seven eighteenths.

Divisibility rules

These are tricks like, “if the sum of the digits is divisible by three, the original number is divisible by three”. Dozenal has a different set of digits, so the same tricks won’t work.

However, the new tricks are often much easier. I went ahead and figured some out:

  • 2: If the last digit is divisible by two (0, 2, 4, 6, 8, or A), the number is divisible by two.

  • 3: If the last digit is divisible by three (0, 3, 6, or 9), the number is divisible by three.

  • 4: If the last digit is divisible by four (0, 4, or 8), the number is divisible by four.

  • 5: Multiply the last digit by two, and subtract from the rest of the number. If the result is divisible by five, the original number is divisible by five.

    Here’s an example, for the number B3 (one hundred thirty-five).

    Take off the 3 and double it to get 6. The rest of the number is B (eleven). B - 6 = 5, which is divisible by 5, so B3 is also divisible by 5.

  • 6: If the last digit is divisible by six (0 or 6), the number is divisible by six.

  • 7: Multiply the last digit by three, and add it to the rest of the number. If the result is divisible by seven, the original number is divisible by seven.

  • 8: If the last two digits are divisible by eight (0, 8, 14, 20, 28, 34, …), the number is divisible by eight.

    Alternatively, but more of a mouthful, you can see a pattern in the numbers above. If the last digit is 0 or 8 and the dozens digit is even, OR the last digit is 4 and the dozens digit is odd, the number is divisible by eight.

  • 9: If the last two digits are divisible by nine (0, 9, 16, 23, 30, …), the number is divisible by nine.

    Again, there’s a pattern in the numbers, though this one is more complicated.

  • A (ten): Must be divisible by both two and five.

  • B (eleven): Add all the digits. If the result is divisible by eleven, the original number is divisible by eleven.

  • 10 (twelve): If the last digit is 0, the number is divisible by twelve.

  • 11 (thirteen): From left to right, alternate subtracting and adding the digits. If the result is divisible by thirteen, the original number is divisible by thirteen.

    Here’s an example, for 891 (one thousand two hundred sixty-one, or thirteen times ninety-seven).

    Compute 8 - 9 + 1; it’s zero, which is divisible by thirteen, so the original number is too.

  • 12 (fourteen): Must be divisible by both two and seven.

  • 13 (fifteen): Must be divisible by both three and five.

  • 14 (sixteen): If the last two digits are divisible by sixteen (0, 14, 28, 40, 54, …), the number is divisible by sixteen.

The primes start to get trickier after that.

Dozenal and computers

A double-precision floating point number has fifty-three binary digits. That’s only (barely) sixteen decimal digits of precision. In dozenal, it’s slightly worse at almost fifteen digits, since each dozenal digit stores more information.

A byte is 8 bits and can hold a value up to 193. Two bytes is 14 bits and maxes out at 31B13 (a lovely palindrome!). Four bytes is 28 bits, max 9BA461593. Eight bytes is 54 bits, max 839365134A2A240713. Six bytes is, pleasingly, 40 bits.

A kilobyte (kibibyte, if you must) is 714 bytes… but of course, we wouldn’t define it that way if we used dozenal. We like one thousand twenty-four because it’s a power of two that’s coincidentally very close to a power of ten. This doesn’t really happen in dozenal until 2¹⁶ = 107854. But hey, that means we’d never have had this ridiculous inconsistency in advertised hard drive space; we would’ve just said a kilobyte is 1000 bytes from the beginning. If so, a two terabyte hard drive would be described as a paltry 2B6 gigabytes.

With two extra digits, ASCII would have two fewer slots available for punctuation. I wonder which two wouldn’t have made it?

Unicode would have space for 458,8A7 characters.

Some familiar numbers

The square numbers up to 20 are 1, 4, 9, 14, 21, 30, 41, 54, 69, 84, A1, 100, 121, 133, 169, 194, 201, 230, 261, 294, 309, 344, 381, 400.

The primes up to 200 are 2, 3, 5, 7, B, 11, 15, 17, 1B, 25, 27, 31, 35, 37, 3B, 45, 4B, 51, 57, 5B, 61, 67, 6B, 75, 81, 85, 87, 8B, 91, 95, A7, AB, B5, B7, 105, 107, 111, 117, 11B, 125, 12B, 131, 13B, 141, 145, 147, 157, 167, 16B, 171, 175, 17B, 181, 18B, 195, 19B, 1A5, 1A7, 1B1, 1B5, 1B7. Notably and pleasingly (to me, anyway), 101 is composite: it’s 5 × 25.

π = 3.184809493B918664573A

τ = 6.349416967B635108B279

e = 2.8752360698219BA71971

√2 = 1.4B79170A07B85737704B

φ = 1.74BB6772802A46A6A186

It’s suspected that π is a normal number — that is, it has an even distribution of all possible digits no matter what base you write it in.

Okay uh

This is less of a case for base twelve and more that I wanted to write a bunch of math stuff and play in a different base. This post isn’t really going anywhere in particular, so I’ll arbitrarily cut it off here. I hope you enjoyed this hazy thought experiment, if you read this far.

I like to remember sometimes that even many of the things we take for granted — like the way we write numbers — are still arbitrary conventions.

Decimal does suck a bit, though.

Appendix: convert to a base in Python

Curiously, Python can convert integers from any base, but can’t handle floats or convert to an arbitrary base. So here is some code that can, and that also handles negative numbers and floats. And fractions.Fraction. And decimal.Decimal. And complex numbers.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
def tobase(n, b, precision=20, digits='0123456789ABCDEFGHIJKLMNOPQRSTUVWXYZ'):
    # Split off any imaginary part to be dealt with later
    imag = n.imag
    n = real = n.real

    # Handle negation separately
    neg = False
    if n < 0:
        n *= -1
        neg = True

    # Split off any fractional part
    n, frac = divmod(n, 1)
    n = int(n)

    # Convert to base b.  Yep, that's all it takes
    out = []
    while n:
        n, r = divmod(n, b)
        out.append(digits[r])

    if neg:
        out.append('-')

    # Converting to a base moves away from the decimal point, so these digits
    # are in reverse order and need flipping before dealing with a fraction
    out = out[::-1]
    if frac:
        # Leading zero if necessary
        if not out:
            out.append(digits[0])
        out.append('.')
        for _ in range(precision):
            if not frac:
                break
            d, frac = divmod(frac * b, 1)
            out.append(digits[int(d)])

    # Add any imaginary part, keeping in mind that there might not even be a real part
    if imag:
        imagres = tobase(imag, b, precision=precision, digits=digits)
        if not real:
            return imagres + 'j'
        if imag > 0:
            out.append('+')
        out.append(imagres)
        out.append('j')

        # Python's repr wraps fully complex numbers in parentheses
        if real:
            out.insert(0, '(')
            out.append(')')

    return ''.join(out)

Appendix: deriving divisibility rules

How did I figure out the rules above? I’m glad you asked because that’s actually super interesting.

The reason we have divisibility rules at all is that someone has already done a bunch of division for you, by virtue of writing the number out in digits. That’s how you convert an abstract number to something you can write out: you divide repeatedly by twelve, or ten, or whatever base you’re using. A divisibility rule is a trick for taking that work and adapting it to a different divisor.

Easy cases

The easiest cases are for divisibility by d, where the base itself is already divisible by d. You only have to look at the last digit. That’s why picking out even numbers in both decimal and dozenal is so easy — the base is divisible by 2, so the last digit gives away whether an entire number is divisible by 2.

I think this is pretty intuitive. Count up by twos: 2, 4, 6, 8, A, 10. Every sixth number will roll over the ones place into the dozens place, and reset the ones place to zero, which is where it started. Counting like this will eventually find every multiple of two… so every number ending in an even digit must be divisible by two, and no number divisible by two can ever end in an odd digit.

Symbolically, you can write any natural number as:

1
n = 10x + y = (2 · 6x) + y

where y is the ones digit and x is all the rest of the digits. This is just the definition of how we write numbers; 984 is really 98 · 10 + 4.

x must be an integer, since it’s just digits, so 2 · 6x must be even. Iff y is also even, the original number must be even.

You can easily do the same thing for any number that’s a factor of the base. For decimal, that’s two, zero, and ten; for dozenal, it’s two, three, four, six, and twelve. Consider for a moment that that means if we used an odd base, you couldn’t easily tell whether a number were even! It’s only trivial in decimal because ten is a multiple of two.

You can also extend this to work for any number that’s a factor of a power of the base. Dozenal’s divisibility rule for 9 is to look at the last two digits of the number. That’s because:

1
n = 100x + y = (9 · 14x) + y

This time, y is the last two digits, but otherwise it’s the same idea. There’s a similar rule in decimal for divisibility by four (though you only need one digit to work that out in dozenal).

Special cases

Most shenanigans that involve looking at individual digits actually work just as well in dozenal, or any base, but perhaps with different results. Of particular interest: in base b, the same divisibility rules for b - 1 and b + 1 will always work.

Adding all the digits works for b - 1. To prove this, consider a simpler case: instead of adding all the digits together, add the last digit (y) to the rest of the digits (x) to make a new number, s. Then:

1
2
3
4
n = bx + y
s = x + y

n - s = (bx + y) - (x + y) = bx - x = (b - 1) · x

n - s is always divisible by b - 1. Thus, if s is also divisible by b - 1, the original number must be — because it’s just the sum of these other two numbers.

Summing all the digits is just a matter of repeating the above process.

Incidentally, the decimal rule for 3 exists specifically because 3 is a factor of 9, one less than ten. (Eleven is prime, so this doesn’t come into play with dozenal.) You can kind of see how it works above. Hint: n and s have the same remainder when dividing by b - 1.

b + 1 has several possible rules; I used the one I like the most, which is to alternate adding and subtracting. Note that it doesn’t matter whether you start out subtracting or adding; x - y + z is just the negation of -x + y - z.

If we split off two digits separately this time, we get:

1
2
3
4
5
6
n = b²x + by + z
s = x - y + z

n - s = (b²x + by + z) - (x - y + z)
      = (b² - 1)x + (b + 1)y
      = (b + 1)(b - 1)x + (b + 1)y

Same thing happens here: n - s is always divisible by b + 1.

Hard cases

Decimal doesn’t have a “hard” case until seven; dozenal has one as early as five. I basically reverse-engineered the decimal rule for seven to figure out the rest.

Let’s play with my rule for five (multiply the last digit by two, and subtract from the rest of the number) and see what happens.

1
2
3
4
n = 10x + y
s = x - 2y

2n + s = 2(10x + y) + (x - 2y) = 21x

Aha. 21 is twenty-five, which is divisible by the number we’re interested in, five. So 2n + s is always divisible by five. If s is also divisible by five, then 2n must be… and the 2 isn’t contributing to divisibility here, so the original number must also be divisible by five.

I came up with this by looking for a number c that would work out nicely:

1
2
3
4
5
n = 10x + y
s = x + cy

cn - s = (10c - 1)x
s - cn = (-10c + 1)x

For either of those expressions to be divisible by five, I need a number ending in either 1 or B that’s a multiple of five. I listed out the multiples of five until I found one: 5, A, 13, 18, 21, aha! That gives me c = -2.

This works for any (prime…ish…) number, though the values of c start to get a little ridiculous if the number you want to test is very large. I did come up with a divisibility rule for nineteen, for example, by counting multiples: 17, 32, 49, 64, 7B. c = 7, and you just need to multiply the last digit by seven and add it to the rest of the number, then repeat until you can tell whether it’s a multiple of nineteen. Yikes.

Also, note that the expressions above don’t actually do any math with 10, which is really b, the base you’re working in. Exactly the same approach would get you the decimal divisibility rule for seven — seven times three is twenty-one, so c is -2, and the rule is… coincidentally, exactly the same as dozenal’s rule for five.

April’s theme is finish Runed Awakening.

Mel is out of town, so I’m being constantly harassed by cats.

  • irl: I cleaned the hell out of my room.

  • Runed Awakening: I finally figured out the concept for a decent chunk of the world, and built about half of it. I finally, finally, feel like the world and plot are starting to come together. I doubt I’ll have it feature-complete in only another week, but I’m miles ahead of where I was at the beginning of the month. The game is maybe 40% built, but closer to 80% planned, and that’s been the really hard part.

    I’d previously done two illustrations of items as sort of a proof of concept for embedding images in interactive fiction. A major blocker was coming up with a palette that could work for any item and keep the overall style cohesive. I couldn’t find an appropriate palette and I’m not confident enough with color to design one myself, so the whole idea was put on hold there.

    Last week, a ludum dare game led me to the DB32 palette, one of the few palettes I’ve ever seen that has as many (!) as 32 colors and is intended for general use. I grabbed Aseprite at Twitter’s suggestion (which turns out to default to DB32), and I did some pixel art, kind of on a whim! Twitter pretty well destroyed them, so I posted them in a couple batches on Tumblr: batch 1, batch 2, and a little talking eevee portrait.

    I don’t know if the game will end up with an illustration for every object and room — that would be a lot of pixels and these take me a while — but it would be pretty neat to illustrate the major landmarks and most interesting items.

  • spline/flora: I wrote a quick and dirty chronological archive for Flora. I then did a bunch of witchcraft to replace 250-odd old comic pages with the touched-up versions that appear in the printed book.

  • blog: I wrote about elegance. I also started on another post, which is not yet finished. It’s also terrible, sorry.

2016-04-22 01:06:00

Programmers sometimes like to compliment code as elegant, yet I can’t recall ever seeing a satisfying explanation of what “elegant code” is. Perhaps it’s telling that I see “elegant” used much less often by more experienced programmers, who opt for more concrete commentary.

Surely elegance is a quality to strive for, but how are we to strive for something we can’t define? “I know it when we see it” isn’t good enough.

I think about this from time to time. Here’s what I’ve come up with.

Some definitions

I get a gut feeling when something is elegant, and a different gut feeling altogether when something is hacky; I suspect most programmers experience the same. The strongest pattern I’ve found is this:

Elegance is about expressing exactly what you mean — no more, no less.

Conversely, I could define a hack as something that doesn’t remotely express what you mean, but happens to have a close-enough effect.

That’s not to say all code lies on a linear spectrum between two extremes. There’s some complexity here, because “what you mean” is less concrete than the shape of your code or what happens when it executes.

Consider my recent example of links recreated in JavaScript. You might implement such a faux link with some jQuery.

1
2
3
$('#link').click(function() {
    window.location = 'http://www.google.com/';
});

Isn’t that elegant? It’s short, it’s sweet, and it does exactly as it says: when the link element is clicked, navigate to Google.

No, of course not. jQuery is elegant, perhaps, for some set of simple operations. This code is a hack, but in a way that only a human could reckon. What the author actually meant was a link — not “an element that navigates when clicked upon”, but a link, the fundamental concept that makes the Web what it is. The concrete impact of this is that a bunch of stuff humans expect from links, like the ability to middle click, is conspicuously missing.

Okay, what if you could reproduce all of that extra functionality? What if you painstakingly researched how links behave on every operating system and in every browser, and recreated that with JavaScript? No one would look at the resulting pile of special cases and call it elegant. And yet it wouldn’t really be a hack, either. The code would express “simulate everything a link does”, and while that’s not the same as having a link, it’s at least fairly close. It’d fall into a third unnamed category where a problem is solved fairly rigorously, but the outcome isn’t pretty.

The trick here is, again, all about meaning. We like to pretend that programming is a purely abstract thing, and perhaps some of the ideas are, but the languages and tools are all ultimately designed for humans. They’re designed to make sense to humans (as much as possible within given constraints, anyway), and they’re designed to solve problems humans have.

Elegance is what happens when we find a way to express what we mean with the units of meaning that our tools provide.

Sometimes that’s not possible, so the rest of this post — spoilers! — will be some concrete examples that have crossed my path recently. Maybe they’ll give you a better idea of when and why I frown at computers.

ZDoom and PickActor

Every live object in ZDoom — monsters, weapons, pickups, etc. — is called an “actor” (or sometimes a “thing”). ZDoom’s scripting language has a PickActor function for, essentially, finding what actor another actor is looking at. You might use this to find what monster is under the player’s crosshair and show a health bar over it, say.

There’s a catch. ZDoom’s scripting language can not manipulate actors directly; there is no actor type. Instead, an actor can have a “TID” (“thing ID”). Most actor-related functions thus accept a TID and affect every actor with that TID simultaneously. Appropriately, PickActor doesn’t (can’t!) return the actor it finds, but assigns that actor a TID so you can manipulate it indirectly.

By default, PickActor will refuse to overwrite an existing TID, and returns 1 to indicate it found a target or 0 to indicate it didn’t. It has two flags that can change its behavior: one to force overwriting an existing TID, and one to change the return value to the found actor’s TID.

This is everything you need to know to understand the problem, which is: how do you use PickActor to pick an actor?

The actor might already have a TID, intended for some other purpose. If you use no flags, the function will return 1 to indicate success, but the target won’t have your chosen TID, so any attempts to manipulate the target will silently do nothing. If you use the flag that forces changing a TID, you’ll almost certainly break some other effect that needed to be able to identify that actor. If you use the flag that returns an existing TID, you might end up manipulating multiple actors, because actors can share a TID.

It seems that there’s no way at all to use this function correctly!

In truth, there is, but it relies on a little outside knowledge: scripts block the entire simulation. Unless you explicitly sleep (or run for far too long and get forcibly killed), the game state cannot change out from under you.

With that in mind, the solution is:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
    // Get the existing TID, if there is one.
    // A TID of 0 is the same as not having a TID, so "setting" it to 0 will not
    // change anything's TID.
    int old_tid = PickActor(..., tid=0, flags=PICKAF_RETURNTID);

    // Find a TID that's not currently in use.
    int new_tid = UniqueTID();

    // Do the "real" call, which both forcibly sets the TID and returns true if
    // something was actually found.
    if (PickActor(..., tid=new_tid, flags=PICKAF_FORCETID)) {
        do_some_stuff_with(new_tid);
        do_some_other_stuff_with(new_tid);

        // Restore the actor's TID to its original value.
        Thing_ChangeTID(new_tid, old_tid);
    }

This relies on calling PickActor twice: once to get the target’s old TID, and once to change it. As long as both calls have the same arguments, the result must be the same, because the game state is frozen for the duration of this code. If you need to operate on the target for more than one frame, well… you have some more work to do.

The workaround is certainly not elegant. “Look for an actor in this direction, twice” is not what I wanted to express. And yet it’s not a hack, either. The code above demonstrably does the correct thing in all cases, and is suitable as a permanent solution. It occupies that nebulous third category of “complete, but not pretty”.

PickActor, on the other hand, is a shambles. I don’t know if you can really call an API a “hack”, but if you can, I would definitely like to do so right here. The function alone does the wrong thing, the “force” flag does the wrong thing, and the correct solution of calling it twice is not remotely obvious.


I care about this because modding an ancient game is largely a hobbyist affair, and there are plenty of people in the Doom community for whom tinkering with ZDoom is their first exposure to programming. They aren’t going to realize the caveats of this function, let alone know how to fix them.

I don’t want to rag on anyone in particular who’s just trying to make a fun game mod, but there is a whole lot of bad ZDoom code floating around. People try stuff until it looks like it works, and then they leave it be. I don’t blame non-professionals for that. I blame tools that don’t offer the right building blocks for modders to express what they mean.

Tool design is important! It’s why I pick on programming languages. If the fundamental pieces at your disposal are awkwardly-shaped, you’ll have a much harder time expressing what you actually intended.

Inform 7

More spoilers: these are all going to be from video games.

Video games are surprisingly difficult to express. Programming languages are languages, so they usually inherit some of the properties of human languages: code is read (executed) in sequence, and large amounts of code can be broken into a hierarchy for easier understanding. For a great many applications, that works great.

For simulations (a superset of video games), that all goes to hell. You have a great many things all acting independently and simultaneously, each with slightly different concerns and behavior. Expressing that elegantly in a linear, hierarchical language is a lost cause right from the start. Operating systems could be described in much the same way, and I’ve never heard anyone call OS code particularly elegant either. We’ve barely even figured out how to expose threads in a language, without the associated mountain of gotchas and pitfalls.

Inform 7 is interesting because it’s explicitly designed for games, and even a particular kind of game. (Contrast with the more popular style of “something like C, but made to fit my engine”.) I should do a full post on it sometime, but here’s the tl;dr.

Inform 7 is intended for writing interactive fiction games (like Zork). The syntax resembles written English, roughly mirroring how the resulting game is played. For example:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
The mysterious cavern is a room.
"Numerous stalagtites cast eerie shadows on the walls."

A golden crown is a wearable thing in the mysterious cavern.
"A golden glint pierces the darkness."
The description is "It's a golden crown."
Understand "glint" as the golden crown.

After wearing the golden crown:
    end the story finally saying "All hail the new queen!"

This is a full and complete game, which you can win by wearing the crown. There’s a lot of stuff crammed in here, and hopefully you get the general idea: objects are defined with English statements of fact, not by fenced-off blocks.

Inform 7 is particularly interesting because it attempts to address an ancient question of game design: how do you define the interaction between two mechanics?

This game’s win condition is wearing the crown — i.e., wearing the crown triggers the effect of winning the game. That code has to go somewhere, but does it go in the implementation of wearing or the implementation of the crown? Probably in the crown, since it’s a oneoff effect.

Consider, then, if I had a whole set of armor items with different defense ratings. Is that the concern of the action or the individual objects? It depends. If the vast majority of wearable objects are armor, then surely the act of wearing is concerned with handling armor. If there are only a few wearables that act as armor, they’re outliers, and should implement their own handling. Between these two extremes lies an ambiguous middle ground where it may not be clear who should really be responsible.

Every interaction in a simulated world has at least three possible “parents”: the action, who’s doing the action, and what the action is being done to. Expressing this in a language designed around hierarchy can feel clumsy when the “parent” is ambiguous, because a hierarchy necessarily requires that code only have one parent. (I’m not talking just about inheritance hierarchies, either; your actual source code can only exist in one place, nested inside one class, which is nested inside a namespace, which is nested inside a file, and so on.)

Inform 7 mostly throws the problem out the window. We can’t escape hierarchy entirely, because it’s how we as humans tend to make sense of the world. But instead of forcing the design of your game into a series of nested blocks, Inform 7 more or less lets you define anything anywhere. You can write “After” rules that apply to particular actions with particular objects, and you can put those rules anywhere in your source code: alongside the definition of wearing (well, not in this case, because that’s in the standard library), alongside the definition of the crown, or even somewhere else entirely that deals with all the ways to end the game. It’s completely up to you, and the compiler largely doesn’t care. You can even put the rule before you announce that the golden crown is an object at all, and the compiler will do a pretty good job of figuring it out.

You can still have a hierarchy, but it’s imposed entirely by you. Inform 7 lets you split your source text into a tree of volumes, books, parts, chapters, and sections, named and arranged however you like. You can see this in the source text for the example game, Bronze by Emily Short.


I tell you all this so I can compare it to the other extreme: NetHack. NetHack, written in C, is what you might call “data-oriented”. Consider for example peffects(), a 500-line function that single-handedly implements every possible effect a potion can have. polymon(), the function responsible for polymorphing the player into a monster, contains all manner of little details like halting the process of stoning if you polymorph into a stoning-resistant monster.

I once tried writing a roguelike that used entity component to avoid this kind of mess — no small feat, since it involves a whole lot of fighting back against the language design. Having beaten on that for a while, I can safely say I vastly prefer the Inform 7 model. It’s flexible enough for most purposes, and it has multiple layers of overriding, so you can make an object that prevents the golden crown from winning the game without having to touch the existing rule!

Given the constraints of how we read and write text (and code), and given that the intended usage is still fairly broad, I daresay Inform 7’s approach is fairly elegant. It allows me to clearly say what I mean in most cases, and that’s often enough.

Of course, I wouldn’t have brought it up without some more specific anecdotes.

Locksmith by Emily Short

That’s the name of an extension bundled with Inform 7. Extension names always include the name of the author, which is charming.

Locks and keys are a staple of interactive fiction, and Inform 7 has some built-in support for them. However, it requires you to explicitly name which key you’re using to unlock something. If this is part of a puzzle, it might be appropriate; in the common case of “iron key” unlocking “iron door”, it’s a little tedious.

Locksmith by Emily Short addresses this problem by splitting the “LOCK” and “UNLOCK” actions into two parts. If you say “UNLOCK DOOR WITH KEY”, that’s regular old unlocking. If you only say “UNLOCK DOOR”, that’s a new action called unlocking keylessly, and it tries to find a matching key in your inventory. “LOCK” works similarly.

Now, I want to make a padlock. You don’t need a key to close a padlock, so I want to hijack “locking keylessly” to work without needing a key at all. Both “CLOSE PADLOCK” and “LOCK PADLOCK” should do the same thing, so I use an instead rule to redirect both actions. I also want to make the padlock support the standard “open” property, but I define it in terms of whether it’s currently locked, since that’s how a padlock works.

1
2
3
4
5
6
Definition: the padlock is open rather than closed when it is not locked.
Instead of closing the padlock:
    try locking keylessly the padlock.
Instead of locking keylessly the padlock:
    now the padlock is locked;
    say "You snap the padlock shut.  It makes a chunky, satisfying click."

I compile this and type “CLOSE PADLOCK” and… the game freezes, trapped forever in an infinite loop. Or maybe the stack would overflow eventually.

What’s the problem? It’s not quite in my code; it’s an interaction with the extension, which has some convenience features. If you try to keylessly lock something that’s open (usually a door!), a before rule kicks in and makes you automatically close it first. Before rules happen before instead rules, so we have the following sequence of events.

  1. The player tries to close the (open) padlock.
  2. My code redirects this into having the player lock the padlock instead.
  3. The extension sees that the player is trying to lock something that’s open, and has the player try to close it first.
  4. GOTO 1

This is the downside of having a lot of override mechanisms. Arguably, the extension’s convenience behavior shouldn’t be in the actual implementation of “locking keylessly” rather than in before rules, but there are decent arguments to be made both ways.

The solution is remarkably simple: just swap the actions.

1
2
3
4
5
6
Definition: the padlock is open rather than closed when it is not locked.
Instead of locking keylessly the padlock:
    try closing the padlock.
Instead of closing the padlock:
    now the padlock is locked;
    say "You snap the padlock shut.  It makes a chunky, satisfying click."

And now I get exactly what I want:

> close padlock

You snap the padlock shut. It makes a chunky, satisfying click.

Simple, but perhaps not obvious. Something here is inelegant, yet everyone involved was pretty clearly expressing what they meant. I wonder what the lesson is.

Doors

Ah, good, more Inform 7 stuff, and more door stuff as well!

Doors are pretty straightforward in Inform 7: they have two sides, and each side goes in a room. The player can go through the door to get from one side to the other. Easy.

I want to have a door that can lead to several different places. In other words, the front side is fixed in place, but the back side can move around.

Inform 7 won’t let you do this, full stop. That means it’s time for an adventure.


The obvious solution is to fake it.

  1. Make a bunch of identical fake-door objects (which aren’t actually doors) and put one in each place the door can appear.
  2. Hide them from everywhere the door isn’t supposed to be.
  3. Override all the actions that could conceivably take the player through the door to just teleport them instead.

This is pretty clearly a hack. What I mean is a door, and this is as far away from a door as is possible to get. As is common with hacks, there are also thorny downsides to every step.

  1. Identical objects can be slightly awkward. In particular, if you try to use a custom command that works on objects you have seen but can’t currently see (like “remember”, perhaps), you may get a rather awful disambiguation prompt like: “Which do you mean? The green door, the green door, the green door, or the green door?”

    Also, since they aren’t doors, no door-specific behavior will work on them. Pathfinding won’t work, because the spaces aren’t actually connected. I want to implement looking through a door, and that won’t work. Built-in mechanisms like “Instead of going through the green door” won’t work.

    You might have slightly more luck if you used real doors that all lead nowhere, and very carefully hooked only the handling for where a door leads. But that has problems with part 2.

  2. Hiding objects is more complicated than you’d think! The most common suggestion I’ve seen is to do this:

    1
    2
    Instead of doing something with the hidden item:
        say "You can't see any such thing."
    

    That is, of course, the default response when you try to refer to an object that doesn’t exist. And this is a terrible hack! If you change that default response, this hack will get out of sync. If you use one of the available extensions to improve the error handling, this will stick out like a sore thumb. And just like with the padlock, a before rule will still trigger first.

    An easier option is to actually move the dummy objects around. Of course, if you decide to use real single-sided doors, you’re back to the original problem: you can’t move a door.

  3. Inform 7’s movement is remarkably complex. You can push objects from room to room and ride in vehicles. Merely teleporting the player would break these; you would need to reimplement them in some fashion.

None of these problems are insurmountable, but certainly this approach is difficult to implement correctly.


The solution I went for was to make it work by force.

You see, Inform 7 actually compiles to a very different language called Inform 6, which is more traditional and C-like (though with some pretty funky syntax in places). Parts of the standard library are implemented in Inform 6, and the detailed comments in those files reveal some interesting details.

One, Inform 6 models the entire world as… a hierarchy. Everything is one big tree. The room contains the player, and the player contains their inventory — the only thing distinguishing an inventory is that the parent is a person. Even worn objects are children like any other, except that they have an invisible “worn” flag.

This model presents a problem with doors, which exist in two places at once. The secret is… they actually don’t. The physical door object moves to follow the player! There’s some special consideration for doors in places to make them appear to be in both places from other characters’ points of view, so you’d never notice the difference.

Two, Inform 6 actually had an entire feature for objects that faked being in more than one room, called “floating”. These were apparently such a huge and buggy pain in the ass that Inform 7 drastically limited floating objects to only two specific categories: doors, and a certain kind of non-interactive decorations.

Three, every Inform 7 door ends up with a found_in property in Inform 6, which is an array of the two places the door should exist.

Well, hey, that sounds easy. You can embed chunks of Inform 6 code in Inform 7, so I wrote some code that would edit that array directly.

Foiled! It didn’t work. The problem is that going through a door takes you to the “other side”. “Other side” is a real physical property — an Inform 6 function that picks a side based on where the player currently is. Alas, rather than consulting the array, this function has the two sides hardcoded. It looks like this:

1
2
3
4
5
6
with door_to [ 
    loc ! room of actor
    ;
    loc = location;
    if (loc == thedark) loc = real_location;
    if (loc == I483_red_room) return I384_green_room; return I483_red_room;],

door_to is just the Inform 6 name of the “other side” property. loc is an argument to the function bounded by square brackets. Like I said, funky syntax.

I did try changing “the other side of the green door” from Inform 7 land, but it seems the compiler special-cases this one particular property name and intercepts any attempt to change it dynamically. Foiled again!

The solution I settled on was to rewrite the “other side” function for this particular door entirely. I ended up with:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
Include (-
[ HackedOtherSide loc;
    loc = location;
    if (loc == thedark) loc = real_location;
    if (loc == (+ the green door +).&found_in-->0) return (+ the green door +).&found_in-->1; return (+ the green door +).&found_in-->0;
];
[ HackedDirection loc;
    loc = location;
    if (loc == thedark) loc = real_location;
    if (loc == (+ the green door +).&found_in-->0) return (+ southwest +); return (+ northeast +);
];
-).

To hackily override the door props:
    (-
    (+ the green door +).door_dir = HackedDirection;
    (+ the green door +).door_to = HackedOtherSide;
    -).

When play begins:
    hackily override the door_to.

To hackily force the back side of (portal - a door) to (where - an object):
    (- ({portal}.&found_in)-->1 = {where}; -).

(- ... -) denotes embedded Inform 6 code; (+ ... +) denotes embedded Inform 7 object names; foo-->0 is an array subscript. door_to is another property that tells you what direction the door is in, based on what room you’re in.

Right. So.

This is definitely… uglier, at a glance, than faking the door. On the other hand, it’s pretty much transparent to the rest of the language and standard library. The door is a real door that genuinely moves from place to place. Anything that operates on a door goes through the code I’ve overridden here, so it’s fairly bulletproof.

The one issue not addressed here is route-finding — the standard library can cache information about connections between rooms to make route-finding faster, and after moving the door, that cache is no longer valid. A quick read of the comments reveals a trivial fix: call SignalMapChange() after moving the door around.

So. Is this elegant?

It sure doesn’t look pretty, and it requires meddling with guts that I’m not even supposed to know about. That’s not particularly elegant.

And yet! I only did this in the first place because it lets all the rest of my code express exactly what I want. I don’t have to keep remembering that my doors are fake and sticking special cases everywhere. This code is self-contained, robust, and forgettable. It sacrifices elegance so that other code doesn’t have to.

I think that’s something to value.

Starbound versus Aseprite

My door hack is not so much about code elegance, but interface elegance. It does some strange things in order to preserve all the guarantees and semantics of a fundamental existing interface: the door.

I’d like to close with a more literal example of that.

Hardware-accelerated programs generally don’t use native GUI controls. This generally means: video games implement their own buttons, textboxes, etc. I’ve heard rumors that it’s possible to mash native GUI controls into an accelerated canvas, but I can’t recall ever having seen it done.

I got into Starbound for a while last year. I even ran a little private server for some friends. And one constant aggravation I had was that Tab doesn’t move the focus between textboxes in dialogs. The “connect to server” dialog is just three boxes: hostname, username, password. It seems so obvious that you should be able to tab between these, but you cannot. Whoever built the GUI didn’t implement tabbing.

Some other common keyboard shortcuts are wrong, which trips me up from time to time. Ctrlarrow moves the cursor only one character at a time, rather than one word at a time. Shiftarrow moves one word at a time, rather than selecting a character. CtrlBackspace appears to clear an entire textbox, rather than deleting the previous word.

It’s not surprising that a few things would slip through the cracks. Game GUI libraries have to be reimplemented from scratch, and that’s an awful lot of work just to get the buttons and textboxes that other developers get to take for granted.

A few days ago, I tried Aseprite, a pixel art editor. The entire interface is custom-drawn (in a pixellated style, no less), which had me worried that I’d encounter the same problems as in Starbound.

To my great surprise, I didn’t! All the keys I’m used to work just fine. I can even double-click a word to select it. I don’t know if this is a third-party library or something the Aseprite developer built themselves, but someone put a lot of work into making this feel natural.

I don’t usually like custom UIs at all. Between the very appropriate design and the attention to detail, I’m pretty sold on Aseprite’s.

I might even call it… elegant.

It doesn’t matter that I haven’t seen the code. Maybe the code is a mess. In fact it almost certainly is. The end result is what’s elegant here, because it completely disappears. I can forget I’m not using native controls.

Maybe that’s a better definition of elegance, then: elegance is transparent. It’s so natural that it disappears entirely. A hack stands out because it’s unusual and jarring; an elegant solution (or interface, or whatever) fits together so smoothly that you forget the problem ever existed.

April’s theme is finish Runed Awakening.

It took a few days to clear my plate and get up to speed, and I’m a little worried at how much time blog obligations seem to be eating this month, but I made some good progress.

  • twitter: I installed Windows 95 in a VM and documented the experience. I also wrote a good Twitter bot that tweets every word, distorted by some Unicode shenanigans.

  • blog: I wrote about emoji, at considerable length.

  • Runed Awakening: Ehh. I ran dry on major ideas and spent several days mostly struggling to drum up some more (or avoiding the question altogether, which is how the Twitter bot came to be). I did end up making a little progress, but not what I would call a week’s worth.

I’m halfway through the month and don’t have a whole lot to show for it, unfortunately. Learning how to have ideas is… difficult. With Doom architecture, I at least found a trick for getting my foot in the door: find an empty space and just draw something in it. I don’t know how to translate that to a text-based puzzle game where everything has to make a bit more narrative sense, which is a little frustrating.

I’ve had some more inspiration today, so that’s good, but I’m going to run dry again pretty quickly if I can’t find a reproducible way to generate new ideas. I have a hard time just sitting and thinking in the first place, which is a huge handicap. I’ve had the best luck when doing something tedious and purely physical, so my mind can wander, but I’ve never been able to duplicate the effect deliberately.

April’s theme is finish Runed Awakening.

It took a few days to clear my plate and get up to speed, and I’m a little worried at how much time blog obligations seem to be eating this month, but I made some good progress.

  • blog: I scrapped a bunch of manual thumbnails from git and finally switched to Pelican’s thumbnailer plugin. I also wrote about my first computer(s).

  • art: I painted an Eevee, and it came out okay. I also did a quick painting inspired by Runed Awakening work.

  • Runed Awakening: Aha! Much like making a Doom level, the hardest part is just having ideas, and then having more ideas to fill in the details of the bigger ideas. I’ve got enough big ideas; it’s the little ones that are proving tricky. I did manage to fix a ton of bugs, finally add several alternative solutions for getting past the ███████, implement the ████████ ██ ████, add a new ████████ and an item that can ████████, and jot down some notes on future direction. It’s not quite a week’s worth of work, but it’s a solid few days’ worth, and I feel like it’s a big step forward. I hope I can keep it going!

I love emoji. I love Unicode in general. I love seeing plain text become more expressive and more universal.

But, Internet, I’ve noticed a worrying trend. Both popular media and a lot of tech circles tend to assume that “emoji” de facto means Apple’s particular font.

I have some objections.

A not so brief history of emoji

The Unicode Technical Report on emoji also goes over some of this.

Emoji are generally traced back to the Japanese mobile carrier NTT DoCoMo, which in February 1999 released a service called i-mode which powered a line of wildly popular early smartphones. Its messenger included some 180 small pixel-art images you could type as though they were text, because they were text, encoded using unused space in Shift JIS.

(Quick background, because I’d like this to be understandable by a general audience: computers only understand numbers, not text, so we need a “character set” that lists all the characters you can type and what numbers represent them. So in ASCII, for example, a capital “A” is passed around as the number 65. Computers always deal with bytes, which can go up to 255, but ASCII only lists characters up to 127 — so everything from 128 to 255 is just unused space. Shift JIS is Japan’s equivalent to ASCII, and had a lot more unused space, and that’s where early emoji were put.)

Naturally, other carriers added their own variations. Naturally, they used different sets of images, but often in a different order, so the same character might be an apple on one phone and a banana on another. They came up with tables for translating between carriers, but that wouldn’t help if your friend tried to send you an image that your phone just didn’t have. And when these characters started to leak outside of Japan, they had no hope whatsoever of displaying as anything other than garbage.

This is kind of like how Shift JIS is mostly compatible with ASCII, except that for some reason it has the yen sign ¥ in place of the ASCII backslash \, producing hilarious results. Also, this is precisely the problem that Unicode was invented to solve.

I’ll get back to all this in a minute, but something that’s left out of emoji discussions is that the English-speaking world was developing a similar idea. As far as I can tell, we got our first major exposure to graphical emoticons with the release of AIM 4.0 circa May 2000 and these infamous “smileys”:

Pixellated, 4-bit smileys from 2000

Even though AIM was a closed network where there was little risk of having private characters escape, these were all encoded as ASCII emoticons. That simple smiley on the very left would be sent as :-) and turned into an image on your friend’s computer, which meant that if you literally typed :-) in a message, it would still render graphically. Rather than being an extension to regular text, these images were an enhancement of regular text, showing a graphical version of something the text already spelled out. A very fancy ligature.

Little ink has been spilled over this, but those humble 4-bit graphics became a staple of instant messaging, by which I mean everyone immediately ripped them off. ICQ, MSN Messenger, Yahoo! Messenger, Pidgin (then Gaim), Miranda, Trillian… I can’t name a messenger since 2003 that didn’t have smileys included. All of them still relied on the same approach of substituting graphics for regular ASCII sequences. That had made sense for AIM’s limited palette of faces, but during its heyday MSN Messenger included 67 graphics, most of them not faces. If you sent a smiling crescent moon to someone who had the graphics disabled (or used an alternative client), all they’d see was a mysterious (S).

So while Japan is generally credited as the source of emoji, the US was quite busy making its own mess of things.

Anyway, Japan had this mess of several different sets of emoji in common use, being encoded in several different incompatible ways. That’s exactly the sort of mess Unicode exists to sort out, so in mid-2007, several Google employees (one of whom was the co-founder of the Unicode Consortium, which surely helped) put together a draft proposal for adding the emoji to Unicode. The idea was to combine all the sets, drop any duplicates, and add to Unicode whatever wasn’t already there.

(Unicode is intended as a unification of all character sets. ASCII has \, Shift JIS has ¥, but Unicode has both — so an English speaker and a Japanese speaker can both use both characters without getting confused, as long as they’re using Unicode. And so on, for thousands of characters in dozens of character sets. Part of the problem with sending the carriers’ emoji to American computers was that the US was pretty far along in shifting everything to use Unicode, but the emoji simply didn’t exist in Unicode. Obvious solution: add them!)

Meanwhile, the iPhone launched in Japan in 2008. iOS 2.2, released in November, added the first implementation of emoji — but using SoftBank’s invented encoding, since they were only on one carrier and the characters weren’t yet in Unicode. A couple Apple employees jumped on the bandwagon around that time and coauthored the first official proposal, published in January 2009. Unicode 6.0, the first version to include emoji, was released in October 2010.

iPhones worldwide gained the ability to use its emoji (now mapped to Unicode) with the release of iOS 5.0 in October 2011.

Android didn’t get an emoji font at all until version 4.3, in July 2013. I’m at a loss for why, given that Google had proposed emoji in the first place, and Android had been in Japan since the HTC Magic in May 2009. It was even on NTT DoCoMo, the carrier that first introduced emoji! What the heck, Google.

The state of things

Consider this travesty of an article from last week. This Genius Theory Will Change the Way You Use the “Pink Lady” Emoji:

Unicode, creators of the emoji app, call her the “Information Desk Person.”

Oh, dear. Emoji aren’t an “app”, Unicode didn’t create them, and the person isn’t necessarily female. But the character is named “Information Desk Person”, so at least that part is correct.

It’s non-technical clickbait, sure. But notice that neither “Apple” nor the names of any of its platforms appear in the text. As far as this article and author are concerned, emoji are Apple’s presentation of them.

I see also that fileformat.info is now previewing emoji using Apple’s font. Again, there’s no mention of Apple that I can find here; even the page that credits the data and name sources doesn’t mention Apple. The font is even called “Apple Color Emoji”, so you’d think that might show up somewhere.

Telegram and WhatsApp both use Apple’s font for emoji on every platform; you cannot use your system font. Slack lets you choose, but defaults to Apple’s font. (I objected to Android Telegram’s jarring use of a non-native font; the sole developer explained simply that they like Apple’s font more, and eventually shut down the issue tracker to stop people from discussing it further.)

The latest revision of Google’s emoji font even made some questionable changes, seemingly just for the sake of more closely resembling Apple’s font. I’ll get into that a bit later, but suffice to say, even Google is quietly treating Apple’s images as a de facto standard.

The Unicode Consortium will now let you “adopt” a character. If you adopt an emoji, the certificate they print out for you uses Apple’s font.

It’s a little unusual that this would happen when Android has been more popular than the iPhone almost everywhere, even since iOS first exposed its emoji keyboard worldwide. Also given that Apple’s font is not freely-licensed (so you’re not actually allowed to use it in your project), whereas Google’s whole font family is. And — full disclosure here — quite a few of them look to me like they came from a disquieting uncanny valley populated by plastic people.

Terrifying.

Granted, the iPhone did have a 20-month head start at exposing the English-speaking world to emoji. Plus there’s that whole thing where Apple features are mysteriously assumed to be the first of their kind. I’m not entirely surprised that Apple’s font is treated as canonical; I just have some objections.

Some objections

I’m writing this in a terminal that uses Source Code Pro. You’re (probably) reading it on the web in Merriweather. Miraculously, you still understand what all the letters mean, even though they appear fairly differently.

Emoji are text, just like the text you’re reading now, not too different from those goofy :-) smileys in AIM. They’re often displayed with colorful graphics, but they’re just ideograms, similar to Egyptian hieroglyphs (which are also in Unicode). It’s totally okay to write them a little differently sometimes.

This the only reason emoji are in Unicode at all — the only reason we have a universal set of little pictures. If they’d been true embedded images, there never would have been any reason to turn them into characters.

Having them as text means we can use them anywhere we can use text — there’s no need to hunt down a graphic and figure out how to embed it. You want to put emoji in filenames, in source code, in the titlebar of a window? Sure thing — they’re just text.

Treating emoji as though they are a particular set of graphics rather defeats the point. At best, it confuses people’s understanding of what the heck is going on here, and I don’t much like that.

I’ve encountered people who genuinely believed that Apple’s emoji were some kind of official standard, and anyone deviating from them was somehow wrong. I wouldn’t be surprised if a lot of lay people believed Apple invented emoji. I can hardly blame them, when we have things like World Emoji Day, based on the date on Apple’s calendar glyph. This is not a good state of affairs.


Along the same lines, nothing defines an emoji, as I’ve mentioned before. Whether a particular character appears as a colored graphic is purely a property of the fonts you have installed. You could have a font that rendered all English text in sparkly purple letters, if you really wanted to. Or you could have a font that rendered emoji as simple black-and-white outlines like other characters — which is in fact what I have.

Well… that was true, but mere weeks before that post was published, the Unicode Consortium published a list of characters with a genuine “Emoji” property.

But, hang on. That list isn’t part of the actual Unicode database; it’s part of a “technical report”, which is informative only. In fact, if you look over the Unicode Technical Report on emoji, you may notice that the bulk of it is merely summarizing what’s being done in the wild. It’s not saying what you must do, only what’s already been done. The very first sentence even says that it’s about interoperability.

If that doesn’t convince you, consider that the list of “emoji” characters includes # and *. Yes, the ASCII characters on a regular qwerty keyboard. I don’t think this is a particularly good authoritative reference.

Speaking of which, the same list also contains ©, ®, and — and Twitter’s font has glyphs for all three of them: ©, ®, . They aren’t used on web Twitter, but if you naïvely dropped twemoji into your own project, you’d see these little superscript characters suddenly grow to fit large full-width squares. (Worse, all three of them are a single solid color, so they’ll be unreadable on a dark background.) There’s an excellent reason for this, believe it or not: Shift JIS doesn’t contain any of these characters, so the Japanese carriers faked it by including them as emoji.

Anyway, the technical report proper is a little more nuanced, breaking emoji into a few coarse groups based on who implements them. (Observe that it uses Apple’s font for all 1282 example emoji.)

I care about all this because I see an awful lot of tech people link this document as though it were a formal specification, which leads to a curious cycle.

  1. Apple does a thing with emoji.
  2. Because Apple is a major vendor, the thing it did is added to the technical report.
  3. Other people look at the report, believe it to be normative, and also do Apple’s thing because it’s “part of Unicode”.
  4. (Wow, Apple did this first again! They’re so ahead of the curve!)

After I wrote the above list, I accidentally bumbled upon this page from emojipedia, which states:

In addition to emojis approved in Unicode 8.0 (mid-2015), iOS 9.1 also includes emoji versions of characters all the way back to Unicode 1.1 (1993) that have retroactively been deemed worthy of emoji presentation by the Unicode Consortium.

That’s flat-out wrong. The Unicode Consortium has never deemed characters worthy of “emoji presentation” — it’s written reports about the characters that vendors like Apple have given colored glyphs. This paragraph congratulates Apple for having an emoji font that covers every single character Apple decided to put in their emoji font!

This is a great segue into what happened with Google’s recent update to its own emoji font.

Google's emoji font changes

Android 6.0.1 was released in December 2015, and contained a long-overdue update to its emoji font, Noto Color Emoji. It added newly-defined emoji like 🌭 U+1F32D HOT DOG and 🦄 U+1F984 UNICORN FACE, so, that was pretty good.

ZWJ sequences

How is this a segue, you ask? Well, see, there are these curious chimeras called ZWJ sequences — effectively new emoji created by mashing multiple emoji together with a special “glue” character in the middle. Apple used (possibly invented?) this mechanism to create “diverse” versions of several emoji like 💏 U+1F48F KISS. The emoji for two women kissing looks like a single image, but it’s actually written as seven characters: woman + heart + kiss + woman with some glue between them. It’s a lot like those AIM smileys, only not ASCII under the hood.

So, that’s fine, it makes sense, I guess. But then Apple added a new chimera emoji: a speech bubble with an eyeball in it, written as eye + speech bubble. It turned out to be some kind of symbol related to an anti-bullying campaign, dreamed up in conjunction with the Ad Council (?!). I’ve never seen it used and never heard about this campaign outside of being a huge Unicode nerd.

Lo and behold, it appeared in the updated font. And Twitter’s font. And Emoji One.

Is this how we want it to work? Apple is free to invent whatever it wants by mashing emoji together, and everyone else treats it as canonical, with no resistance whatsoever? Apple gets to deliberately circumvent the Unicode character process?

Apple appreciated the symbol, too. “When we first asked about bringing this emoji to the official Apple keyboard, they told us it would take at least a year or two to get it through and approved under Unicode,” says Wittmark. The company found a way to fast-track it, she says, by combining two existing emoji.

Maybe this is truly a worthy cause. I don’t know. All I know is that Apple added a character (designed by an ad agency) basically on a whim, and now it’s enshrined forever in Unicode documents. There doesn’t seem to be any real incentive for them to not do this again. I can’t wait for apple + laptop to become the MacBook Pro™ emoji.

(On the other hand, I can absolutely get behind ninja cat.)

Gender diversity

I take issue with using this mechanism for some of the “diverse” emoji as well. I didn’t even realize the problem until Google copied Apple’s implementation.

The basic emoji in question are 💏 U+1F48F KISS and 💑 U+1F491 COUPLE WITH HEART. The emoji technical report contains the following advice, emphasis mine:

Some multi-person groupings explicitly indicate gender: MAN AND WOMAN HOLDING HANDS, TWO MEN HOLDING HANDS, TWO WOMEN HOLDING HANDS. Others do not: KISS, COUPLE WITH HEART, FAMILY (the latter is also non-specific as to the number of adult and child members). While the default representation for the characters in the latter group should be gender-neutral, implementations may desire to provide (and users may desire to have available) multiple representations of each of these with a variety of more-specific gender combinations.

This reinforces the document’s general advice about gender which comes down to: if the name doesn’t explicitly reference gender, the image should be gender-neutral. Makes sense.

Here’s how 💏 U+1F48F KISS and 💑 U+1F491 COUPLE WITH HEART look, before and after the font update.

Pictured: straight people, ruining everything

Before, both images were gender-agnostic blobs. Now, with the increased “diversity”, you can choose from various combinations of genders… but the genderless version is gone. The default — what you get from the single characters on their own, without any chimera gluing stuff — is heteromance.

In fact, almost every major font does this for both KISS and COUPLE WITH HEART, save for Microsoft’s. (HTC’s KISS doesn’t, but only because it doesn’t show people at all.)

Google’s font has changed from “here are two people” to “heterosexuals are the default, but you can use some other particular combinations too”. This isn’t a step towards diversity; this is a step backwards. It also violates the advice in the very document that’s largely based on “whatever Apple and Google are doing”, which is confounding.

Sometimes, Apple is wrong

It also highlights another problem with treating Apple’s font as canonical, which is that Apple is occasionally wrong. I concede that “wrong” is a fuzzy concept here, but I think “surprising, given the name of the character” is a reasonable definition.

In that sense, everyone but Microsoft is wrong about 💏 U+1F48F KISS and 💑 U+1F491 COUPLE WITH HEART, since neither character mentions gender.

You might expect 🙌 U+1F64C PERSON RAISING BOTH HANDS IN CELEBRATION and 🙏 U+1F64F PERSON WITH FOLDED HANDS to depict people, but Apple only shows a pair of hands for both of them. This is particularly bad with PERSON WITH FOLDED HANDS, which just looks like a high five. Almost every other font has followed suit (CELEBRATION, FOLDED HANDS). Google used to get this right, but changed it with the update.

Celebration changed to pat-a-cake, for some reason

👿 U+1F47F IMP suggests, er, an imp, especially since it’s right next to other “monster” characters like 👾 U+1F47E ALIEN MONSTER and 👹 U+1F479 JAPANESE OGRE. Apple appears to have copied its own 😈 U+1F608 SMILING FACE WITH HORNS from the emoticons block and changed the smile to a frown, producing something I would never guess is meant to be an imp. Google followed suit, just like most other fonts, resulting in the tragic loss of one of my favorite Noto glyphs and the only generic representation of a demon.

This is going to wreak havoc on all my tweets about Doom

👯 U+1F46F WOMAN WITH BUNNY EARS suggests a woman. Apple has two, for some reason, though that hasn’t been copied quite as much.

⬜ U+2B1C WHITE LARGE SQUARE needs a little explanation. Before Unicode contained any emoji (several of which are named with explicit colors), quite a few character names used “black” to mean “filled” and “white” to mean “empty”, referring to how the character would look when printed in black ink on white paper. “White large square” really means the outline of a square, in contrast to ⬛ U+2B1B BLACK LARGE SQUARE, which is solid. Unfortunately, both of these characters somehow ended up in virtually every emoji font, despite not being in the original lists of Japanese carriers’ emoji… and everyone gets it wrong, save for Microsoft. Every single font shows a solid square colored white. Except Google, who colors it blue. And Facebook, who has some kind of window frame, which it colors black for the BLACK glyph.

When Apple screws up and doesn’t fix it, everyone else copies their screw-up for the sake of compatibility — and as far as I can tell, the only time Apple has ever changed emoji is for the addition of skin tones and when updating images of their own products. We’re letting Apple set a de facto standard for the appearance of text, even when they’re incorrect, because… well, I’m not even sure why.

Hand gestures

Returning briefly to the idea of diversity, Google also updated the glyphs for its dozen or so “hand gesture” emoji:

Hmm I wonder where they got the inspiration for these

They used to be pink outlines with a flat white fill, but now are a more realistic flat style with the same yellow as the blob faces and shading. This is almost certainly for the sake of supporting the skin tone modifiers later, though Noto doesn’t actually support them yet.

The problem is, the new ones are much harder to tell apart at a glance! The shadows are very subtle, especially at small sizes, so they might as well all be yellow splats.

I always saw the old glyphs as abstract symbols, rather than a crop of a person, even a cartoony person. That might be because I’m white as hell, though. I don’t know. If people of color generally saw them the same way, it seems a shame to have made them all less distinct.

It’s not like the pink and white style would’ve prevented Noto from supporting skin tones in the future, either. Nothing says an emoji with a skin tone has to look exactly like the same emoji without one. The font could easily use the more abstract symbols by default, and switch to this more realistic style when combined with a skin tone.

💩

And finally, some kind of tragic accident has made 💩 U+1F4A9 PILE OF POO turn super goofy and grow a face.

What even IS that now?

Why? Well, you see, Apple’s has a face. And so does almost everyone else’s, now.

I looked at the original draft proposal for this one, and SoftBank (the network the iPhone first launched on in Japan) also had a face for this character, whereas KDDI did not. So the true origin is probably just that one particular carrier happened to strike a deal to carry the iPhone first.

Interop and confusion

I’m sure the rationale for many of these changes was to reduce confusion when Android and iOS devices communicate. I’m sure plenty of people celebrated the changes on those grounds.

I was subscribed to several Android Telegram issues about emoji before the issue tracker was shut down, so I got a glimpse into how people feel about this. One person was particularly adamant that in general, the recipient should always see exactly the same image that the sender chose. Which sounds… like it’s asking for embedded images. Which Telegram supports. So maybe use those instead?

I grew up on the Internet, in a time when ^_^ looked terrible in mIRC’s default font of Fixedsys but just fine in PIRCH98. Some people used MS Comic Chat, which would try to encode actions in a way that looked like annoying noise to everyone else. Abbreviations were still a novelty, so you might not know what “ttfn” means.

Somehow, we all survived. We caught on, we asked for clarification, we learned the rules, and life went on. All human communication is ambiguous, so it baffles me when people bring up “there’s more than one emoji font” as though it spelled the end of civilization. Someone might read what you wrote and interpret it differently than you intended? Damn, that is definitely a new and serious problem that we have no idea how to handle.

It sounds to me how this would’ve sounded in 1998:

A: ^_^
B: Wow, that looks totally goofy over here. I’m using mIRC.
A: Oh, I see the problem. Every IRC client should use Arial, like PIRCH does.

That is, after all, the usual subtext: every font should just copy whatever Apple does. Let’s not.

Look, science!

Conveniently for me, someone just did a study on this. Here’s what I found most interesting:

Overall, we found that if you send an emoji across platform boundaries (e.g., an iPhone to a Nexus), the sender and the receiver will differ by about 2.04 points on average on our -5 to 5 sentiment scale. However, even within platforms, the average difference is 1.88 points.

In other words, people still interpret the same exact glyph differently — just like people sometimes interpret the same words differently.

The gap between same-glyph and different-glyph is a mere 0.16 points out of a 10-point scale, a mere 1.6%. The paper still concludes that the designs should move closer together, and sure, they totally should — towards what the characters describe.

To underscore that idea, note the summary page discusses U+1F601 😁 GRINNING FACE WITH SMILING EYES across five different fonts. Surely this should express something positive, right? Grins are positive, smiling eyes are positive; this might be the most positive face in Unicode. Indeed, every font was measured as expressing a very positive emotion, except Apple’s, which was apparently controversial but averaged out to slightly negative. Looking at the various renderings, I can totally see how Apple’s might be construed as a grimace.

So in the name of interoperability, what should font vendors do here? Push Apple (and Twitter and Facebook, by the look of it) to change their glyph? Or should everyone else change, so we end up in a world where two-thirds of people think “grinning face with smiling eyes” is expressing negativity?

A diversion: fonts

Perhaps the real problem here is font support itself.

You can’t install fonts or change default fonts on either iOS or Android (sans root). That Telegram developer who loves Apple’s emoji should absolutely be able to switch their Android devices to use Apple’s font… but that’s impossible.

It’s doubly impossible because of a teensy technical snag. You see,

  • Apple added support for embedding PNG images in an OpenType font to OS X and iOS.

  • Google added support for embedding PNG images in an OpenType font to FreeType, the font rendering library used on Linux and Android. But they did it differently from Apple.

  • Microsoft added support for color layers in OpenType, so all of its emoji are basically several different monochrome vector images colored and stacked together. It’s actually an interesting approach — it makes the font smaller, it allows pieces to be reused between characters, and it allows the same emoji to be rendered in different palettes on different background colors almost for free.

  • Mozilla went way out into the weeds and added support for embedding SVG in OpenType. If you’re using Firefox, please enjoy these animated emoji. Those are just the letter “o” in plain text — try highlighting or copy/pasting it. The animation is part of the font. (I don’t know whether this mechanism can adapt to the current font color, but these particular soccer balls do not.)

We have four separate ways to create an emoji font, all of them incompatible, none of them standard (yet? I think?). You can’t even make one set of images and save it as four separate fonts, because they’re all designed very differently: Apple and Google only support regular PNG images, Microsoft only supports stacked layers of solid colors, and Mozilla is ridiculously flexible but still prefers vectors. Apple and Google control the mobile market, so they’re likely to win in the end, which seems a shame since their approaches are the least flexible in terms of size and color and other text properties.

I don’t think most people have noticed this, partly because even desktop operating systems don’t have an obvious way to change the emoji font (so who would think to try?), and partly because emoji mostly crop up on desktops via web sites which can quietly substitute images (like Twitter and Slack do). It’s not a situation I’d like to see become permanent, though.

Consider, if you will, that making an emoji font is really hard — there are over 1200 high-resolution images to create, if you want to match Apple’s font. If you used any web forums or IM clients ten years ago, you’re probably also aware that most smiley packs are pretty bad. If you’re stuck on a platform where the default emoji font just horrifies you (for example), surely you’d like to be able to change the font system-wide.

Disconnecting the fonts from the platforms would actually make it easier to create a new emoji font, because the ability to install more than one side-by-side means that no one font would need to cover everything. You could make a font that provides all the facial expressions, and let someone else worry about the animals. Or you could make a font that provides ZWJ sequences for every combination of an animal face and a facial expression. (Yes, please.) Or you could make a font that turns names of Pokémon into ligatures, so e-e-v-e-e displays as (eevee icon), similar to how Sans Bullshit Sans works.

But no one can do any of this, so long as there’s no single extension that works everywhere.

(Also, for some reason, I’ve yet to get Google’s font to work anywhere in Linux. I’m sure there are some fascinating technical reasons, but the upshot is that Google’s browser doesn’t support Google’s emoji font using Google’s FreeType patch that implements Google’s own font extension. It’s been like this for years, and there’s been barely any movement on it, leaving Linux as the only remotely-major platform that can’t seem to natively render color emoji glyphs — even though Android can.)

Appendix

Some miscellaneous thoughts:

  • I’m really glad that emoji have forced more developers to actually handle Unicode correctly. Having to deal with commonly-used characters outside of ASCII is a pretty big kick in the pants already, but most emoji are also in Plane 1, which means they don’t fit in a single JavaScript “character” — an issue that would otherwise be really easy to overlook.

  • On the other hand, it’s a shame that the rise of emoji keyboards hasn’t necessarily made the rest of Unicode accessible. There are still plenty of common symbols, like ♫, that I can only type on my phone using the Japanese keyboard. I do finally have an input method on my desktop that lets me enter characters by name, which is nice. We’ve certainly improved since the olden days, when you just had to memorize that Alt0233 produced an é… or, wait, maybe English Windows users still have to do that.

  • Breadth of font support is still a problem outside of emoji, and in a plaintext environment there’s just no way to provide any fallback. Google’s Noto font family aspires to have full coverage — it’s named for “no tofu”, referring to the small boxes that often appear for undisplayable characters — but there are still quite a few gaps. Also, on Android, a character that you don’t have a font for just doesn’t appear at all, with no indication you’re missing anything. That’s one way to get no tofu, I guess.

  • Brands™ running ad campaigns revolving around emoji are probably the worst thing. Hey, if we had a standard way to make colored fonts, then Guinness could’ve just released a font with a darker 🍺 U+1F37A BEER MUG and 🍻 U+1F37B CLINKING BEER MUGS, rather than running a ridiculous ad campaign asking Unicode to add a stout emoji.

  • If you’re on a platform that doesn’t ship with an emoji font, you should really really get Symbola. It covers a vast swath of Unicode with regular old black-and-white vector glyphs, usually using the example glyphs from Unicode’s own documents.

  • The plural is “emoji”, dangit. ∎

Sources:

Forum activity

The forums are dead quiet. No one is posting. A lone tumbleweed rolls by.

Maybe you should do something about this.