Originally written by Jan Nedoma.
In our time, the world is getting smaller. Thanks to multinational corporations you’ll see the same goods in the shops in many countries all around the world, you’ll see the same TV show on TV in the US and in Iceland. Of course, this phenomena also applies to computer games. Game distributors would tell you games in local language usually sell much better, and the same could be said about indie games. But before the game can be “exported” to foreign markets, it needs to go through the process called “localization”. This article will discuss how to make your game localization-ready and what problems you’re likely to encounter.
Introduction
By localization we typically mean translating the game to a different
language, but generally speaking, it’s about adapting the game to
cultural differences of a given country. You will find surprisingly few
materials concerning this part of game development in the internet. This
article is based on my personal experiences with game localizations and
the procedures
described here have been successfully used in real life. Nevertheless,
neither me or anyone else knows everything, and I’m not claiming
everything described here is the only and perfect way. Consider this
article a set of suggestions, based on real-life experience. If you have
any corrections and/or suggestions, you can contact me at [nedoma (at)
dead-code (dot) org].
This article is written from the programmer’s point of view (on
Windows platform), but most of the concepts are generally usable for
localization.
To make your game easily localizable means making it
localization-ready. Let’s get through several most important areas
affecting the game localization readiness.
Localizing the in-game texts
The string table – creating and maintaining
The in-game texts are the most obvious localization target. There are
several approaches, but ideally all the in-game texts should be
summarized in a big text file (let’s call is a “string table”), which
can be easily handed to translators to do the actual localization work.
The game should be able to load all the texts from this file and use
them. Which means, by simply replacing the string table file you’re able
to switch languages on-the-fly at runtime (or at install time,
whichever way you prefer). But how to achieve this ideal situation?
Firstly, every single text needs to be augmented by a unique identifier.
This will allow us to internally only reference the string IDs (which
don’t change) and the actual text content can be switched without
affecting the game code. Assigning the IDs to text strings is a quite
complicated task, because it’s hard to manage large quantities of text
and avoid duplicities, not to forget anything etc. Of course, it depends
on the game type, hence on the amount of text. In case of simple action
games which only contain a couple of strings such as “New game” and
“You are dead” it’s easy to maintain a string table by hand, but a large
RPG or adventure game will probably require some kind of utility which
will do the dirty work for you.
The approach I’m using is to develop the game (relatively) ignoring
the future translation, and only when the game is finalized enough
(namely the texts are more or less final), I use some kind of tool to
extract all the localizable texts from the game files and to export them
to a string table. During this process every text is also assigned a
unique ID, and the original text
occurrences in game files are replaced by this ID.
This approach brings some pitfalls, though:
1) How to extract all the texts used in the entire game? I’ll give
you a single advice: try to design the game data files structure so that
it’s as easy as possible. The most important thing to remember is, forget about texts
hard coded in the game executable. All the texts should be stored in
data files, or in the game scripts. It’s better to use text-based data
files, not binary ones. The popular XML format is a good candidate,
because when choosing the right structure you’ll be able to extract all
the localizable texts using some generic tool, which doesn’t even have
to know the actual structure of your game files. For example, imagine an
XML file describing units in a real-time strategy game:
<?xml version="1.0"?>
<Units>
<Unit>
<Name Localizable="true">Archer</Name>
<Hitpoints>100</Hitpoints>
<Script>archer.script</Script>
<Description Localizable="true">Ineffective for close combat,
but deadly for ranged attacks.</Description>
</Unit>
<Unit>
<Name Localizable="true">Catapult</Name>
<Hitpoints>500</Hitpoints>
<Script>catapult.script</Script>
<Description Localizable="true">A slow but very powerful unit.</Description>
</Unit>
</Units>
Notice that the “Name” and “Description” entities are marked with a
“Localizable” attribute. That means we can make a generic tool which
will load an XML document, scan all the entities (without looking for
any specific names) and if they contain the “Localizable” attribute,
their content will be exported to a string table. The advantage of the
XML format is that there are many ready-to-use parsers available, so you
don’t have to write your own. But of course, you can use a similar
approach when dealing with your custom file formats.
If your game uses some kind of scripting language, you’ll probably want to extract all the string constants from the scripts. In my opinion, the best approach here is to modify the script compiler itself, because it knows exactly when it encountered a string constant when compiling the script, including the exact line and column in the source file. It shouldn’t be a big problem to extract the string constant and replace it with an identifier. Sometimes scripts contain string constants which are not supposed to be localized. For this purpose I’m using an ignore list and a library of “known code patterns” to filter out irrelevant strings.
2) The process described above will yield the following results. All
the in-game texts will be stored in a string table and their original
location will be replaced by some unique identifier. Even though it’s
basically what we wanted to achieve, the problem is, that the original
files will become a little unreadable after that. Imagine the following
script:
Frank.Talk("Hi Joe, how are you?");
Joe.Talk("Yeah, I'm okay.");
Frank.Talk("I gotta go.");
After our intervention and after exporting the texts, the script will become something like this:
Frank.Talk("STRING0001");
Joe.Talk("STRING0002");
Frank.Talk("STRING0003");
While it’s pretty obvious what the first script does, the second one
is totally unreadable because of the absence of the texts, which can be a
problem for any subsequent script changes. There’s an elegant solution
I’ve seen in some LucasArts games. Just keep both the original text AND
the identifier in the source file, using some unambiguous syntax. For
example:
Frank.Talk("STRING0001|Hi Joe, how are you?");
Joe.Talk("STRING0002|Yeah, I'm okay.");
Frank.Talk("STRING0003|I gotta go.");
As you can see, the texts in the scripts are now stored like this:
“identifier | original text”. This approach brings us two advantages.
Firstly, the code is still readable, and secondly, there’s a chance to
get back to the original text if needed. For example, if the game cannot
find any string table, it will still run, but it will only display the
texts on the right side of the “pipe” character. Additionally, it’s easy
to re-export all the texts from the game files, because the extraction
tool will know
exactly which texts have already been exported, and which ones are newly
added (the ones not using the “pipe” syntax yet).
We haven’t looked yet at how does the generated string table look
like. The decision is up to you. I’m using a simple text format, where
each line contains one string. There’s the identifier, a “tab” character
separator and the actual localizable text. The translator can simply
open the file in any text editor and replace the texts with their
translated equivalents. The string table from our example would look
like this:
STRING0001 Hi Joe, how are you?
STRING0002 Yeah, I'm okay.
STRING0003 I gotta go.
And the German translator will send you back something like:
STRING0001 Hi Joe, wie geht's?
STRING0002 Alles bestens.
STRING0003 Ich muss weg.
You can improve the string table format to suit your needs. For
example, lines starting with semicolon can be treated as comments (the
game will ignore them when loading the table) so that the
translators/designers can add notes to the string table file, etc.
The string ID itself can be improved too. For example, you can use
some standardized format to encode the character speaking and/or the
location where it’s being used etc. These are things that make
translators’ work a lot easier.
In case of large games the designer will probably need to add
comments to the table, to describe the context of some of the texts,
otherwise the translators might get lost and the translation quality
will suffer.
Another useful feature is to allow string table entries to reference
other entries in the table. Typically one text appears in the game in
different situations. If you allow the inter-table references, the
translator can only translate the text once and “redirect” all the other
occurrences of the same text to the translated entry. I don’t recommend
exporting multiple occurences of the same string as a single string
table entry, because they might need different translations in other
language (depending on the context). Leave it to the translator to
decide which strings should be the same.
Using the string table in your program.
All right, now we know how to create the string table, but how are we
going to use it? Our game engine needs to load the table into memory.
Considering the data structure (an identifier with a text attached)
we’ll probably want to use some sort of hash table. If you’re
programming in C++, the STL map container is probably a good choice.
Loading the string table file should be a trivial matter. We’ll
simply read the file line by line; if the line is empty or starts with a
semicolon (i.e. it’s a comment), we’ll ignore the line. Otherwise we
split it into two parts: the part left of the “tab” character (the
identifier) and the part right of the “tab” (the actual text) and we’ll
store the pair in a hash table. Again, there’s a lot of space for
improvements, such as reporting
duplicate entries, reporting lines without a “tab” character (the
translator accidentally replaced it by a space) etc.
A special note about whether to keep all the texts in memory at once.
It might look like memory wasting to you, but the truth is most
“talkative” games only contain a couple of hundreds of kilobytes of
text. Very few games contain more than one megabyte of text. Considering
the average memory capacity of
today’s PCs, I think you can safely store the entire string table in
memory. But if you’re still not convinced, or if you’re targeting a
platform with limited memory (PocketPC), the solution can be dividing
the table into more parts, for example generic texts and then specific
texts for each level of the game etc. Then you’ll load and unload parts
of the string table as needed.
From the programmer’s point of view, it’s a good idea to encapsulate
the string table into a class, which will keep all the texts and provide
a method
(preferably a static one), which will accept the original text and
return a translated text. By “original text” I mean the text in the
“identifier|original text” format. The method will do the following:
- Check if the original text contains the identifier.
- If not, the text is not supposed to be translated, so simply return the original text we received.
- If yes, the method will split the original text to the identifier and the text part.
- Using the identifier, it will try to find a translated text in the string table.
- If the string table contains this identifier, a translated text is returned.
- If not, the method returns the textual part of the string it received.
That way we achieved that even if there’s no translated version of
the text, the game will still display SOMETHING (the original
untranslated version). Certainly better than displaying an empty string
or some error.
Once again, we can further improve the process. A debug version of the game can report texts with missing translations, or it can return the translated text including the string table ID, so that the ID is displayed on screen. That can be useful for beta testing the translation. If the testers encounter some text with wrong translation or some grammar mistake, they can write down the identifier of the text and report it back to the translator.
The only thing left is to find all the “end points” of the program,
where the texts are being used, and change these to call the method of
our string table class to replace the original text with a translated
one. What do I mean by “end points”? Typically these are the parts of
the code where the text is actually presented to the user. Whether it be
displaying it on screen or writing it to some text file etc. There are
usually relatively few such points in a game engine (if it’s well
designed, that is).
Let’s see one (dumb) example of how to add localization support to
our code. Let’s assume we have a piece of code, which displays a message
box whenever it encounters some critical error. The original code would
look like this:
MessageBox(NULL, "A critical error has occured.", "Error", MB_ICONERROR);
To display the message in another language, we’ll modify the code like this:
MessageBox(NULL, CStringTable::Translate("SYSSTR0001|A critical error has occurred."), CStringTable::Translate("SYSSTR0002|Error"), MB_ICONERROR);
CStringTable is the class encapsulating our string table, Translate() is a static method, which gets the original text as a parameter and returns the translated text
(described above).
The code got a bit more complicated, but not too much. Alternatively, you can use a macro to further shorten the code:
#define LOC(String) (CStringTable::Translate(String))
And the resulting code would look like this:
MessageBox(NULL, LOC("SYSSTR0001|A critical error has occurred."),
LOC("SYSSTR0002|Error"), MB_ICONERROR);
Similarly, you’ll need to modify all the pieces of your code where the texts are in some way presented to the user.
Storing the in-game texts
ASCII versus Unicode
Note: The following section is intentionally a little simplified.
I’m just trying to describe the concepts and motivations, not to make
any in-depth analysis of the history of character encoding 🙂 Also, for
further simplicity I’m going to use the term “ASCII” for any 8 bit
characters, even though, technically speaking, this term isn’t always
completely accurate.
Another area of interest is the actual storage of the game texts in memory. Historically, most programs were using (and are still using) texts stored in the form of 8 bit characters, i.e. one character takes one byte of memory. That means one can use up to 256 different characters. The ASCII norm, where this character storage originates, was designed to store just the basic Latin alphabet (a to z, A to Z), the numbers and the symbols. Even 7 bits were enough to describe those (128 characters at most). As the software was getting more and more complex, it became necessary to store various other characters, specific to other languages (most European languages, other than English, are using some kind of accented characters). The remaining unused 128 characters were used for storing these national characters. The problem is, if you take all the national characters used by various languages, you’ll get far more than 128 of them. That’s why the concept of so called “code pages” was introduced. There were several groups of national languages defined, and it was possible to switch the code page. That means, the 128 non-English characters were getting different meaning depending on what code page was currently being used. For example, for European languages we’d define groups like: Western European, Central and Eastern European, Cyrillic, Greek etc. Similarly the code page concept supports languages such as Arabic, Hebrew and others.
To make things even more complicated, there are several standards for
code pages, incompatible with each other. Some encoding was used in
DOS, other in Windows, other is an ISO standard, there are some local
standards…
What does that mean for game localization? If we are storing texts as
8 bit characters (i.e. the classical C “char” data type) we’re able to
display at most 256 different characters. If we want to support multiple
languages (we do!) we’ll have to deal with code pages. In reality, this
means one thing: our game must use different fonts for different code
pages. But we’ll talk about fonts in the next chapter.
I think it’s obvious that the concept of code pages is far from being
ideal. In programmer’s jargon I’d call it a “hack”, or how to stuff
infinite number of characters into 256 available cells as easily as
possible. Also notice that I didn’t mention
Asian languages yet…
The effort of solving the national characters problem once and for all resulted in a standard called Unicode. The goal of Unicode is to consolidate the fragmented code pages for various languages/regions and to define a standard set of all possible characters in all languages (or at least their vast majority 🙂
From the programmer’s point of view it’s important to notice, that –
as opposed to ASCII – one byte is no longer enough for storing a single
character. Currently Unicode defines over 90 thousands of different
characters. There are several standards for storing Unicode characters,
two most commonly used standards are UTF-8 and UTF-16. The “UTF”
abbreviation stands for
“Unicode Transformation Format”, the number is a number of bits used for
storage. You may object that neither 8 or 16 bits are enough to
describe 90,000 different characters. The point is, that a single
Unicode character can be described by multiple bytes. The number of
bytes is different for various characters. In case of UTF-8, one
character can take up one to five bytes (and “common” characters, i.e.
latin, only take one byte, therefore e.g. English text looks the same in
UTF-8 as it looks in ASCII). UTF-16 uses 16 bits (word) to store
characters, and even though in theory 16 bits are still not enough,
practically one 16 bit word is equal to one Unicode character (unless
you’re using some “obscure” language, such as ancient Japanese).
All right, that was a very brief introduction to Unicode, let’s get back to game localization.
If you want to avoid the code page problems or if you want to support
Asian languages, your game engine should store texts in Unicode format.
More and more modern games are using Unicode, simply because
localization is now an important part of game development process, and
Unicode makes localization easier.
Above I mentioned two ways of storing texts, UTF-8 and UTF-16. UTF-8
uses normal 8 bit chars, UTF-16 uses the wchar_t data type, which is
typically 16 bit integer (note: this is true for Windows platform; Linux
uses 32 bit wchar_t by default). One great advantage of UTF-8 is that
if you already have a finished old program using 8 bit characters, you
don’t need to change anything to store UTF-8 strings. That’s a great
time saver. Also all the standard C-style string manipulation functions
(strcpy, strlen etc.) will work on UTF-8 strings. But of course, you
must remember that one UTF-8 byte isn’t necessarily equal to one
character (as I said, one UTF-8 encoded Unicode character can take up
multiple bytes). It means, for example, that the strlen function will
return the number of bytes in the string, NOT number of characters! Also
if you’re accessing individual bytes of the string using the []
operator, you’re getting bytes, not characters. But even with these
disadvantages, UTF-8 is an (almost) ideal solution for those developers
who need to add Unicode support to their legacy code quickly and easily.
Nevertheless, if you are starting to design a new system, I recommend
using the wchar_t type for storing text strings. The ANSI C standard
now contains all the string manipulation functions in two versions: the
classical ones, working with “char” data type (strcpy, strlen…) and
their equivalents for the “wchar_t” type (wcspy, wcslen…). Similarly,
STL offers you both “string” and “wstring” types.
Of course, storing the texts is one thing, and displaying them on
screen is another thing. But we’ll get to that later in the Fonts
chapter.
Unicode on various platforms
Of course, the problem of national characters and localizations is
not game-specific and needs to be handled on operating system and
development platform levels. So let’s take a brief look at how some of
the most popular platforms deal with Unicode.
Linux was historically built on ASCII characters, therefore Unicode
support is commonly realized using the UTF-8 encoding on this platform,
and it became de facto standard for string interchange.
The situation on the Windows platform is a bit more interesting. Windows systems have been for a long time developed in two parallel branches. On one side, it was the Win9x branch (Win95, Win98, WinMe) targeted at low-end hardware. These systems were using 8 bit ASCII strings. On the other hand, the WinNT branch (WinNT, Win2000, WinXP, Vista) are using 16 bit characters from the beginning. But to maintain backward compatibility with programs written for Win9x, the NT-based systems must be able to handle ASCII strings as well. In reality, all the Windows API functions working with strings are provided in two variations. The functions use the same name, but the ones working with 8 bit characters end with “A” (for ANSI) and the ones working with 16 bit characters end with “W” (for Wide). For example, we’ll find the MessageBox function as MessageBoxA and MessageBoxW. The “A” functions only work as a stub, which converts the string parameters from ASCII to Unicode and calls the “W” function. On Win9x systems this conversion is also partially supported, but the other way around and only for a couple of selected functions. Fortunately, Microsoft later released a separate library called “Microsoft Layer for Unicode”, which allows programs written for Win9x to use the entire set of Unicode variations of API functions.
Unlike Windows NT, the Windows CE (Windows Mobile) platform for
mobile devices got rid of the compatibility and only supports 16 bit
characters. If you are developing a game for WinCE, you can still use
normal ASCII strings, but whenever you need to call some API function
expecting a string, you’ll need to convert the string to UTF-16.
Modern runtime platforms Java and .NET are using 16 bit characters
exclusively, even though they provide a large set of conversion routines
from/to other text encodings.
In the end of this chapter I’d like to mention two useful Windows API functions for converting strings between 8 bit and 16 bit characters. MultiByteToWideChar converts from 8 bit characters to 16 bit and WideCharToMultiByte converts from 16 bit characters to 8 bit. The important thing is that both of these function can (among others) handle UTF-8 encoded Unicode strings.
Fonts
In the previous chapters we were talking about how to separate
in-game texts from the rest of the game and how to store them in memory.
The result of all our efforts should be a localized text displayed on
screen. Of course, we need fonts to actually render the text, but there
are many possible approaches. Let’s take a closer look at some of them,
especially taking localization into consideration.
The traditional approach to game text display is to load a special texture, which contains all the characters, and then render each character as a single quad, textured by the appropriate part of the texture with the image of a specific letter. This approach is understandable and easily implemented. The problem is, the font texture only contains a limited set of characters, typically 256 of them, i.e. a single code page (more on code pages in the previous chapter). To be able to display texts for various languages, our game will need to be able to use different fonts for different code pages (for example a texture with Russian font will contain completely different characters than a texture with Greek font). Now, it depends on how the font texture is made. There are generally two ways. Either the texture is manually created by an artist using utilities such as Bitmap Font Builder or FONText, or the texture is automatically generated by the engine at runtime. In the first case it will be necessary to manually create font textures for all the code pages our game is supposed to support. In the second case the game has to know which code page should be used when generating the font texture. For example, if we are using Win32 API for generating the font, we have to create the font using the CreateFont function, which accepts a code page identifier as its 9th parameter.
As you can see, we’re once again sinking into the limitations of code
pages and ASCII strings. Besides, this approach has several more or
less fatal limitations:
- By painting the text ourselves, letter by letter, we’re bypassing the font rendering engine. No matter if we’re using Windows API, FreeType or some other font rendering library, these libraries are usually designed for rendering continuous blocks of text. This allows them to render the text with respect to certain typographic rules. For example, they’re using so called kerning, i.e. different space between various letter pairs (kerning pairs), which improves the look of the rendered text, especially for larger text sizes. If we are rendering the text ourselves, we’re missing these advantages (or we have to reimplement them).
- By painting text letter by letter we’re losing advanced text formatting options. And here we’re getting back to localization. Some languages, namely Hebrew and Arabic, are writing text from right to left (right-to-left, RTL). While Windows API directly supports RTL texts, and we can enable the support by specifying one parameter of the ExtTextOut function, when painting the text ourselves we have to handle RTL ourselves as well. You might say implementing right-to-left text rendering shouldn’t be too hard, but it’s more complicated that that. Hebrew and Arabic texts can contain parts written in Latin (for example names) or numbers, and these parts need to be rendered left-to-right. It’s so called bidi text (bidi = bi-directional).
- And there are the Asian languages. While in case of European languages we can use code pages, it’s not quite feasible to render all Chinese characters into a single texture. We have to look for other solutions.
The problems described can be solved using the following method.
Instead of using the existing font rendering engines just for painting
individual letters into a font texture, we can let them paint into the
texture the entire text we’re about to display on screen. That way we
can use all the advanced text formatting functions practically for free.
Of course, this approach also has its deal of disadvantages. Depending
on the amount of text the resulting
texture can be quite large and it will take a lot of video memory. Also,
it might be necessary to divide the texture into smaller parts,
depending on the capabilities of the video card.
Compared to the static font texture, this approach is more time and
resource demanding when generating the text texture. It’s not feasible
to re-generate the text texture in each frame. Fortunately, each text
caption typically stays on screen for some time, therefore it’s possible
to implement some sort of texture cache, which will hold, e.g. 10 most
recently rendered texts. The text texture will be generated in one
frame, and reused in all subsequent frames.
In any case, it’s up to your consideration to decide which of the
described ways will better suit your needs. I just tried to summarize
the advantages and disadvantages than might not be obvious at first.
Other aspects of game localization
In the previous chapters we talked about game localization,
especially about storing the game texts and their rendering using fonts.
These are no doubt the most important areas, but localization also
affects other parts of development. Let’s take a look at these in this
final chapter.
Graphics
There’s a single most important rule: try to avoid using any text
directly in graphics whenever possible. All the texts should be rendered
programmatically using fonts. Keep in mind that all the textures
containing text will need to be repainted for all language versions of
your game. If you are creative, you can sometimes avoid using texts
altogether. For example, don’t use buttons with text labels, use icons
instead. Or just display a textual tooltip when the mouse is hovered
over the button.
User interface
By user interface I mean various windows for saving game, main menu,
confirmation
dialogs etc. Even when designing the user interface you should think
about localization. Keep a lot of additional free space for all the text
labels. Remember that the translated text can be much longer than the
original. This is especially true if your original language is English.
English is in general quite brief and other languages usually need much
more space to describe the same thing. Also remember, that for
Asian languages you’ll probably need to use bigger fonts, so make the
text area slightly higher than necessary.
Funny fact: German is probably one of the most critical languages
when it comes to text lengthening. In Oblivion the player is picking up
healing potions, described as “Light potion of healing” in the
inventory. German localizers translated the name as “Schwacher Trank der
Lebensenergie-Wiederherstellung”, which wouldn’t fit in the reserved
screen space, so they had to abbreviate the name to
“Schw.Tr.d.Le.en.-W.”, which is… kind of hard to comprehend 🙂 It’s a
nice example of badly designed user interface, resulting in poor
localization.
Voiceovers and videos
If your game contains voiceovers, keep these sound files separate
from the generic sound effects. How to organize and name the voiceover
files depends on situation, but you can use the string identifier from
the string table to name the sound file. For example, if your string
table contains something like:
STRING0001 Hi Joe, how are you?
The voiceover file for this line would be stored in a file called
“string0001.mp3”. The game engine can then automatically find and play
the voice over for the line being displayed on screen. Also, if your
string table ID contains the character the line belongs to, you can
easily filter lines for a specific voice actor etc. Well designed string
table structure even allows you to generate entire scripts for each
voice actor.
Always allow subtitles display for videos in case the localizers
won’t be able to record localized voiceovers. You can use standard
formats for video subtitles, such as .SUB or .SRT, because there are
existing editing tools for them.
Keyboard input
If your game uses any kind of keyboard input, such as the player name or saved game description, don’t use DirectInput for this purpose, but the Windows API messages, like WM_CHAR. That way Windows does the dirty job of mapping various keyboard layouts to national characters, “dead keys” handling etc.
Cultural references
When designing the game try to avoid references and jokes specific
for one language and/or nation. They are hard to translate and
complicate the localization process. If you have to use these, try to
provide the translators with a sufficient description, so that they can
adapt it to their local cultural environment.
Try to avoid game situations that can be offending to some national
or religious groups (remember the Muhammad cartoons controversy, for
example).
Conclusion
This article concentrated on the often-omitted area of localization
readiness of game engines. We went through several most important
topics, revealed some of the possible pitfalls and tried to find
solutions. If you are planning to implement localization support in your
game, hopefully this article will help you to avoid some dead ends.
Thanks for reading.