Devblog: Working towards Unicode

[Krom]

This is an open-ended article about Unicode in KaM Remake. Easy at it sounds, Unicode is better to have than not. However there are few complications that need to be sorted out and YOUR opinion might help.

http://www.kamremake.com/devblog/adding ... e-support/

Your advice and opinions are welcome in comments and in this thread!

[kocsis1david]

Some programming languages and .NET uses 2 byte letters, it can contain any currently used letter and still easy to use. Only historical letters, e.g. egyptian hieroglyphs cannot be encoded with it, but it's probably not needed. I'm really for fixed size character size, even if 4 byte is used. Memory is a smaller problem than performance, and simplicity is always good in my opinion. Windows API is also using LPWSTR, which is a pointer to 2 byte chars.

[sado1]

If there was a simple way to tab between/copy and paste all the player names in the lobby, then Unicode names wouldn't be a problem anymore.

[Shadaoe]

I think player names and mission names should stay ANSI, otherwise it'll easily get messy whenever you want to type one of these, and as you pointed out lobby passwords should use letters that everyone can type

[Nissarin]

I think that game files should be ascii only, you can always add some metadata (e.g. something like one txt per locale in mission folder) which will contain localized names of mission name/description, etc.
As for player names/passwords - ascii IMHO it would be a lot easier on everybody (might be a good idea to ask some Chinese players), it's just easier to communicate, copy-paste names in chat is just ridiculous idea. Passwords don't matter that much, since the player hosting the game will usually discus it with other beforehand but I don't think it's worth the trouble to have them localized.
Besides that if you ever wanted to implement some external authorization then anything other than ascii is asking for troubles.


Rant
Personally I think that utf8 is a better choice, simply because it's compatible with ascii so it's less messy to implement in existing project (it's somewhat compatible with ascii only libraries), utf16 in the other hand must be supported by any piece of code which deals with strings, unless you live in Asia utf16 is useless.
/Rant

[woloszek]

Wow Krom ! You say in many languages sorry I must wrote this

[Krom]

@Kocsis: UTF16 is just like that, it uses 2byte chars unless it needs to encode something really rare, then it uses 2 more bytes.

@sado1: Still it would be an issue if you would have to write some Chinese players nicks outside the lobby.

@Shadaoe: I'm inclined towards this more and more after writing the article and reading the comments

@Nissarin: UTF16 is more straight in processing. For now we just assume it's 2 byte chars (ignore the 4byte chars).

@Woloszek: Haha, thanks

[kocsis1david]

I meant one character is exactly 2 byte long, not more. If there are char above U+FFFF, it works as it would be two different letter.
So yours works the same if it ignores 4 byte chars.

[Krom]

@Kocsis: That's exactly how we are going to handle this for now