- \n
- file names in the Windows Filing System not being recognised<\/li>\n
- string manipulations yielding unexpected results<\/li>\n
- text displayed to the user being completely unintelligible, or sprinkled with \ufffd<\/li>\n<\/ul>\n
This article sets out the background, describes the problems and recommends solutions. It is aimed as authors working with \u0192h<\/span><\/span>7; as of 2022, most new plugins are likely to be written for \u0192h<\/span><\/span>7 only, and the language pack facility in \u0192h<\/span><\/span>7 makes it more likely that unexpected characters will be encountered.<\/p>\n
Developers of plugins aimed at multiple users (typically published via the Plugin Store) should assume that they need to cater for all possible characters. Authors of plugins for private use may encounter fewer problems but might still find the recommendations helpful.<\/p>\n
Character Encodings Relevant to Plugin Authors<\/h3><\/summary>
There are a number of character encodings relevant to plugin authors (in roughly historical order):<\/p>\n
ASCII<\/a><\/h4>\n
ASCII supports unaccented English characters, digits 0-9, plus some punctuation and other symbols, and control characters such as Tab or Line Feed (New Line), to a total of 128 codes (the bottom 7 bits of a single byte). Not coincidentally (as many later encodings chose to align with ASCII), ASCII maps the same single byte codes to the same characters as some encodings that came later, which has the useful effect that if all you need is the upper- and lower-case English alphabet, you probably won’t see any problems. But once you wander outside the English walled garden, more characters are needed, and things get more complicated.<\/p>\n
‘ANSI’\/Windows Code Pages<\/a><\/h4>\n
‘ANSI’ is not a character encoding. Rather, it is a misnomer used to refer to the collection of Windows code pages introduced by Microsoft and others during the 1980s and 1990s to support more languages than English, and still supported (more or less) today.<\/p>\n
Each character in a code page is encoded using a single byte. The first 128 character codes mirror ASCII; then (up to) another 128 locale-specific characters are added. One major consequence of the approach is that the same byte value represents many different characters or none.<\/p>\n
For example:<\/p>\n
- \n
- Windows-1252<\/a> (the Latin alphabet), which is is the most-used single-byte character encoding in the world (and confusingly is also often called ‘ANSI’) uses byte value DE for the letter \u00de\u00a0<\/strong>(because we all need a thorn<\/a>, right, especially if our ancestors ran \u00dee Olde Cake Shoppe?)<\/li>\n
- Windows-1253 (Greek) uses byte value DE for \u03ae<\/strong>.<\/li>\n
- Windows-1255 (Hebrew) doesn’t use byte value DE at all.<\/li>\n<\/ul>\n
These characters may not seem to you to be particularly useful, but they illustrate the point that text encoded using one code page may yield unexpected results if it’s decoded using another.<\/p>\n
Another problem arises because there are languages (e.g. Thai) for which 128 characters are not enough. Thai may not be a huge issue for plugin authors, but there are places in Europe whose alphabet is not wholly supported by e.g. Windows 1252, for example the Dutch ij<\/strong> or the Polish \u0142<\/strong>.<\/p>\n
So, code pages aren’t an entirely satisfactory solution to handling characters from multiple languages.<\/p>\n
Unicode<\/a><\/h4>\n
Unicode is a character encoding standard <\/i>that sets out to cover every reasonable writing system in the world (and a few unreasonable ones as well, like Klingon) without ambiguity. It defines unique ‘code points’ for every character in every language, and it can be implemented by various character encodings (Unicode Transformation Formats or UTFs) that map code points (and thus characters) to unique byte sequences. Two UTFs are relevant to plugin authors: UTF-16 and UTF-8.<\/p>\n
UTF-16<\/a><\/h4>\n
UTF-16 encodes all characters in either 2 or 4 bytes. It is the only popular character encoding that is incompatible with ASCII (e.g. the English character X coded in ASCII is NOT the same code as character X coded in UTF-16.) It comes in 2 variants, depending on the underlying architecture or Endianness<\/a> of the computer system involved: UTF16-LE and UTF16-BE. Windows uses UTF-16LE, with a BOM (byte order mark)
0xFF 0xFE<\/code> at the start of a character stream to signify which variant is in use.<\/p>\nMicrosoft was an early adopter of Unicode. They used UTF-16 widely within their operating and filing systems, and sometimes for plain text and word processing files, and recommended it to application developers. (A word of caution: when reading Microsoft library documentation in your idle moments, any reference you find to ‘Unicode’ should almost always be interpreted as a reference to ‘UTF-16LE’).<\/p>\n
Almost all of the computing and Internet world adopted UTF-8<\/a>, not UTF-16, and in 2019 Microsoft changed their stance and started recommending UTF-8 to application developers; however, changing all their existing system code to use UTF-8 would be a huge endeavour, so many Microsoft libraries etc. still assume UTF-16 (or Windows Code Pages\/’ANSI’).<\/p>\n
<\/a>UTF-8<\/a><\/h4>\n
UTF-8 is now the dominant character encoding in computing and the Internet. It encodes all characters in 1,2,3 or 4 bytes. The first 128 characters of UTF-8 correspond one-to-one with ASCII, so that valid ASCII text is also valid UTF-8 text.\u00a0UTF-8 does not require a BOM, nor is one recommended, but it is sometimes added at the start of a character stream in the form
0xEF 0xBB 0xBF<\/code>. (If a BOM is added, it stops text from being backwards compatible with ASCII.) Where \u0192h<\/span><\/span> uses UTF-8 it often includes a BOM.<\/p>\n<\/div><\/details><\/div>\nWhere Might Encoding Issues Occur?<\/h3>\n
Encoding issues may occur anywhere that a plugin\u00a0interacts with external entities\/resources, or within the plugin\u00a0when manipulating strings.<\/p>\n
![\"\"](\"data:image\/svg+xml;base64,PHN2ZyB3aWR0aD0iMSIgaGVpZ2h0PSIxIiB4bWxucz0iaHR0cDovL3d3dy53My5vcmcvMjAwMC9zdmciPjwvc3ZnPg==\")
<\/p>\nMost issues can be avoided by adopting UTF-8 wherever possible, and the steps to do this are detailed in Recommendations<\/a> below, following detailed explanations of the issues involved.<\/p>\n
Accessing Project Data<\/h3><\/summary>
The following encodings are supported for project\u00a0GEDCOM\u00a0files and plugins\u00a0in \u0192h<\/span><\/span>7.<\/p>\n
- Windows-1253 (Greek) uses byte value DE for \u03ae<\/strong>.<\/li>\n
- Windows-1252<\/a> (the Latin alphabet), which is is the most-used single-byte character encoding in the world (and confusingly is also often called ‘ANSI’) uses byte value DE for the letter \u00de\u00a0<\/strong>(because we all need a thorn<\/a>, right, especially if our ancestors ran \u00dee Olde Cake Shoppe?)<\/li>\n
![\"\"](\"https:\/\/fhug.org.uk\/kb\/wp-content\/uploads\/2022\/01\/Character-encoding-1024x593.jpg\")
<\/p>\n