Unicode, UTF-8, and Character Encoding: What Every Developer Should Know

You copy a name from a web form and it saves to your database as é. You open a CSV in Excel and the accented characters are all wrong. A user pastes an emoji into a text field and your entire MySQL insert fails with a cryptic error. You add a BOM to a file so Excel reads it correctly, and now your shell script breaks on the first line.

These are character encoding bugs. They are among the most disorienting issues a developer can face because the symptoms look random — the same string can display correctly in one context and turn into garbage in another. The cause is almost always the same: two pieces of software disagree on how bytes map to characters.

This guide explains that mapping from the ground up: what character encodings are, how Unicode and UTF-8 work, and how to diagnose and fix the most common real-world bugs.

The Pre-Unicode Era: ASCII and the Extended ASCII Chaos

Every character stored in a computer is ultimately a number. A character encoding is the lookup table that maps numbers to characters and back.

The first widely-adopted encoding was ASCII (American Standard Code for Information Interchange), published in 1963. ASCII uses 7 bits, giving it 128 slots (0–127). That covers the English alphabet in upper and lower case, the ten digits, common punctuation, and 33 non-printing control characters like newline and tab.

128 characters is enough for English. It is nowhere near enough for French, German, Spanish, Arabic, Chinese, Japanese, or the thousands of other written languages in the world.

The Extended ASCII Problem

The 8th bit in a byte was available, which meant 128 more slots (128–255). Different organizations and operating systems rushed to fill those slots differently:

• Windows-1252 (also called CP1252): Used by Microsoft Windows in Western Europe. Puts characters like é, ñ, and € in the upper half.
• ISO-8859-1 (Latin-1): An international standard for Western European languages. Overlaps significantly with Windows-1252 but is not identical — the range 128–159 is control characters in Latin-1 but printable characters in Windows-1252.
• ISO-8859-5: Covers Cyrillic.
• ISO-8859-6: Covers Arabic.
• Shift JIS, EUC-JP, Big5: Entirely different multi-byte schemes for East Asian languages.

The result was that a file created on one system would display as garbage on another unless both agreed on the same encoding — and there was no reliable way to signal which encoding a file used. Software guessed, and guessed wrong constantly.

This is the problem Unicode was created to solve.

Unicode: One Character Set for Every Language

Unicode is not an encoding. It is a character set — a massive lookup table that assigns a unique number to every character in every writing system in use, plus thousands of symbols, emoji, and historical scripts.

Each entry in that table is called a code point, written as U+ followed by a hexadecimal number. A few examples:

The Unicode standard currently defines over 140,000 characters and is updated regularly.

Unicode Planes

Unicode code points range from U+0000 to U+10FFFF, giving just over 1.1 million possible slots. That space is divided into 17 planes of 65,536 code points each.

• Plane 0 — Basic Multilingual Plane (BMP): U+0000 to U+FFFF. Contains virtually every character in common use today: Latin, Greek, Cyrillic, Arabic, Hebrew, CJK, and more.
• Plane 1 — Supplementary Multilingual Plane (SMP): U+10000 to U+1FFFF. Emoji, historic scripts, musical notation.
• Planes 2–16: Additional CJK unified ideographs, rare historic scripts, and unassigned space.

The emoji that break MySQL tables live in Plane 1. That detail matters, as we will see shortly.

UTF-8: How Unicode is Actually Stored

Unicode defines what number each character gets. It does not specify how to store those numbers as bytes on disk or in memory. That is the job of a Unicode encoding.

UTF-8 is by far the most widely used Unicode encoding on the web and in modern software. It is variable-width, meaning different characters take different numbers of bytes:

How UTF-8 Byte Sequences Work

UTF-8 is self-synchronizing. The first byte of any character tells you how many bytes follow:

• 0xxxxxxx — 1-byte character (ASCII). Any valid ASCII file is valid UTF-8.
• 110xxxxx 10xxxxxx — 2-byte character.
• 1110xxxx 10xxxxxx 10xxxxxx — 3-byte character.
• 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx — 4-byte character.

Continuation bytes always start with 10, so you can immediately tell whether any given byte is the start of a new character or the middle of one. This makes it possible to resynchronize after reading a partial stream.

Concrete example — encoding é (U+00E9):

U+00E9 in binary is 11101001. That falls in the 2-byte range. The template 110xxxxx 10xxxxxx produces 11000011 10101001 — bytes 0xC3 0xA9 — which is exactly what UTF-8 stores for é.

Why UTF-8 Dominates

UTF-8 has three properties that made it win:

1. ASCII compatibility. Any ASCII-only file is valid UTF-8. Migrating existing systems did not require touching files that had no non-ASCII content.
2. Space efficiency for Latin scripts. English text uses one byte per character, the same as ASCII. Only characters outside ASCII cost more bytes.
3. No byte-order ambiguity. The byte structure is unambiguous without any header. (Compare this to UTF-16, which requires a byte-order mark.)

UTF-16 and UTF-32: When You Will Encounter Them

UTF-8 is not the only Unicode encoding, and some environments use others.

UTF-16 encodes BMP characters as two bytes (one 16-bit code unit) and supplementary plane characters as four bytes (two 16-bit code units, called a surrogate pair). Windows internals, Java strings, JavaScript strings (at the engine level), and .NET strings all use UTF-16 internally.

UTF-32 uses exactly four bytes per character, always. It wastes space but makes random access by character index trivially fast. Some database internals use it.

The Byte Order Mark (BOM)

UTF-16 has an endianness problem. A two-byte sequence can be stored as big-endian (0x00 0x41 for A) or little-endian (0x41 0x00). A receiver needs to know which order was used.

The solution is the Byte Order Mark: code point U+FEFF placed at the very start of the file. In UTF-16 big-endian, this appears as the bytes FE FF. In UTF-16 little-endian, it appears as FF FE. A reader that sees one or the other knows the byte order.

UTF-8 technically has a BOM too (EF BB BF) but it is meaningless for UTF-8 because UTF-8 has no byte-order ambiguity. This is where it causes problems:

• Excel and CSV files. Excel uses the UTF-8 BOM as a signal to interpret a CSV as UTF-8. Without it, Excel defaults to the system locale encoding (often Windows-1252), mangling accented characters. Many developers add the BOM specifically to make Excel happy.
• Shell scripts. A shell script that starts with #!/bin/bash must have that sequence as the very first bytes. If a UTF-8 BOM precedes the shebang, the kernel sees EF BB BF 23 21 and does not recognize the shebang, causing the script to fail or run with the wrong interpreter.

The rule of thumb: add the BOM for Excel-targeted CSV files, never add it to shell scripts or JSON files.

Common Real-World Encoding Bugs

Mojibake: Reading UTF-8 as Latin-1

Mojibake (文字化け) is the Japanese term for garbled text caused by an encoding mismatch. The most common variant on the modern web: a UTF-8 string is read as if it were Latin-1 (ISO-8859-1 or Windows-1252).

The character é is U+00E9. In UTF-8 it is stored as the two bytes 0xC3 0xA9. If a program reads those two bytes as Latin-1, it sees two separate characters: 0xC3 is Ã and 0xA9 is ©. The string café becomes café.

That specific corruption pattern — é for é, è for è, à for à — is the telltale sign of UTF-8 being interpreted as Latin-1. You will see it most often in:

• Databases where the connection charset was not set to UTF-8 on insert.
• Emails sent through older mail servers.
• Files copied between systems with mismatched locale settings.

The fix is to identify where the mismatch occurs and correct the encoding declaration at that layer — not to patch the corrupted string after the fact.

The Replacement Character (U+FFFD)

When a decoder encounters a byte sequence that is not valid in the expected encoding, it has to do something. The Unicode standard recommends substituting the replacement character U+FFFD, displayed as <0xFFFD>. If you see ? boxes or the <0xFFFD> character in output, a decoder encountered bytes it could not interpret.

Common causes:

• Truncating a multi-byte UTF-8 sequence. If you cut a string at a byte boundary that falls inside a 3-byte CJK character, the partial sequence is invalid.
• Passing a Latin-1 encoded file to a strict UTF-8 decoder.
• Receiving data from a source that mixes encodings.

MySQL's utf8 vs utf8mb4: The Emoji Problem

This is one of the most well-known encoding bugs in web development. MySQL has two character sets named utf8 and utf8mb4.

MySQL's utf8 is not real UTF-8. It supports only code points up to U+FFFF — exactly the Basic Multilingual Plane — using a maximum of three bytes per character. Emoji live in Plane 1 (U+10000 and above) and require four bytes. When you try to insert a four-byte UTF-8 character into a column defined as utf8, MySQL silently truncates the value or throws an error.

utf8mb4 is actual UTF-8 with full four-byte support. It was introduced specifically to handle emoji and other Supplementary Plane characters.

The correct approach: always use utf8mb4 for new tables and databases. Also set the connection character set and collation explicitly:

ALTER DATABASE mydb CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
ALTER TABLE posts CONVERT TO CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

And set the connection parameter in your application config:

charset=utf8mb4

Using utf8 in MySQL is a legacy footgun. Treat it as deprecated.

String Length vs Character Count in Python

Python 2 stored strings as byte sequences by default. len("café") returned 5 for a UTF-8 string — because é takes two bytes — not 4. Correct multi-byte handling required explicit u"" unicode literals and careful encode/decode discipline.

Python 3 fixed this: str is a sequence of Unicode code points, so len("café") returns 4. Byte data moved to the bytes type with explicit .encode() and .decode() calls.

One subtlety remains. Python 3's len() counts code points, not grapheme clusters. The character é can be represented as a single code point (U+00E9) or as e (U+0065) plus a combining acute accent (U+0301). Both render identically but produce different len() values. Multilingual applications that expose a character count to users should use a Unicode segmentation library rather than raw len().

When you inspect JSON data containing multilingual strings you may notice this — what looks like a single character could be two code points internally.

Encoding Comparison Table

How to Detect and Fix Encoding Issues

Detection

In Python:

import chardet
 
with open("mystery_file.txt", "rb") as f:
    raw = f.read()
 
result = chardet.detect(raw)
print(result)  # {'encoding': 'Windows-1252', 'confidence': 0.73, 'language': ''}

The chardet library uses statistical analysis to guess encoding from byte patterns. The confidence score is honest — encoding detection is heuristic, not deterministic.

In the browser: Open DevTools, go to the Network tab, find the response for an HTML page, and check the Content-Type header. It should read text/html; charset=utf-8. If it is absent or wrong, the browser will guess, often incorrectly.

In a database: Check the character set at the database, table, and column level:

SHOW VARIABLES LIKE 'character_set%';
SHOW CREATE TABLE posts;

Fixing a Mojibake Database

If data was inserted as UTF-8 but stored as Latin-1 (a common migration mistake), you can sometimes recover it:

-- Read the corrupted data as binary, then interpret as UTF-8
UPDATE posts
SET body = CONVERT(CAST(CONVERT(body USING latin1) AS BINARY) USING utf8mb4)
WHERE body REGEXP '[Ã]';

This only works if the original bytes are intact and were merely mis-labeled. If the data was truly corrupted — bytes were dropped or substituted — recovery is not possible without a backup.

Prevention

• Always declare encoding explicitly. Never rely on defaults.
• Set <meta charset="UTF-8"> in every HTML document.
• Set Content-Type: application/json; charset=utf-8 on API responses.
• Specify the connection charset when opening database connections.
• Use UTF-8 in your editor and configure your version control system to enforce it for text files.

For encoding tasks beyond character sets — such as converting binary data to text-safe representations — the encode and decode text tool on FileCrank handles Base64 conversion without any setup. The existing encoding article covers Base64, URL encoding, and HTML entities if you need those. If you are working through configuration files and need to verify that multilingual strings look correct, you can inspect JSON data directly in the browser.

What to Take Away

Character encoding bugs feel mysterious because the failure mode — garbled or missing characters — shows up far from the actual cause. But the underlying mechanism is simple: two programs are using different lookup tables to convert bytes to characters.

Keep these three rules in front of you:

1. Always use UTF-8 everywhere unless a specific system forces something else. Set it explicitly; never rely on defaults.
2. In MySQL, use utf8mb4, not utf8. The difference matters the moment a user sends an emoji.
3. When you see mojibake, find the boundary. Look for where bytes cross a system that does not know their encoding. Fix the declaration at that point, not the symptoms downstream.

Understanding the mechanics — code points, byte sequences, encoding declarations — means you can reason through a new encoding bug systematically instead of guessing. The next time you see é in your output, you will know immediately what happened and where to look.

All FileCrank tools run entirely in your browser. No files are uploaded to a server. If you are working with text that contains multilingual content, try the JSON Viewer or the Base64 Encode/Decode tool — both handle Unicode correctly without any configuration.