What is Unicode?

Unicode is the international standard that assigns a unique number, called a code point, to virtually every character in every writing system — Latin, Cyrillic, Arabic, Chinese, emoji, mathematical symbols, and more. Before Unicode, dozens of incompatible encodings caused text to garble when moved between systems. Unicode's single, universal map lets any character be represented and exchanged reliably worldwide.

A code point is written as U+ followed by a hexadecimal number, such as U+0041 for the capital letter A or U+1F600 for a grinning-face emoji. The standard defines well over a million possible code points and currently assigns more than 149,000 characters. Crucially, code points are abstract numbers; how they are stored as bytes is a separate concern handled by encodings.

The most common encoding is UTF-8, which represents each code point in one to four bytes and is backward-compatible with ASCII for the first 128 characters. This efficiency and compatibility made UTF-8 the dominant encoding on the web. Other encodings like UTF-16 exist, and mismatches between the encoding a file was written in and the one used to read it are the classic cause of mojibake — text turned into garbage symbols.

Unicode introduces subtleties that matter in data work. Some visible characters are composed of multiple code points, such as an accented letter or an emoji with a skin-tone modifier, so counting 'characters' can differ from counting code points or bytes. Normalization forms reconcile different code-point sequences that render identically, and invisible or look-alike characters can hide in text, which is why detection and normalization tools are useful.