History of Character Codes

ASCII

In 1968, ASCII is standardized which defined numeric codes for various characters, with the numeric values running from 0 to 127. These characters contain the numbers 0-9, the letters a-z and A-Z, some basic punctuation symbols, some control codes that originated with Teletype machines, and a blank space - into the 7-bit binary integers. For the 8-bit PCs, bytes could hold values ranging from 0 to 255, but ASCII only went up to 127. So the ASCII was extended to hold more characters (128 ~ 255) (i.e. accented characters).

But 255 characters isn't still enough to hold various characters in the world (i.e. accented characters used in Western Europe and the Cyrillic alphabet used for Russian)

Unicode

Unicode started out using 16-bit characters instead of 8-bit characters to solve the problem above. 16 bits means that you can hold 65,536 distinct characters. Unfortunately, It turns out that even 16 bits isn't still enough to hold all of characters. So the modern Unicode specification uses a wider range of codes, 0-1,114,111 (0x10ffff in base-16).

Definitions

The Unicode standard describles how characters are represented by code points [1]. A Unicode string is a sequence of code points, which are numbers from 0 to 0x10ffff. This sequence needs to be represented as a set of bytes(meaning, values from 0-255) in memory. So this refers to a concept called encoding.

decoding/encoding

The process of converting the Unicode string (a sequence of code points) to a sequence of bytes is encoding, while the reverse process is encoding.

UTF-8 is the most commonly supported encoding probably. Encodings don't have to handle every possible Unicode character, and most encodings don't. UTF-8 uses the following rules:

# if the code point is < 128, it's represented by the corresponding byte value.

# if the code point is between 128 and 0x7ff, it’s turned into two byte values between 128 and 255.