Text Files: Encodings and Parsing Text

In the previous article we covered the basics of working with files. Here comes what is specific to text: encodings and parsing the contents line by line.

Encodings

On disk a file is just a sequence of bytes. To turn them back into letters, you need to know which scheme to use for the conversion: many such schemes have been invented, and different languages historically ended up with different ones.

The modern standard is UTF-8: it covers every alphabet in the world at once, including emoji 😊. The rule for a beginner is simple: write and read in UTF-8, stating the encoding explicitly with the encoding parameter.

Python 3.13
1234567
text = "Привет, мир! Hello, world!"

with open('text_utf8.txt', 'w', encoding='utf-8') as file:
    file.write(text)

with open('text_utf8.txt', 'r', encoding='utf-8') as file:
    print(file.read())
Привет, мир! Hello, world!

Trouble starts when the encodings don't match: the file was written using one scheme and is read using another. Let's try to read our file as ascii — an old encoding that only has Latin letters, digits, and punctuation:

Python 3.13
12345678
with open('text_utf8.txt', 'w', encoding='utf-8') as file:
    file.write("Привет, мир! Hello, world!")

try:
    with open('text_utf8.txt', 'r', encoding='ascii') as file:
        print(file.read())
except UnicodeDecodeError as e:
    print(f"Decoding error: {e}")
Decoding error: 'ascii' codec can't decode byte 0xd0 in position 0: ordinal not in range(128)

The very first byte of a Cyrillic letter doesn't fit into ascii, and Python stops with a UnicodeDecodeError. The opposite situation looks the same: writing Cyrillic in an encoding that doesn't know it raises a UnicodeEncodeError.

The try/except construct is here only to keep the program from breaking off on the error. It has a lesson of its own later in the course, so there is no need to dig into it now.

Parsing a Configuration File

Program settings are often kept in a text file: every line is a "key = value" pair. Below are two files side by side: config.ini with the settings and main.py, which builds a dictionary out of them.

1def read_config(filename):2  config = {}3 4  with open(filename, 'r', encoding='utf-8') as file:5      for line in file:6          key, value = line.split('=', 1)  # cut at the first '=' only7          config[key.strip()] = value.strip()8 9  return config10 11 12settings = read_config('config.ini')13 14print(settings)15print(f"Theme: {settings['theme']}, language: {settings['language']}")16 

Run main.py, and the output is:

{'theme': 'dark', 'language': 'en', 'autosave': 'True'}
Theme: dark, language: en

Test Your Understanding

A file was written in UTF-8 but opened with encoding='ascii'. What happens?

In the next article we'll look at structured data formats — JSON and CSV.