Python CSV UnicodeDecodeError: fix the encoding without losing text

An error such as UnicodeDecodeError: 'utf-8' codec can't decode byte... usually means Python is reading the file with the wrong text encoding. It does not mean the commas are broken.

First try a fresh UTF-8 export from the program that created the CSV. That is usually easier and safer than guessing. If you cannot re-export it, the method below converts a confirmed source encoding into a new UTF-8 file without replacing the original.

Find out what the file actually contains

An encoding is the rule used to turn stored bytes into readable characters. UTF-8 and Windows-1252 use different rules, so the same bytes can produce different text—or an error.

Before changing your script:

1. Open a copy of the CSV in a plain-text editor. Confirm it contains a header and records, not an HTML error page or an Excel .xlsx workbook renamed to .csv.
2. Check the export settings or ask the sender which encoding was used. If there is a CSV UTF-8 option, make a fresh export and keep the original.
3. Inspect a few names or product descriptions with accents. For example, Café must remain Café, not Café.

Do not fix an import by adding errors="ignore" or errors="replace". Those options can delete or replace characters while allowing the program to finish. A successful decode is also not proof of the correct encoding: some wrong encodings accept the bytes but produce nonsense.

Use the confirmed encoding

You need Python 3.12 or newer and a terminal. Check py --version in Windows PowerShell or python3 --version in macOS/Linux Terminal. If needed, install Python from the official downloads page, reopen the terminal, and retry. No additional packages are needed.

Create a folder named csv-encoding and put a copy of the source file in it as export.csv. Save this script as convert_encoding.py in the same folder, using plain text and the exact .py extension:

import argparse
from pathlib import Path

parser = argparse.ArgumentParser(description="Convert a known encoding to UTF-8.")
parser.add_argument("file")
parser.add_argument("--encoding", required=True)
parser.add_argument("--output", required=True)
args = parser.parse_args()

try:
    if Path(args.file).resolve() == Path(args.output).resolve():
        raise ValueError("Use a different filename for the output.")
    # Preserve line endings; decoding stays strict.
    with open(args.file, encoding=args.encoding, newline="") as source:
        text = source.read()
    with open(args.output, "x", encoding="utf-8", newline="") as target:
        target.write(text)
    print(f"Created {args.output}. Check names and punctuation before importing.")
except (OSError, UnicodeError, LookupError, ValueError) as error:
    parser.exit(1, f"Stopped: {error}\n")

This is for small files that fit in memory. It converts text, not separators, columns, or spreadsheet types.

Open a terminal in the folder. On Windows, type powershell in File Explorer’s address bar. On macOS/Linux, type cd in Terminal, drag in the folder, and press Enter.

Only if the source is confirmed as Windows-1252, run this in Windows PowerShell:

py convert_encoding.py export.csv --encoding cp1252 --output export-utf8.csv

On macOS/Linux:

python3 convert_encoding.py export.csv --encoding cp1252 --output export-utf8.csv

Use the sender’s actual encoding instead of cp1252 when it differs:

A BOM is a short marker at the start of some text files. Its absence does not identify another encoding. Do not cycle through the table until one command happens to succeed.

Check the conversion and retry your original task

Open export-utf8.csv in a text editor. Compare several distinctive words with the source program or sender’s confirmed text. Include accented names, currency symbols, and punctuation if present.

Then use export-utf8.csv in your original Python import with encoding="utf-8" and newline="" when reading CSV. For a ready-made structure check, the CSV reading guide accepts UTF-8 files and reports malformed rows.

Your result is a separate readable file that the import can decode. It is not proof that the data, delimiter, or column names are correct. A semicolon-separated export still needs the correct CSV reader settings or a comma-separated re-export.

If it still fails

• Another decoding error: the stated encoding may be wrong, the file may be damaged, or it may not be text. Ask for a new export; do not weaken error handling.
• Readable but mangled names: stop using that result. Return to the original and confirm the encoding. Already-corrupted text cannot reliably be repaired by another conversion.
• First header looks odd: a UTF-8 signature may have been read as ordinary text. Use utf-8-sig for that confirmed input.
• Output already exists: pick another name, such as export-utf8-2.csv. Existing files are never overwritten.

The script runs once and stops. Keep the source until you have checked the converted file and completed the intended import.

Sources and useful links

• Python Unicode HOWTO — explains encodings and why ignoring decode errors can lose text.
• Python codec names — reference for UTF-8 signatures, UTF-16, and Windows-1252.
• Python CSV reference — explains explicit encodings and CSV newline handling.