pola-rs/polars

Confusing error message when reading CSV with duplicate column name

Open

#28,310 opened on Jul 9, 2026

View on GitHub
 (1 comment) (1 reaction) (0 assignees)Rust (2,826 forks)batch import
A-io-csvP-mediumacceptedbuggood first issuepython

Repository metrics

Stars
 (38,496 stars)
PR merge metrics
 (Avg merge 8d 3h) (156 merged PRs in 30d)

Description

Checks

  • I have checked that this issue has not already been reported.
  • I have confirmed this bug exists on the latest version of Polars.

Reproducible example

import io
import polars as pl

data = "a,a_duplicated_0,a\n1,2,3"
print(pl.read_csv(io.StringIO(data)))

Log output

Traceback (most recent call last):
  File "/home/elt/code/polars-test/csv_dup.py", line 5, in <module>
    print(pl.read_csv(io.StringIO(data)))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/elt/code/polars-test/.venv/lib/python3.12/site-packages/polars/_utils/deprecation.py", line 128, in wrapper
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/elt/code/polars-test/.venv/lib/python3.12/site-packages/polars/_utils/deprecation.py", line 128, in wrapper
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/elt/code/polars-test/.venv/lib/python3.12/site-packages/polars/_utils/deprecation.py", line 128, in wrapper
    return function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/elt/code/polars-test/.venv/lib/python3.12/site-packages/polars/io/csv/functions.py", line 581, in read_csv
    df = _read_csv_impl(
         ^^^^^^^^^^^^^^^
  File "/home/elt/code/polars-test/.venv/lib/python3.12/site-packages/polars/io/csv/functions.py", line 733, in _read_csv_impl
    pydf = PyDataFrame.read_csv(
           ^^^^^^^^^^^^^^^^^^^^^
polars.exceptions.ComputeError: found more fields than defined in 'Schema'

Consider setting 'truncate_ragged_lines=True'.

Issue description

Duplicate column names in the CSV file are renamed into name_duplicated_N, however a column with this name might already exist in the file. (This might actually happen if someone writes back this pl.DataFrame as a csv file.) When this happens, the previous column in the schema is overwritten, so the schema ends up with fewer fields than the dataframe, leading to an obscure error. It would be nice if this was caught and renamed until all names are unique.

Expected behavior

Raise a DuplicateError instead, printing the duplicated name.

Installed versions

--------Version info---------
Polars:              1.42.1
Index type:          UInt32
Platform:            Linux-6.18.33.2-microsoft-standard-WSL2-x86_64-with-glibc2.39
Python:              3.12.3 (main, Mar 23 2026, 19:04:32) [GCC 13.3.0]
Runtime:             rt32

----Optional dependencies----
Azure CLI            <not installed>
adbc_driver_manager  <not installed>
altair               <not installed>
azure.identity       <not installed>
boto3                <not installed>
cloudpickle          <not installed>
connectorx           <not installed>
deltalake            <not installed>
fastexcel            <not installed>
fsspec               <not installed>
gevent               <not installed>
google.auth          <not installed>
great_tables         <not installed>
matplotlib           <not installed>
numpy                <not installed>
openpyxl             <not installed>
pandas               <not installed>
polars_cloud         <not installed>
pyarrow              <not installed>
pydantic             <not installed>
pyiceberg            <not installed>
sqlalchemy           <not installed>
torch                <not installed>
xlsx2csv             <not installed>
xlsxwriter           <not installed>

Contributor guide