What this usually means
Pandas dtype unexpected conversion occurs because pandas' type inference is designed to preserve data but often chooses a 'safe' type that may not match your expectations. The most common trigger is NaN or None: integer columns cannot hold NaN in standard numpy int64, so pandas promotes to float64. Similarly, mixed-type columns (e.g., ints with 'NA' strings) become object dtype. Pandas 0.24+ introduced nullable integer types (Int64 with capital I) to address this, but conversion can still happen silently during merges, groupbys, or when using apply(). Another frequent cause is the 'categories' dtype being assigned during concat or Categorical operations, which is invisible until you check .cat.categories.
The first ten minutes — establish facts before touching code.
- 1Run df.dtypes immediately after loading data – compare to expected schema
- 2Use df.info(verbose=True, null_counts=True) to see non-null counts and dtypes per column
- 3Check for NaN in numeric columns: df.isnull().any() or df['col'].isnull().sum()
- 4Test with a small sample: df.head(100).to_dict('list') to see actual mixed types
- 5Enable pandas options: pd.set_option('future.no_silent_downcasting', True) to catch future warnings
The specific files, logs, configs, and dashboards that usually own this bug.
- searchread_csv() call – check dtype parameter or converters for specific columns
- searchconcat() calls – ensure all DataFrames have same dtypes, use dtype backfill
- searchgroupby().agg() – inspect result dtypes; .agg() may change int to float for count/mean
- searchfillna() operations – if filling with a string, numeric column becomes object
- searchapply() functions – if function returns None or mixed types, column dtype may change
- searchpd.to_numeric() errors – if errors='coerce', non-convertible become NaN and float
- searchSQL or Parquet import – schema might force strings for numeric-like values
Practical causes, not theory. These are the things you will actually find.
- warningNaN in integer column forces float64 (standard pandas behavior before nullable types)
- warningMixed types in same column (numbers and strings) → object dtype
- warningUsing groupby with as_index=False and agg('sum') on nullable integer returns float64
- warningConcat of DataFrames with different categorical categories recodes as object
- warningfillna() with a non-NaN value of different type (e.g., fillna(0) on string column becomes object)
- warningReading CSV with parse_dates=True but some dates are malformed → object dtype
- warningUsing pd.NA or pd.NaT in a column that started as int → promotes to Int64 (nullable) but can convert back to float on operations
Concrete fix directions. Pick the one that matches your root cause.
- buildUse pd.Int64Dtype() (nullable integer) for columns that may have NA: df['col'] = df['col'].astype('Int64')
- buildSpecify dtype in read_csv: pd.read_csv('file.csv', dtype={'col': 'Int64'})
- buildFor groupby, cast result dtypes explicitly: result['count'] = result['count'].astype('Int64')
- buildUse pd.Categorical() constructor with explicit categories to avoid silent conversion
- buildAfter concat, use df = df.convert_dtypes() to infer best nullable types
- buildFor datetime columns, use pd.to_datetime with errors='coerce' then dropna or fill with NaT
- buildSet pd.set_option('future.infer_string', True) to use StringDtype instead of object
A fix you cannot prove is a guess. Close the loop.
- verifiedRun df.dtypes and confirm expected dtypes (e.g., Int64, float64, datetime64[ns])
- verifiedCheck for NaN presence: df.isnull().sum().sum() should be zero if you filled
- verifiedVerify memory usage: df.memory_usage(deep=True) to confirm object columns are gone
- verifiedTest a round-trip: write to Parquet and read back, then compare dtypes
- verifiedRun a sample aggregation: df.groupby('key').agg({'value': 'mean'}).dtypes
- verifiedEnable pandas' own dtype warnings: pd.set_option('mode.chained_assignment', 'warn')
Things that make this bug worse or harder to find.
- warningBlindly using .fillna(0) on a column that should stay integer – it may become float
- warningAssuming .astype(int) will work on a column with NaN – it will raise ValueError
- warningUsing .convert_dtypes() without verifying it produces desired types (it may convert to string instead of object)
- warningIgnoring FutureWarning about silent downcasting – these will break in future pandas versions
- warningUsing .infer_objects() on data with mixed types – it often does nothing
- warningAssuming groupby('key')['int_col'].sum() returns int – it returns float if there's any NaN
The Mysterious Parquet Write Failure
Timeline
- 09:15Deployed new ETL pipeline that reads CSV, transforms, writes Parquet
- 10:30Airflow DAG fails with ArrowNotImplementedError: Unsupported type for column 'user_id': uint64
- 10:35Checked CSV schema: user_id column has values like 12345678901, no negatives
- 10:40Read CSV into pandas: df['user_id'].dtype shows int64 – not uint64?
- 10:45Found that after some filtering, user_id had NaN for null rows – dtype changed to float64
- 10:50Applied fillna(0) to user_id: dtype became float64, still not uint64
- 11:00Discovered that Parquet writer saw float64 and attempted to convert to uint64? No, actually the issue was that after fillna, column was float64 and PyArrow refused to cast to uint64
- 11:10Fixed by casting to pd.UInt64Dtype() after fillna: df['user_id'] = df['user_id'].astype('UInt64')
- 11:15Pipeline succeeded; logged the root cause in runbook
We had a simple ETL: read a CSV of user events, filter out some rows, then write to Parquet for downstream analytics. The CSV had a user_id column that was always a 11-digit integer, no negatives, no decimals. On dev, everything worked fine. But in production, the Parquet write started failing with a cryptic ArrowNotImplementedError: 'Unsupported type for column user_id: uint64'. I was confused because pandas showed int64, not uint64. How could it become uint64?
I dug deeper. The CSV had some rows with missing user_id. When pandas reads a column with missing integer values, it converts to float64 (because NaN can't be represented in int64). I had a step that filled those NaN with 0 using fillna(0). That kept the dtype as float64. Then, I had a transform that multiplied user_id by 1 (to ensure integer type?) – but that still kept float64. Finally, when writing to Parquet, PyArrow saw the column as float64 and tried to convert it to uint64 internally (because all values were positive integers), but float64 doesn't map cleanly to uint64 in Arrow, hence the error.
The fix was simple: after fillna, explicitly cast to nullable unsigned integer: df['user_id'] = df['user_id'].astype('UInt64'). That gave us a proper unsigned integer type that PyArrow could write directly. The lesson: never assume pandas keeps your intended dtype across operations. Always verify dtypes after any transformation that might introduce NaN, and use nullable types from pandas' extension arrays to avoid silent promotion to float.
Root cause
fillna(0) on a float64 column (due to NaN) kept float64, which PyArrow couldn't convert to uint64 on write.
The fix
Explicitly cast to pd.UInt64Dtype() after fillna to maintain unsigned integer type.
The lesson
Always check dtypes after any operation that could introduce NaN, and use nullable integer types to avoid unexpected float conversion.
Standard numpy arrays cannot represent NaN in integer types. When pandas encounters a missing value in an integer column, it must convert to float64 to store NaN. This is a deliberate design choice that predates nullable integer types. The only workaround before pandas 0.24 was to use object dtype, which is memory-inefficient.
Pandas 0.24 introduced nullable integer types like Int64 (capital I) that use pd.NA instead of np.nan and can handle missing values without conversion to float. However, these types are not used by default. You must opt in via astype('Int64') or pd.array(..., dtype='Int64'). Operations like groupby, merge, or concat may still revert to float64 if you're not careful.
When concatenating DataFrames that have categorical columns with different categories, pandas may convert the result to object dtype. This happens because the union of categories is not automatically computed. For example, df1['color'] has categories ['red','blue'], df2['color'] has ['green']. Concat gives object dtype, not a new categorical with three categories.
To avoid this, use pd.concat([df1, df2], ignore_index=True).astype('category') after concat, or manually unify categories before concat: df1['color'] = df1['color'].cat.add_categories(df2['color'].cat.categories). This is a common pitfall when combining datasets from different sources.
After groupby, agg functions like sum, mean, count return float64 if any group has NaN or if the column has nullable integer type. This is because the result dtype must accommodate potential NaN from empty groups. For count, it's especially annoying because count never returns NaN, but pandas still returns int64 by default? Actually, count returns int64 for non-nullable columns, but for nullable Int64, it returns Int64. However, if you use as_index=False and then reset_index, dtypes can change.
Best practice: after groupby, explicitly cast numeric columns back to desired type. For example: result['count'] = result['count'].astype('Int64'). Also consider using observed=True to avoid empty groups that introduce NaN.
Pandas' own missing value sentinels (pd.NA for nullable integers/strings, pd.NaT for datetime) behave differently from np.nan. Using pd.NA in a column that started as float64 will keep it float64, but using pd.NA in an Int64 column keeps it Int64. However, operations like sum() on an Int64 column with NA may return NA (not 0). This is correct but surprising if you expect np.nan behavior.
Mixing pd.NA with np.nan in the same column can cause dtype to become object. Always use a single missing value convention. For datetime, pd.NaT is the only missing value, but if you have a column with both pd.NaT and np.nan, it becomes object. Use pd.isna() to check for any missing value.
Pandas future versions (2.0+) will raise FutureWarning when a downcast happens silently, e.g., converting int64 to float64 due to NaN. You can enable this warning now with pd.set_option('future.no_silent_downcasting', True). This will alert you to every silent conversion, allowing you to fix before it breaks in future versions.
Also consider using the new 'future.infer_string' option to automatically use StringDtype instead of object for string columns. This reduces memory and prevents many dtype issues. Set it at startup: pd.set_option('future.infer_string', True).
Frequently asked questions
Why does my integer column become float64 after reading a CSV with missing values?
Pandas uses numpy arrays internally, and numpy cannot represent NaN in integer types. When the CSV has a missing value (empty cell or 'NA'), pandas reads the column as float64 so it can store NaN. To avoid this, either fill missing values during read (na_values=0) or use nullable integer type: pd.read_csv('file.csv', dtype={'col': 'Int64'}).
How do I convert a float64 column back to int64 without losing data?
First ensure there are no NaN values: df['col'] = df['col'].fillna(0) or drop rows with dropna(). Then convert: df['col'] = df['col'].astype('int64'). If you want to preserve NaN, use nullable Int64: df['col'] = df['col'].astype('Int64').
Why does groupby('key')['int_col'].sum() return float64?
If any group is empty or if the column has NaN, the result dtype becomes float64 to accommodate NaN. Even if there are no NaN, pandas may still return float64 for sum when using nullable integer types. To force int, use: result['sum'] = result['sum'].astype('Int64') or ensure no missing values and use .sum(min_count=1).
How can I prevent pandas from converting my string column to category on concat?
This happens when one of the input DataFrames has a categorical column. Before concat, convert all categorical columns to string or object: df['col'] = df['col'].astype(str). Alternatively, after concat, convert back: df['col'] = df['col'].astype('category'). To avoid entirely, use pd.concat with ignore_index=True but that doesn't always fix it.
What is the difference between pd.NA and np.nan? Which should I use?
pd.NA is pandas' own missing value sentinel that works with nullable types (Int64, StringDtype, etc.). np.nan is a float NaN that forces columns to float64. Prefer pd.NA when using nullable types, especially for integer columns. Use pd.NA consistently to avoid type promotion. You can convert np.nan to pd.NA with df['col'] = df['col'].where(df['col'].notna(), other=pd.NA).