Skip to content

BUG: dtypes cast when reading JSON #44693

Open
@ConstantinoSchillebeeckx

Description

  • I have checked that this issue has not already been reported.
  • I have confirmed this bug exists on the latest version of pandas.
  • I have confirmed this bug exists on the master branch of pandas.

Reproducible Example

import pandas as pd
import json

# Correct behavior with pd.read_csv

csv_data = """
date,bool,good_str,good_int,good_float,mixed_types,int_w_missing,str_w_missing
2021-01-01,0,a,1,1.0,1,1,a
2021-01-01,1,b,2,2.2,b,,
"""
    
with open("foo.csv", "w") as f:
    f.write(csv_data)

# this correctly raises ValueError: could not convert string to float: 'b'
df_csv = pd.read_csv("foo.csv", dtype={"int_w_missing":"Int64", "mixed_types": "float"})

# ---------------

# Incorrect? behavior with pd.read_json

json_data = [
        {
            "date": "2021-01-01",
            "good_int": 5,
            "int_w_missing": 10,
            "good_str": "114117967",
            "mixed_types": "455590"
        },
        {
            "date": "2021-01-01",
            "good_int": 5,
            "good_str": "114011434",
            "mixed_types": "something"
        }
]


with open("foo.json", "w") as f:
    f.write(json.dumps(json_data))
    
# this does not raise anything
df_json = pd.read_json("foo.json", orient='records', dtype={"int_w_missing":"Int64", "mixed_types": "float"})

# checking dtypes we see "int_w_missing" nor "mixed_types" were cast
df_json.dtypes

Issue Description

Using dtype with pd.read_json:

  1. does not raise raise a ValueError when casting something like the str "something" to a float
  2. does not correctly cast to Int64 when casting a int field with missing values

Expected Behavior

  1. raise a ValueError when casting something like the str "something" to a float
  2. correctly cast to Int64 when casting a int field with missing values

Installed Versions

INSTALLED VERSIONS

commit : 945c9ed
python : 3.8.11.final.0
python-bits : 64
OS : Darwin
OS-release : 20.6.0
Version : Darwin Kernel Version 20.6.0: Mon Aug 30 06:12:21 PDT 2021; root:xnu-7195.141.6~3/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 1.3.4
numpy : 1.21.2
pytz : 2021.1
dateutil : 2.8.2
pip : 21.1.3
setuptools : 57.4.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.6.3
html5lib : None
pymysql : 1.0.2
psycopg2 : None
jinja2 : 3.0.1
IPython : 7.27.0
pandas_datareader: None
bs4 : 4.10.0
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.4.3
numexpr : None
odfpy : None
openpyxl : 3.0.8
pandas_gbq : None
pyarrow : 5.0.0
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : 0.8.9
xarray : None
xlrd : None
xlwt : None
numba : None

Metadata

Metadata

Assignees

No one assigned

    Labels

    BugIO JSONread_json, to_json, json_normalize

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions