Description
- I have checked that this issue has not already been reported.
- I have confirmed this bug exists on the latest version of pandas.
- I have confirmed this bug exists on the master branch of pandas.
Reproducible Example
import pandas as pd
import json
# Correct behavior with pd.read_csv
csv_data = """
date,bool,good_str,good_int,good_float,mixed_types,int_w_missing,str_w_missing
2021-01-01,0,a,1,1.0,1,1,a
2021-01-01,1,b,2,2.2,b,,
"""
with open("foo.csv", "w") as f:
f.write(csv_data)
# this correctly raises ValueError: could not convert string to float: 'b'
df_csv = pd.read_csv("foo.csv", dtype={"int_w_missing":"Int64", "mixed_types": "float"})
# ---------------
# Incorrect? behavior with pd.read_json
json_data = [
{
"date": "2021-01-01",
"good_int": 5,
"int_w_missing": 10,
"good_str": "114117967",
"mixed_types": "455590"
},
{
"date": "2021-01-01",
"good_int": 5,
"good_str": "114011434",
"mixed_types": "something"
}
]
with open("foo.json", "w") as f:
f.write(json.dumps(json_data))
# this does not raise anything
df_json = pd.read_json("foo.json", orient='records', dtype={"int_w_missing":"Int64", "mixed_types": "float"})
# checking dtypes we see "int_w_missing" nor "mixed_types" were cast
df_json.dtypes
Issue Description
Using dtype
with pd.read_json
:
- does not raise raise a ValueError when casting something like the
str
"something" to afloat
- does not correctly cast to
Int64
when casting aint
field with missing values
Expected Behavior
- raise a ValueError when casting something like the
str
"something" to afloat
- correctly cast to
Int64
when casting aint
field with missing values
Installed Versions
INSTALLED VERSIONS
commit : 945c9ed
python : 3.8.11.final.0
python-bits : 64
OS : Darwin
OS-release : 20.6.0
Version : Darwin Kernel Version 20.6.0: Mon Aug 30 06:12:21 PDT 2021; root:xnu-7195.141.6~3/RELEASE_X86_64
machine : x86_64
processor : i386
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8
pandas : 1.3.4
numpy : 1.21.2
pytz : 2021.1
dateutil : 2.8.2
pip : 21.1.3
setuptools : 57.4.0
Cython : None
pytest : None
hypothesis : None
sphinx : None
blosc : None
feather : None
xlsxwriter : None
lxml.etree : 4.6.3
html5lib : None
pymysql : 1.0.2
psycopg2 : None
jinja2 : 3.0.1
IPython : 7.27.0
pandas_datareader: None
bs4 : 4.10.0
bottleneck : None
fsspec : None
fastparquet : None
gcsfs : None
matplotlib : 3.4.3
numexpr : None
odfpy : None
openpyxl : 3.0.8
pandas_gbq : None
pyarrow : 5.0.0
pyxlsb : None
s3fs : None
scipy : None
sqlalchemy : None
tables : None
tabulate : 0.8.9
xarray : None
xlrd : None
xlwt : None
numba : None