BUG: fix read_json ignoring the dtype with the pyarrow engine #60997
Conversation
Co-authored-by: William Ayd <william.ayd@icloud.com>
|
Could any reviewers point me in the right direction with the failing test? Can't seem to figure out what is causing it |
|
I believe that issue appears in pyarrow < 16. In your test you can check that pyarrow version and filter the warning if that is detected |
Looks like that has worked, thanks! |
pandas/tests/io/json/test_pandas.py
Outdated
| @@ -2183,6 +2186,30 @@ def test_read_json_dtype_backend( | |||
| # string_storage setting -> ignore that for checking the result | |||
| tm.assert_frame_equal(result, expected, check_column_type=False) | |||
|
|
|||
| @td.skip_if_no("pyarrow") | |||
| @pytest.mark.filterwarnings("ignore::DeprecationWarning") | |||
There was a problem hiding this comment.
Can you add this filterwarnings mark only if pyarrow < 16?
There was a problem hiding this comment.
I couldn't find a way to do that with pytest filterwarnings
There was a problem hiding this comment.
def test_foo(..., request):
if pa_version_under16p0:
request.applymarker(pytest.mark.filterwarnings(...))
pandas/tests/io/json/test_pandas.py
Outdated
| version_tuple = tuple(map(int, pa.__version__.split("."))) | ||
|
|
||
| if version_tuple[0] < 16: |
There was a problem hiding this comment.
You can use pandas.util.version import Version to just do Version(pa.__version__) < Version("16.0")
There was a problem hiding this comment.
Looks like as of 5c581fc, this filter got removed?
There was a problem hiding this comment.
Yep sorry forgot to leave a comment. I couldn't get this method to work, So I have gone back to using the decorator. I have filtered it to just ignore the BlockManager deprecation warning, is this specific enough?
There was a problem hiding this comment.
What errors did you encounter?
There was a problem hiding this comment.
I didn't encounter any errors but it wasn't properly filtering the warning so the test kept failing
|
Thanks @will-larkin |
Added code to check if dtypes have been passed in, if they are, read the JSON with the schema built from those dtypes
Split read JSON function up for the respective engines
closes BUG: read_json silently ignores the dtype when engine=pyarrow #59516 (Replace xxxx with the GitHub issue number)
Tests added and passed if fixing a bug or adding a new feature
All code checks passed.
Added type annotations to new arguments/methods/functions.
Added an entry in the latest
doc/source/whatsnew/vX.X.X.rstfile if fixing a bug or adding a new feature.