Skip to content

Fix Json.cast_storage double-encoding large_string and string_view columns - #8682

Merged
lhoestq merged 3 commits into
huggingface:mainfrom
SulimanAbdulrazzaq:fix/json-cast-large-string
Oct 5, 2026
Merged

lhoestq merged 3 commits into
huggingface:mainfrom
SulimanAbdulrazzaq:fix/json-cast-large-string

Conversation

@SulimanAbdulrazzaq

Copy link
Copy Markdown
Contributor

Json.cast_storage only treats a pa.StringArray as already holding JSON text. large_string and string_view arrays are not subclasses of it, so they fall through to the generic branch and every value gets JSON-encoded a second time. The column then decodes back to plain strings instead of objects, and no error is raised.

A common way to hit this is pandas 3, which converts a str column to large_string:

import pandas as pd
from datasets import Dataset, Features, Json

df = pd.DataFrame({"meta": ['{"a": 1}', "[1, 2]"]})
ds = Dataset.from_pandas(df, features=Features({"meta": Json()}))
ds["meta"][:]
# main:        ['{"a": 1}', '[1, 2]']   (stored as '"{\"a\": 1}"', ...)
# this PR:     [{'a': 1}, [1, 2]]

cast_column("x", Json()) on a Value("large_string") or Value("string_view") column has the same problem.

The fix checks the Arrow type with pa.types.is_string / is_large_string / is_string_view in place of the isinstance check. Strings that are not valid JSON are still encoded as JSON strings, as before.

Tests: test_json_cast_storage_from_strings, parametrized over the three string types, and test_dataset_cast_large_string_column_to_json. Before the fix, the large_string and string_view cases fail. With the fix, tests/features passes (394 passed, 129 skipped) and so do the Json tests in tests/test_arrow_dataset.py (pandas 3.0.6, pyarrow 25.0.1).

ClassLabel.cast_storage has the same isinstance(storage, pa.StringArray) check. That one is tracked separately in #8681, so this PR leaves it alone.

â€Ķlumns

Json.cast_storage only recognized pa.StringArray as already holding JSON
text. large_string and string_view arrays fell through to the generic
branch, which JSON-encoded every value again, so a column of JSON strings
decoded back to plain strings instead of objects. This happens e.g. with
Dataset.from_pandas(df, features=...) on pandas 3, whose str columns are
converted to large_string.
@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@SulimanAbdulrazzaq

Copy link
Copy Markdown
Contributor Author

@lhoestq when you have a moment, could you approve the workflow runs and take a look? I merged current main into the branch so the checks run against it. The earlier red run failed at collecting tests/test_fingerprint_tokenizer_stability.py (huggingface-hub 2.0.0 was installed), which this change does not touch.

@lhoestq lhoestq left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm :)

@lhoestq
lhoestq merged commit db7d616 into huggingface:main Oct 5, 2026
7 of 14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants