Uh oh!
There was an error while loading. Please reload this page.
ARROW-15329: [Python] Add character limit to Table.to_string() - #12148
ARROW-15329: [Python] Add character limit to Table.to_string()#12148wjones127 wants to merge 4 commits into
Conversation
Thanks for opening a pull request! If this is not a minor PR. Could you open an issue for this pull request on JIRA? https://issues.apache.org/jira/browse/ARROW Opening JIRAs ahead of time contributes to the Openness of the Apache Arrow project. Then could you also rename pull request title in the following format? or See also: |
There was a problem hiding this comment.
Ideally, we could always truncate "correctly" and consistently, that is, truncate at a delimiter and show closing brackets: [[1,2,3,4,...]]
But several questions first:
- Is the delimiter always a comma or known in a variable?
- Is the table representation always encoded as a list of lists? Or can there be a less/more nesting?
Assuming "yes" for the questions above. There are several cases that can occur:
- No truncation:
[[1,2,3,4]] - Truncate does not reaches the final end bracket (# of brackets is arbitrary):
[[1,2,3,4]... - Truncate mid-value:
[10,20,30,4... - Truncate at a value delimiter:
[[1,2,3,... - Truncate at a list delimiter:
[[1,2,3,4],... - Truncate at a list ending bracket:
[[1,2,3,4]...
To display all truncated cases "correctly" and consistently, after slicing the cols_char_limit, you would need
- a stack to keep track of open/close bracket pairs
- check which symbol is truncation occurring on.
Then resolve as follows:
- If it is a delimiter, then add
...and match with closing brackets - If it is a closing bracket
- and there are only closing bracket left, then add them to obtain the full representation.
- add
...and match with closing brackets
- Else it is truncating a value, so remove (or add) characters until reaching a delimiter/bracket
Most of these checks can be identified with regex.
There was a problem hiding this comment.
@edponce I fully agree that, ideally, this truncation is "smart" about where to cut off and add ....
But in general it's also the question to what extent this is worth the extra complexity (depending on how complex it would be of course). Instead of parsing the string, another option could also be to slice the number of elements before converting to string (although for nested data types that won't necessarily work as desired).
Now, on the short-term (for 0.7.0), I personally find it more important that we at least do some truncation (because currently the repr can be completely useful / annoying by flooding your terminal)
There was a problem hiding this comment.
I understand and agree with the current approach. Simply leaving notes as food for thought or future reference.
There was a problem hiding this comment.
Thanks for the feedback. I implemented a very basic version of this for now. This looks pretty good for this example:
>>>fromrandomimportsample, choice>>>importpyarrowaspa>>>arr_int=pa.array(range(50))
>>>tree_parts= ["roots", "trunk", "crown", "seeds"]
>>>arr_list=pa.array([sample(tree_parts, k=choice(range(len(tree_parts)))) for_inrange(50)])
>>>arr_struct=pa.StructArray.from_arrays([arr_int, arr_list], names=['int_nested', 'list_nested'])
>>>arr_map=pa.array(
... [
... [(part, choice(range(10))) forpartinsample(tree_parts, k=choice(range(len(tree_parts))))]
... for_inrange(50)
... ],
... type=pa.map_(pa.utf8(), pa.int64())
... )
>>>table=pa.table({
... 'int': pa.chunked_array([arr_int] *10),
... 'list': pa.chunked_array([arr_list] *10),
... 'struct': pa.chunked_array([arr_struct] *10),
... 'map': pa.chunked_array([arr_map] *10),
... })
>>>print(table)
pyarrow.Tableint: int64list: list<item: string>child0, item: stringstruct: struct<int_nested: int64, list_nested: list<item: string>>child0, int_nested: int64child1, list_nested: list<item: string>child0, item: stringmap: map<string, int64>child0, entries: struct<key: stringnotnull, value: int64>notnullchild0, key: stringnotnullchild1, value: int64----int: [[0,1,2,3,4,5,6,7,8,9,...,40,41,42,43,44,45,46,47,48,49],[0,1,2,3,4,5,6,7,8,9,...,40,41,42,43,44,45,46,47,48,49],[0,1,2,3,...]...]
list: [[["seeds","trunk","roots"],["trunk","crown"],["crown"],["trunk"],["crown"],[],["roots","seeds"],["roots"],["trunk","roots"]...]...]
struct: [ --is_valid: allnotnull--child0 type: int64
[
0,
1,
2,
3,
4,
5,
6,...]...]
map: [[ keys:["seeds","crown","trunk"]values:[7,8,7], keys:["roots","crown"]values:[8,4], keys:["crown","roots","trunk"]...]...]The unfortunate thing is it will have bad behavior in the case of string columns containing [. For example,
>>>pa.table({'x': pa.array(["["*100]*500)})
x: [["[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[","[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[",...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]...]I think that kind of behavior is pretty unavoidable until we push this limit into the PrettyPrinter implementation itself.
There was a problem hiding this comment.
Is there a JIRA/PR for implementing similar functionality in C++ Pretty Printer?
There was a problem hiding this comment.
Just created one: https://issues.apache.org/jira/browse/ARROW-15363
There was a problem hiding this comment.
The default in the docstring does not matches the value in function arguments.
pitrou
commented
Mar 30, 2022
@jorisvandenbossche Do you want to take a look at this PR? |
34750d6 to
c4368faCompareamol-
commented
Mar 30, 2023
Closing because it has been untouched for a while, in case it's still relevant feel free to reopen and move it forward 👍 |
This prevents table column preview from being too long.
The main disadvantage is there is not way to guarantee that we don't truncate the preview in the middle of a value. For example,
[20, 30, 40]could be truncated to[20, 30, 4..., which might be a little misleadning.