ARROW-18152: [Python] DataFrame Interchange Protocol for pyarrow Table

[AlenkaF]

Produce a __dataframe__ object

• Implement the DataFrame, Column and Buffers class
• Test pa.Table -> pd.DataFrame

What should be looked into after the initial test:

• Data without missing values (produce a validity buffer in case of no missing values)
  Update: Columns without missing values are defined as non-nullable for now.
• Boolean values do not transfer correctly (only the first element is produced)
  Update: casting boolean column/array to uint8 solves this issue (boolean arrays are bit packed which is not supported by the protocol)
• Variable-length strings
  Update: Bit-width for the offset buffer dtype must be set to 32 instead of 64.
• DictionaryArray
  Update: Pandas implementation seems to expect the column of categories to be an instance of PandasColumn instead of general __dataframe__ column object.

File"/Users/alenkafrim/repos/pyarrow-dev-9/lib/python3.9/site-packages/pandas/core/interchange/from_dataframe.py", line185, incategorical_column_to_seriesassertisinstance(cat_column, PandasColumn), "categories must be a PandasColumn"AssertionError: categoriesmustbeaPandasColumn

• Bitmasks
  Update: Pandas implementation doesn't yet support bitmasks:

File"/Users/alenkafrim/repos/pyarrow-dev-9/lib/python3.9/site-packages/pandas/core/interchange/from_dataframe.py", line395, inbuffer_to_ndarrayraiseNotImplementedError(f"Conversion for {dtype} is not yet supported.")
NotImplementedError: Conversionfor (<DtypeKind.BOOL: 20>, 1, 'b', '=') isnotyetsupported.

---

This code in the PR tested with pandas implementation as a consumer currently works with integers, floats, booleans, strings and timestamps without missing values:

importpyarrowaspaimportpandasaspdfromdatetimeimportdatetimeasdttable=pa.table(
{
"a": [1, 2, 3, 4], # dtype kind INT = 0"b": [3, 4, 5, 6], # dtype kind INT = 0"c": [1.5, 2.5, 3.5, 4.5], # dtype kind FLOAT = 2"d": [9, 10, 11, 12], # dtype kind INT = 0"e": [True, True, False, False], # dtype kind BOOLEAN = 20"f": ["a", "", "c", "d"], # dtype kind STRING = 21"g": [dt(2007, 7, 13), dt(2007, 7, 14),
dt(2007, 7, 15), dt(2007, 7, 16)] # dtype kind DATETIME = 22
}
)
exchange_df=table.__dataframe__()
exchange_df._df# pyarrow.Table# a: int64# b: int64# c: double# d: int64# e: bool# f: string# g: timestamp[us]# ----# a: [[1,2,3,4]]# b: [[3,4,5,6]]# c: [[1.5,2.5,3.5,4.5]]# d: [[9,10,11,12]]# e: [[true,true,false,false]]# f: [["a","","c","d"]]# g: [[2007-07-13 00:00:00.000000,2007-07-14 00:00:00.000000,2007-07-15 00:00:00.000000,2007-07-16 00:00:00.000000]]frompandas.core.interchange.from_dataframeimportfrom_dataframefrom_dataframe(exchange_df)
abcdefg0131.591a2007-07-131242.51012007-07-142353.5110c2007-07-153464.5120d2007-07-16

---

Consume a __dataframe__ object

• Implement from_dataframe method

[github-actions]

https://issues.apache.org/jira/browse/ARROW-18152

[github-actions]

⚠️ Ticket has not been started in JIRA, please click 'Start Progress'.

[AlenkaF]

Example: selecting 5 chunks for an array of length 12. If chunk_size=2, we get 6 chunks, if chunk_size=3, we get 4 chunks =) So we end up producing 4 chunks with chunk_size=3 plus an empty chunk.

[honno]

FWIW we have a compliance suite for interchange protocol adopters over at data-apis/dataframe-interchange-tests. It's a bit awkward to use as you'd have to write a compatibility layer in wrappers.py, but might be interesting. (Cool to see you're working on this!)

[AlenkaF]

Thanks for the info @honno, the compliance suite is definitely something we have to use!
(Cool to see you continue working on the data API standards! 😉 )

[AlenkaF]

Will close this PR as I moved the work into another branch: #14804