Add variable-length string support by kgryte · Pull Request #45 · data-apis/dataframe-api

kgryte · 2021-06-24T23:39:26Z

This PR

adds variable-length string support to the dataframe interchange protocol.
modifies the protocol to return a dictionary containing all possible data buffers, rather than separate methods for retrieving data, validity, and offsets buffers, as discussed in consortium meetings.
modifies the protocol to return both the buffers and their associated dtypes. This is necessary in order to interpret, e.g., the offsets and mask buffers.
updates source code documentation (punctuation, typos, and spelling).
manually copies string data to and from byte arrays, as it assumes that strings are stored as object dtype. The implementation will need to be updated to accommodate pandas' string extension dtype which is based on arrow. Currently, the string extension dtype is considered experimental and subject to change. The use of object dtype is still used as the default string dtype for backward compatiblity.
uses a byte array mask for indicating missing values in string data buffers. This can be updated to use a bit array for space efficiency, at the cost of additional code complexity.
currently does not handle the scenario where a non-pandas dataframe provided to __dataframe__ uses a bit array to indicate missing values.

* Add a summary document for the dataframe interchange protocol Summarizes the various discussions about and goals/non-goals and requirements for the `__dataframe__` data interchange protocol. The intended audience for this document is Consortium members and dataframe library maintainers who may want to support this protocol. The aim is to keep updating this till we have captured all the requirements and answered all the FAQs, so we can actually design the protocol after and verify it meets all our requirements. Closesgh-29 * Process some review comments * Process a few more review comments. * Link to Release callback semantics in Arrow C Data Interface docs * Add design requirements for column selection and df metadata * Edit the nested/heterogeneous dtypes non-requirement * Add requirements for chunking and memory layout description Also address some smaller review comments. * Add TBD notes on dataframe-array connection and from_dataframe Also add more details on the Arrow C Data Interface. * Address review comments * Add details on implementation options * Add details about the C implementation * Add an image of the dataframe model and its memory layout. * Add link to discussion on array-dataframe connection * Some more updates for review comments * Update table to indicate Arrow does support categoricals. * Add section on dtype format strings * Reflow some lines * Add a requirement on semantic meaning of NaN/NaT, and timezone detail * Textual tweak: say columns in a data frame are ordered * Update requirements document for recent decisions/insights

Add a prototype of the dataframe interchange protocol

rgommers · 2021-06-28T16:16:11Z

Thanks @kgryte! It may be useful to close this PR and resend it against main, now that the other PRs it needed are merged.

rgommers

This looks very good, thanks @kgryte! I haven't tested it yet, but just reading through the code I have only a couple of comments.

…o variable-length-string-support

kgryte · 2021-06-28T17:54:27Z

@rgommers Will close this and submit against main in short order.

jorisvandenbossche · 2021-07-08T13:56:29Z

Will close this and submit against main in short order.

You can actually change the target branch by clicking the "Edit" button next to title, so then you don't need to close / open a new PR

jorisvandenbossche

I think the separate get_data_buffer and get_offsets methods might be a bit problematic. Typically you only want to do one pass over the data and create all buffers at once.

I know this is only a dummy implementation, and presumably the created buffers could be cached on the object so the separate methods don't calculate it twice. But might be useful to think about separate methods vs a single get_buffers() (with a specified order)

Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

kgryte · 2021-07-08T15:14:09Z

You can actually change the target branch by clicking the "Edit" button next to title, so then you don't need to close / open a new PR

I initially tried doing that, but doing so added many unrelated changes and muddied this PR. :(

jorisvandenbossche · 2021-07-08T17:28:51Z

I initially tried doing that, but doing so added many unrelated changes and muddied this PR. :(

I think that either merging latest master in this branch or rebasing this branch on top of master should solve that

This is a fresh port of changes made in order to support variable length strings in order to provide a cleaner merge.

kgryte · 2021-07-19T18:30:19Z

Closing this PR out in favor of gh-47.

kgryte · 2021-06-24T23:39:26Z

This PR

adds variable-length string support to the dataframe interchange protocol.
modifies the protocol to return a dictionary containing all possible data buffers, rather than separate methods for retrieving data, validity, and offsets buffers, as discussed in consortium meetings.
modifies the protocol to return both the buffers and their associated dtypes. This is necessary in order to interpret, e.g., the offsets and mask buffers.
updates source code documentation (punctuation, typos, and spelling).
manually copies string data to and from byte arrays, as it assumes that strings are stored as object dtype. The implementation will need to be updated to accommodate pandas' string extension dtype which is based on arrow. Currently, the string extension dtype is considered experimental and subject to change. The use of object dtype is still used as the default string dtype for backward compatiblity.
uses a byte array mask for indicating missing values in string data buffers. This can be updated to use a bit array for space efficiency, at the cost of additional code complexity.
currently does not handle the scenario where a non-pandas dataframe provided to __dataframe__ uses a bit array to indicate missing values.

* Add a summary document for the dataframe interchange protocol Summarizes the various discussions about and goals/non-goals and requirements for the `__dataframe__` data interchange protocol. The intended audience for this document is Consortium members and dataframe library maintainers who may want to support this protocol. The aim is to keep updating this till we have captured all the requirements and answered all the FAQs, so we can actually design the protocol after and verify it meets all our requirements. Closesgh-29 * Process some review comments * Process a few more review comments. * Link to Release callback semantics in Arrow C Data Interface docs * Add design requirements for column selection and df metadata * Edit the nested/heterogeneous dtypes non-requirement * Add requirements for chunking and memory layout description Also address some smaller review comments. * Add TBD notes on dataframe-array connection and from_dataframe Also add more details on the Arrow C Data Interface. * Address review comments * Add details on implementation options * Add details about the C implementation * Add an image of the dataframe model and its memory layout. * Add link to discussion on array-dataframe connection * Some more updates for review comments * Update table to indicate Arrow does support categoricals. * Add section on dtype format strings * Reflow some lines * Add a requirement on semantic meaning of NaN/NaT, and timezone detail * Textual tweak: say columns in a data frame are ordered * Update requirements document for recent decisions/insights

Add a prototype of the dataframe interchange protocol

rgommers · 2021-06-28T16:16:11Z

Thanks @kgryte! It may be useful to close this PR and resend it against main, now that the other PRs it needed are merged.

rgommers

This looks very good, thanks @kgryte! I haven't tested it yet, but just reading through the code I have only a couple of comments.

…o variable-length-string-support

kgryte · 2021-06-28T17:54:27Z

@rgommers Will close this and submit against main in short order.

jorisvandenbossche · 2021-07-08T13:56:29Z

Will close this and submit against main in short order.

You can actually change the target branch by clicking the "Edit" button next to title, so then you don't need to close / open a new PR

jorisvandenbossche

I think the separate get_data_buffer and get_offsets methods might be a bit problematic. Typically you only want to do one pass over the data and create all buffers at once.

I know this is only a dummy implementation, and presumably the created buffers could be cached on the object so the separate methods don't calculate it twice. But might be useful to think about separate methods vs a single get_buffers() (with a specified order)

Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

kgryte · 2021-07-08T15:14:09Z

You can actually change the target branch by clicking the "Edit" button next to title, so then you don't need to close / open a new PR

I initially tried doing that, but doing so added many unrelated changes and muddied this PR. :(

jorisvandenbossche · 2021-07-08T17:28:51Z

I initially tried doing that, but doing so added many unrelated changes and muddied this PR. :(

I think that either merging latest master in this branch or rebasing this branch on top of master should solve that

This is a fresh port of changes made in order to support variable length strings in order to provide a cleaner merge.

kgryte · 2021-07-19T18:30:19Z

Closing this PR out in favor of gh-47.

kgryte · 2021-06-24T23:39:26Z

This PR

adds variable-length string support to the dataframe interchange protocol.
modifies the protocol to return a dictionary containing all possible data buffers, rather than separate methods for retrieving data, validity, and offsets buffers, as discussed in consortium meetings.
modifies the protocol to return both the buffers and their associated dtypes. This is necessary in order to interpret, e.g., the offsets and mask buffers.
updates source code documentation (punctuation, typos, and spelling).
manually copies string data to and from byte arrays, as it assumes that strings are stored as object dtype. The implementation will need to be updated to accommodate pandas' string extension dtype which is based on arrow. Currently, the string extension dtype is considered experimental and subject to change. The use of object dtype is still used as the default string dtype for backward compatiblity.
uses a byte array mask for indicating missing values in string data buffers. This can be updated to use a bit array for space efficiency, at the cost of additional code complexity.
currently does not handle the scenario where a non-pandas dataframe provided to __dataframe__ uses a bit array to indicate missing values.

* Add a summary document for the dataframe interchange protocol Summarizes the various discussions about and goals/non-goals and requirements for the `__dataframe__` data interchange protocol. The intended audience for this document is Consortium members and dataframe library maintainers who may want to support this protocol. The aim is to keep updating this till we have captured all the requirements and answered all the FAQs, so we can actually design the protocol after and verify it meets all our requirements. Closesgh-29 * Process some review comments * Process a few more review comments. * Link to Release callback semantics in Arrow C Data Interface docs * Add design requirements for column selection and df metadata * Edit the nested/heterogeneous dtypes non-requirement * Add requirements for chunking and memory layout description Also address some smaller review comments. * Add TBD notes on dataframe-array connection and from_dataframe Also add more details on the Arrow C Data Interface. * Address review comments * Add details on implementation options * Add details about the C implementation * Add an image of the dataframe model and its memory layout. * Add link to discussion on array-dataframe connection * Some more updates for review comments * Update table to indicate Arrow does support categoricals. * Add section on dtype format strings * Reflow some lines * Add a requirement on semantic meaning of NaN/NaT, and timezone detail * Textual tweak: say columns in a data frame are ordered * Update requirements document for recent decisions/insights

Add a prototype of the dataframe interchange protocol

rgommers · 2021-06-28T16:16:11Z

Thanks @kgryte! It may be useful to close this PR and resend it against main, now that the other PRs it needed are merged.

rgommers

This looks very good, thanks @kgryte! I haven't tested it yet, but just reading through the code I have only a couple of comments.

…o variable-length-string-support

kgryte · 2021-06-28T17:54:27Z

@rgommers Will close this and submit against main in short order.

jorisvandenbossche · 2021-07-08T13:56:29Z

Will close this and submit against main in short order.

You can actually change the target branch by clicking the "Edit" button next to title, so then you don't need to close / open a new PR

jorisvandenbossche

I think the separate get_data_buffer and get_offsets methods might be a bit problematic. Typically you only want to do one pass over the data and create all buffers at once.

I know this is only a dummy implementation, and presumably the created buffers could be cached on the object so the separate methods don't calculate it twice. But might be useful to think about separate methods vs a single get_buffers() (with a specified order)

Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

kgryte · 2021-07-08T15:14:09Z

You can actually change the target branch by clicking the "Edit" button next to title, so then you don't need to close / open a new PR

I initially tried doing that, but doing so added many unrelated changes and muddied this PR. :(

jorisvandenbossche · 2021-07-08T17:28:51Z

I initially tried doing that, but doing so added many unrelated changes and muddied this PR. :(

I think that either merging latest master in this branch or rebasing this branch on top of master should solve that

This is a fresh port of changes made in order to support variable length strings in order to provide a cleaner merge.

kgryte · 2021-07-19T18:30:19Z

Closing this PR out in favor of gh-47.

kgryte · 2021-06-24T23:39:26Z

This PR

adds variable-length string support to the dataframe interchange protocol.
modifies the protocol to return a dictionary containing all possible data buffers, rather than separate methods for retrieving data, validity, and offsets buffers, as discussed in consortium meetings.
modifies the protocol to return both the buffers and their associated dtypes. This is necessary in order to interpret, e.g., the offsets and mask buffers.
updates source code documentation (punctuation, typos, and spelling).
manually copies string data to and from byte arrays, as it assumes that strings are stored as object dtype. The implementation will need to be updated to accommodate pandas' string extension dtype which is based on arrow. Currently, the string extension dtype is considered experimental and subject to change. The use of object dtype is still used as the default string dtype for backward compatiblity.
uses a byte array mask for indicating missing values in string data buffers. This can be updated to use a bit array for space efficiency, at the cost of additional code complexity.
currently does not handle the scenario where a non-pandas dataframe provided to __dataframe__ uses a bit array to indicate missing values.

* Add a summary document for the dataframe interchange protocol Summarizes the various discussions about and goals/non-goals and requirements for the `__dataframe__` data interchange protocol. The intended audience for this document is Consortium members and dataframe library maintainers who may want to support this protocol. The aim is to keep updating this till we have captured all the requirements and answered all the FAQs, so we can actually design the protocol after and verify it meets all our requirements. Closesgh-29 * Process some review comments * Process a few more review comments. * Link to Release callback semantics in Arrow C Data Interface docs * Add design requirements for column selection and df metadata * Edit the nested/heterogeneous dtypes non-requirement * Add requirements for chunking and memory layout description Also address some smaller review comments. * Add TBD notes on dataframe-array connection and from_dataframe Also add more details on the Arrow C Data Interface. * Address review comments * Add details on implementation options * Add details about the C implementation * Add an image of the dataframe model and its memory layout. * Add link to discussion on array-dataframe connection * Some more updates for review comments * Update table to indicate Arrow does support categoricals. * Add section on dtype format strings * Reflow some lines * Add a requirement on semantic meaning of NaN/NaT, and timezone detail * Textual tweak: say columns in a data frame are ordered * Update requirements document for recent decisions/insights

Add a prototype of the dataframe interchange protocol

rgommers · 2021-06-28T16:16:11Z

Thanks @kgryte! It may be useful to close this PR and resend it against main, now that the other PRs it needed are merged.

rgommers

This looks very good, thanks @kgryte! I haven't tested it yet, but just reading through the code I have only a couple of comments.

…o variable-length-string-support

kgryte · 2021-06-28T17:54:27Z

@rgommers Will close this and submit against main in short order.

jorisvandenbossche · 2021-07-08T13:56:29Z

Will close this and submit against main in short order.

You can actually change the target branch by clicking the "Edit" button next to title, so then you don't need to close / open a new PR

jorisvandenbossche

I think the separate get_data_buffer and get_offsets methods might be a bit problematic. Typically you only want to do one pass over the data and create all buffers at once.

I know this is only a dummy implementation, and presumably the created buffers could be cached on the object so the separate methods don't calculate it twice. But might be useful to think about separate methods vs a single get_buffers() (with a specified order)

Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

kgryte · 2021-07-08T15:14:09Z

You can actually change the target branch by clicking the "Edit" button next to title, so then you don't need to close / open a new PR

I initially tried doing that, but doing so added many unrelated changes and muddied this PR. :(

jorisvandenbossche · 2021-07-08T17:28:51Z

I initially tried doing that, but doing so added many unrelated changes and muddied this PR. :(

I think that either merging latest master in this branch or rebasing this branch on top of master should solve that

This is a fresh port of changes made in order to support variable length strings in order to provide a cleaner merge.

kgryte · 2021-07-19T18:30:19Z

Closing this PR out in favor of gh-47.

kgryte · 2021-06-24T23:39:26Z

This PR

adds variable-length string support to the dataframe interchange protocol.
modifies the protocol to return a dictionary containing all possible data buffers, rather than separate methods for retrieving data, validity, and offsets buffers, as discussed in consortium meetings.
modifies the protocol to return both the buffers and their associated dtypes. This is necessary in order to interpret, e.g., the offsets and mask buffers.
updates source code documentation (punctuation, typos, and spelling).
manually copies string data to and from byte arrays, as it assumes that strings are stored as object dtype. The implementation will need to be updated to accommodate pandas' string extension dtype which is based on arrow. Currently, the string extension dtype is considered experimental and subject to change. The use of object dtype is still used as the default string dtype for backward compatiblity.
uses a byte array mask for indicating missing values in string data buffers. This can be updated to use a bit array for space efficiency, at the cost of additional code complexity.
currently does not handle the scenario where a non-pandas dataframe provided to __dataframe__ uses a bit array to indicate missing values.

* Add a summary document for the dataframe interchange protocol Summarizes the various discussions about and goals/non-goals and requirements for the `__dataframe__` data interchange protocol. The intended audience for this document is Consortium members and dataframe library maintainers who may want to support this protocol. The aim is to keep updating this till we have captured all the requirements and answered all the FAQs, so we can actually design the protocol after and verify it meets all our requirements. Closesgh-29 * Process some review comments * Process a few more review comments. * Link to Release callback semantics in Arrow C Data Interface docs * Add design requirements for column selection and df metadata * Edit the nested/heterogeneous dtypes non-requirement * Add requirements for chunking and memory layout description Also address some smaller review comments. * Add TBD notes on dataframe-array connection and from_dataframe Also add more details on the Arrow C Data Interface. * Address review comments * Add details on implementation options * Add details about the C implementation * Add an image of the dataframe model and its memory layout. * Add link to discussion on array-dataframe connection * Some more updates for review comments * Update table to indicate Arrow does support categoricals. * Add section on dtype format strings * Reflow some lines * Add a requirement on semantic meaning of NaN/NaT, and timezone detail * Textual tweak: say columns in a data frame are ordered * Update requirements document for recent decisions/insights

Add a prototype of the dataframe interchange protocol

rgommers · 2021-06-28T16:16:11Z

Thanks @kgryte! It may be useful to close this PR and resend it against main, now that the other PRs it needed are merged.

rgommers

This looks very good, thanks @kgryte! I haven't tested it yet, but just reading through the code I have only a couple of comments.

…o variable-length-string-support

kgryte · 2021-06-28T17:54:27Z

@rgommers Will close this and submit against main in short order.

jorisvandenbossche · 2021-07-08T13:56:29Z

Will close this and submit against main in short order.

You can actually change the target branch by clicking the "Edit" button next to title, so then you don't need to close / open a new PR

jorisvandenbossche

I think the separate get_data_buffer and get_offsets methods might be a bit problematic. Typically you only want to do one pass over the data and create all buffers at once.

I know this is only a dummy implementation, and presumably the created buffers could be cached on the object so the separate methods don't calculate it twice. But might be useful to think about separate methods vs a single get_buffers() (with a specified order)

Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

kgryte · 2021-07-08T15:14:09Z

You can actually change the target branch by clicking the "Edit" button next to title, so then you don't need to close / open a new PR

I initially tried doing that, but doing so added many unrelated changes and muddied this PR. :(

jorisvandenbossche · 2021-07-08T17:28:51Z

I initially tried doing that, but doing so added many unrelated changes and muddied this PR. :(

I think that either merging latest master in this branch or rebasing this branch on top of master should solve that

This is a fresh port of changes made in order to support variable length strings in order to provide a cleaner merge.

kgryte · 2021-07-19T18:30:19Z

Closing this PR out in favor of gh-47.

kgryte · 2021-06-24T23:39:26Z

This PR

adds variable-length string support to the dataframe interchange protocol.
modifies the protocol to return a dictionary containing all possible data buffers, rather than separate methods for retrieving data, validity, and offsets buffers, as discussed in consortium meetings.
modifies the protocol to return both the buffers and their associated dtypes. This is necessary in order to interpret, e.g., the offsets and mask buffers.
updates source code documentation (punctuation, typos, and spelling).
manually copies string data to and from byte arrays, as it assumes that strings are stored as object dtype. The implementation will need to be updated to accommodate pandas' string extension dtype which is based on arrow. Currently, the string extension dtype is considered experimental and subject to change. The use of object dtype is still used as the default string dtype for backward compatiblity.
uses a byte array mask for indicating missing values in string data buffers. This can be updated to use a bit array for space efficiency, at the cost of additional code complexity.
currently does not handle the scenario where a non-pandas dataframe provided to __dataframe__ uses a bit array to indicate missing values.

* Add a summary document for the dataframe interchange protocol Summarizes the various discussions about and goals/non-goals and requirements for the `__dataframe__` data interchange protocol. The intended audience for this document is Consortium members and dataframe library maintainers who may want to support this protocol. The aim is to keep updating this till we have captured all the requirements and answered all the FAQs, so we can actually design the protocol after and verify it meets all our requirements. Closesgh-29 * Process some review comments * Process a few more review comments. * Link to Release callback semantics in Arrow C Data Interface docs * Add design requirements for column selection and df metadata * Edit the nested/heterogeneous dtypes non-requirement * Add requirements for chunking and memory layout description Also address some smaller review comments. * Add TBD notes on dataframe-array connection and from_dataframe Also add more details on the Arrow C Data Interface. * Address review comments * Add details on implementation options * Add details about the C implementation * Add an image of the dataframe model and its memory layout. * Add link to discussion on array-dataframe connection * Some more updates for review comments * Update table to indicate Arrow does support categoricals. * Add section on dtype format strings * Reflow some lines * Add a requirement on semantic meaning of NaN/NaT, and timezone detail * Textual tweak: say columns in a data frame are ordered * Update requirements document for recent decisions/insights

Add a prototype of the dataframe interchange protocol

rgommers · 2021-06-28T16:16:11Z

Thanks @kgryte! It may be useful to close this PR and resend it against main, now that the other PRs it needed are merged.

rgommers

This looks very good, thanks @kgryte! I haven't tested it yet, but just reading through the code I have only a couple of comments.

…o variable-length-string-support

kgryte · 2021-06-28T17:54:27Z

@rgommers Will close this and submit against main in short order.

jorisvandenbossche · 2021-07-08T13:56:29Z

Will close this and submit against main in short order.

You can actually change the target branch by clicking the "Edit" button next to title, so then you don't need to close / open a new PR

jorisvandenbossche

I think the separate get_data_buffer and get_offsets methods might be a bit problematic. Typically you only want to do one pass over the data and create all buffers at once.

I know this is only a dummy implementation, and presumably the created buffers could be cached on the object so the separate methods don't calculate it twice. But might be useful to think about separate methods vs a single get_buffers() (with a specified order)

Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

kgryte · 2021-07-08T15:14:09Z

You can actually change the target branch by clicking the "Edit" button next to title, so then you don't need to close / open a new PR

I initially tried doing that, but doing so added many unrelated changes and muddied this PR. :(

jorisvandenbossche · 2021-07-08T17:28:51Z

I initially tried doing that, but doing so added many unrelated changes and muddied this PR. :(

I think that either merging latest master in this branch or rebasing this branch on top of master should solve that

This is a fresh port of changes made in order to support variable length strings in order to provide a cleaner merge.

kgryte · 2021-07-19T18:30:19Z

Closing this PR out in favor of gh-47.

kgryte · 2021-06-24T23:39:26Z

This PR

adds variable-length string support to the dataframe interchange protocol.
modifies the protocol to return a dictionary containing all possible data buffers, rather than separate methods for retrieving data, validity, and offsets buffers, as discussed in consortium meetings.
modifies the protocol to return both the buffers and their associated dtypes. This is necessary in order to interpret, e.g., the offsets and mask buffers.
updates source code documentation (punctuation, typos, and spelling).
manually copies string data to and from byte arrays, as it assumes that strings are stored as object dtype. The implementation will need to be updated to accommodate pandas' string extension dtype which is based on arrow. Currently, the string extension dtype is considered experimental and subject to change. The use of object dtype is still used as the default string dtype for backward compatiblity.
uses a byte array mask for indicating missing values in string data buffers. This can be updated to use a bit array for space efficiency, at the cost of additional code complexity.
currently does not handle the scenario where a non-pandas dataframe provided to __dataframe__ uses a bit array to indicate missing values.

* Add a summary document for the dataframe interchange protocol Summarizes the various discussions about and goals/non-goals and requirements for the `__dataframe__` data interchange protocol. The intended audience for this document is Consortium members and dataframe library maintainers who may want to support this protocol. The aim is to keep updating this till we have captured all the requirements and answered all the FAQs, so we can actually design the protocol after and verify it meets all our requirements. Closesgh-29 * Process some review comments * Process a few more review comments. * Link to Release callback semantics in Arrow C Data Interface docs * Add design requirements for column selection and df metadata * Edit the nested/heterogeneous dtypes non-requirement * Add requirements for chunking and memory layout description Also address some smaller review comments. * Add TBD notes on dataframe-array connection and from_dataframe Also add more details on the Arrow C Data Interface. * Address review comments * Add details on implementation options * Add details about the C implementation * Add an image of the dataframe model and its memory layout. * Add link to discussion on array-dataframe connection * Some more updates for review comments * Update table to indicate Arrow does support categoricals. * Add section on dtype format strings * Reflow some lines * Add a requirement on semantic meaning of NaN/NaT, and timezone detail * Textual tweak: say columns in a data frame are ordered * Update requirements document for recent decisions/insights

Add a prototype of the dataframe interchange protocol

rgommers · 2021-06-28T16:16:11Z

Thanks @kgryte! It may be useful to close this PR and resend it against main, now that the other PRs it needed are merged.

rgommers

This looks very good, thanks @kgryte! I haven't tested it yet, but just reading through the code I have only a couple of comments.

…o variable-length-string-support

kgryte · 2021-06-28T17:54:27Z

@rgommers Will close this and submit against main in short order.

jorisvandenbossche · 2021-07-08T13:56:29Z

Will close this and submit against main in short order.

You can actually change the target branch by clicking the "Edit" button next to title, so then you don't need to close / open a new PR

jorisvandenbossche

I think the separate get_data_buffer and get_offsets methods might be a bit problematic. Typically you only want to do one pass over the data and create all buffers at once.

I know this is only a dummy implementation, and presumably the created buffers could be cached on the object so the separate methods don't calculate it twice. But might be useful to think about separate methods vs a single get_buffers() (with a specified order)

Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

kgryte · 2021-07-08T15:14:09Z

You can actually change the target branch by clicking the "Edit" button next to title, so then you don't need to close / open a new PR

I initially tried doing that, but doing so added many unrelated changes and muddied this PR. :(

jorisvandenbossche · 2021-07-08T17:28:51Z

I initially tried doing that, but doing so added many unrelated changes and muddied this PR. :(

I think that either merging latest master in this branch or rebasing this branch on top of master should solve that

This is a fresh port of changes made in order to support variable length strings in order to provide a cleaner merge.

kgryte · 2021-07-19T18:30:19Z

Closing this PR out in favor of gh-47.

kgryte · 2021-06-24T23:39:26Z

This PR

adds variable-length string support to the dataframe interchange protocol.
modifies the protocol to return a dictionary containing all possible data buffers, rather than separate methods for retrieving data, validity, and offsets buffers, as discussed in consortium meetings.
modifies the protocol to return both the buffers and their associated dtypes. This is necessary in order to interpret, e.g., the offsets and mask buffers.
updates source code documentation (punctuation, typos, and spelling).
manually copies string data to and from byte arrays, as it assumes that strings are stored as object dtype. The implementation will need to be updated to accommodate pandas' string extension dtype which is based on arrow. Currently, the string extension dtype is considered experimental and subject to change. The use of object dtype is still used as the default string dtype for backward compatiblity.
uses a byte array mask for indicating missing values in string data buffers. This can be updated to use a bit array for space efficiency, at the cost of additional code complexity.
currently does not handle the scenario where a non-pandas dataframe provided to __dataframe__ uses a bit array to indicate missing values.

* Add a summary document for the dataframe interchange protocol Summarizes the various discussions about and goals/non-goals and requirements for the `__dataframe__` data interchange protocol. The intended audience for this document is Consortium members and dataframe library maintainers who may want to support this protocol. The aim is to keep updating this till we have captured all the requirements and answered all the FAQs, so we can actually design the protocol after and verify it meets all our requirements. Closesgh-29 * Process some review comments * Process a few more review comments. * Link to Release callback semantics in Arrow C Data Interface docs * Add design requirements for column selection and df metadata * Edit the nested/heterogeneous dtypes non-requirement * Add requirements for chunking and memory layout description Also address some smaller review comments. * Add TBD notes on dataframe-array connection and from_dataframe Also add more details on the Arrow C Data Interface. * Address review comments * Add details on implementation options * Add details about the C implementation * Add an image of the dataframe model and its memory layout. * Add link to discussion on array-dataframe connection * Some more updates for review comments * Update table to indicate Arrow does support categoricals. * Add section on dtype format strings * Reflow some lines * Add a requirement on semantic meaning of NaN/NaT, and timezone detail * Textual tweak: say columns in a data frame are ordered * Update requirements document for recent decisions/insights

Add a prototype of the dataframe interchange protocol

rgommers · 2021-06-28T16:16:11Z

Thanks @kgryte! It may be useful to close this PR and resend it against main, now that the other PRs it needed are merged.

rgommers

This looks very good, thanks @kgryte! I haven't tested it yet, but just reading through the code I have only a couple of comments.

…o variable-length-string-support

kgryte · 2021-06-28T17:54:27Z

@rgommers Will close this and submit against main in short order.

jorisvandenbossche · 2021-07-08T13:56:29Z

Will close this and submit against main in short order.

You can actually change the target branch by clicking the "Edit" button next to title, so then you don't need to close / open a new PR

jorisvandenbossche

I think the separate get_data_buffer and get_offsets methods might be a bit problematic. Typically you only want to do one pass over the data and create all buffers at once.

I know this is only a dummy implementation, and presumably the created buffers could be cached on the object so the separate methods don't calculate it twice. But might be useful to think about separate methods vs a single get_buffers() (with a specified order)

Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

kgryte · 2021-07-08T15:14:09Z

You can actually change the target branch by clicking the "Edit" button next to title, so then you don't need to close / open a new PR

I initially tried doing that, but doing so added many unrelated changes and muddied this PR. :(

jorisvandenbossche · 2021-07-08T17:28:51Z

I initially tried doing that, but doing so added many unrelated changes and muddied this PR. :(

I think that either merging latest master in this branch or rebasing this branch on top of master should solve that

This is a fresh port of changes made in order to support variable length strings in order to provide a cleaner merge.

kgryte · 2021-07-19T18:30:19Z

Closing this PR out in favor of gh-47.

kgryte added 12 commits June 23, 2021 13:32

Update protocol to support returning an offsets buffer
5a8d030

Add punctuation
3aac477

Update protocol to return dtype along with buffer
c7728e2

Add string support in various methods and add todos
c8000f7

Add support for resolving an offsets buffer
040f928

Add support for returning a data buffer for string dtypes
a982987

Update offsets buffer accessor
fd4d71b

Add implementation to convert a string column
e40f902

Add tests
c122b3c

Handle missing values
0d04af3

Update typing and docs
58fee89

Add comment
2c4a846

kgryte changed the title ~~WIP: Add variable-length string support~~Add variable-length string supportJun 25, 2021

kgryte requested a review from rgommers June 25, 2021 00:09

rgommers added 3 commits June 25, 2021 21:45

Remove outdated figures
f9f259c

Document that strided buffers do not need to be supported
a545faa

rgommers mentioned this pull request Jun 25, 2021
Tracking issue: dataframe protocol implementation #46
Open
8 tasks

Merge pull request#38from data-apis/protocol-impl
52abf7a
Add a prototype of the dataframe interchange protocol

rgommers added enhancement New feature or request interchange-protocol labels Jun 25, 2021

rgommers reviewed Jun 28, 2021
View reviewed changes

Comment threadprotocol/pandas_implementation.py Outdated
Comment threadprotocol/pandas_implementation.py
Comment threadprotocol/pandas_implementation.py Outdated

kgryte added 2 commits June 28, 2021 10:48

Add todo
6010ae7

Merge branch 'main' ofhttps://github.com/data-apis/dataframe-apiint…
ac1a5ca
…o variable-length-string-support

kgryte changed the base branch from df-interchange-protocol-staging to mainJune 28, 2021 17:51

kgryte changed the base branch from main to df-interchange-protocol-stagingJune 28, 2021 17:51

Remove colons
89a7996

jorisvandenbossche reviewed Jul 8, 2021
View reviewed changes

Comment threadprotocol/dataframe_protocol_summary.md Outdated

jorisvandenbossche reviewed Jul 8, 2021
View reviewed changes

Comment threadprotocol/pandas_implementation.py
Comment threadprotocol/pandas_implementation.py

Fix grammar
a3ff4e7
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

kgryte added 3 commits July 8, 2021 09:20

Rename methods
ff84e8c

Rename methods
c954f3c

Update describe_null to indicate a byte array for string dtype
ed64fb7

kgryte added 9 commits July 19, 2021 09:20

Return encoding for missing values
9b9aecf

Update test
4026900

Use invalid value encoding
87d7143

Update copy
56ee2da

Use Arrow format strings
0035c90

Add get_buffers method to the protocol
91ed6a1

Remove individual methods
26fb48d

Update copy
0d0e94b

Refactor to return a dictionary of buffers
9ec830c

kgryte added a commit that referenced this pull request Jul 19, 2021
Port changes from#45
31cbf86
This is a fresh port of changes made in order to support variable length strings in order to provide a cleaner merge.

kgryte mentioned this pull request Jul 19, 2021
Add variable-length string support #47
Merged

kgryte added 2 commits July 19, 2021 11:28

Update comments
0dd4e2c

Fix copy
ade0d76

kgryte closed this Jul 19, 2021

kgryte added 12 commits June 23, 2021 13:32

Update protocol to support returning an offsets buffer
5a8d030

Add punctuation
3aac477

Update protocol to return dtype along with buffer
c7728e2

Add string support in various methods and add todos
c8000f7

Add support for resolving an offsets buffer
040f928

Add support for returning a data buffer for string dtypes
a982987

Update offsets buffer accessor
fd4d71b

Add implementation to convert a string column
e40f902

Add tests
c122b3c

Handle missing values
0d04af3

Update typing and docs
58fee89

Add comment
2c4a846

kgryte changed the title ~~WIP: Add variable-length string support~~Add variable-length string supportJun 25, 2021

kgryte requested a review from rgommers June 25, 2021 00:09

rgommers added 3 commits June 25, 2021 21:45

Remove outdated figures
f9f259c

Document that strided buffers do not need to be supported
a545faa

rgommers mentioned this pull request Jun 25, 2021
Tracking issue: dataframe protocol implementation #46
Open
8 tasks

Merge pull request#38from data-apis/protocol-impl
52abf7a
Add a prototype of the dataframe interchange protocol

rgommers added enhancement New feature or request interchange-protocol labels Jun 25, 2021

rgommers reviewed Jun 28, 2021
View reviewed changes

Comment threadprotocol/pandas_implementation.py Outdated
Comment threadprotocol/pandas_implementation.py
Comment threadprotocol/pandas_implementation.py Outdated

kgryte added 2 commits June 28, 2021 10:48

Add todo
6010ae7

Merge branch 'main' ofhttps://github.com/data-apis/dataframe-apiint…
ac1a5ca
…o variable-length-string-support

kgryte changed the base branch from df-interchange-protocol-staging to mainJune 28, 2021 17:51

kgryte changed the base branch from main to df-interchange-protocol-stagingJune 28, 2021 17:51

Remove colons
89a7996

jorisvandenbossche reviewed Jul 8, 2021
View reviewed changes

Comment threadprotocol/dataframe_protocol_summary.md Outdated

jorisvandenbossche reviewed Jul 8, 2021
View reviewed changes

Comment threadprotocol/pandas_implementation.py
Comment threadprotocol/pandas_implementation.py

Fix grammar
a3ff4e7
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

kgryte added 3 commits July 8, 2021 09:20

Rename methods
ff84e8c

Rename methods
c954f3c

Update describe_null to indicate a byte array for string dtype
ed64fb7

kgryte added 9 commits July 19, 2021 09:20

Return encoding for missing values
9b9aecf

Update test
4026900

Use invalid value encoding
87d7143

Update copy
56ee2da

Use Arrow format strings
0035c90

Add get_buffers method to the protocol
91ed6a1

Remove individual methods
26fb48d

Update copy
0d0e94b

Refactor to return a dictionary of buffers
9ec830c

kgryte added a commit that referenced this pull request Jul 19, 2021
Port changes from#45
31cbf86
This is a fresh port of changes made in order to support variable length strings in order to provide a cleaner merge.

kgryte mentioned this pull request Jul 19, 2021
Add variable-length string support #47
Merged

kgryte added 2 commits July 19, 2021 11:28

Update comments
0dd4e2c

Fix copy
ade0d76

kgryte closed this Jul 19, 2021

kgryte added 12 commits June 23, 2021 13:32

Update protocol to support returning an offsets buffer
5a8d030

Add punctuation
3aac477

Update protocol to return dtype along with buffer
c7728e2

Add string support in various methods and add todos
c8000f7

Add support for resolving an offsets buffer
040f928

Add support for returning a data buffer for string dtypes
a982987

Update offsets buffer accessor
fd4d71b

Add implementation to convert a string column
e40f902

Add tests
c122b3c

Handle missing values
0d04af3

Update typing and docs
58fee89

Add comment
2c4a846

kgryte changed the title ~~WIP: Add variable-length string support~~Add variable-length string supportJun 25, 2021

kgryte requested a review from rgommers June 25, 2021 00:09

rgommers added 3 commits June 25, 2021 21:45

Remove outdated figures
f9f259c

Document that strided buffers do not need to be supported
a545faa

rgommers mentioned this pull request Jun 25, 2021
Tracking issue: dataframe protocol implementation #46
Open
8 tasks

Merge pull request#38from data-apis/protocol-impl
52abf7a
Add a prototype of the dataframe interchange protocol

rgommers added enhancement New feature or request interchange-protocol labels Jun 25, 2021

rgommers reviewed Jun 28, 2021
View reviewed changes

Comment threadprotocol/pandas_implementation.py Outdated
Comment threadprotocol/pandas_implementation.py
Comment threadprotocol/pandas_implementation.py Outdated

kgryte added 2 commits June 28, 2021 10:48

Add todo
6010ae7

Merge branch 'main' ofhttps://github.com/data-apis/dataframe-apiint…
ac1a5ca
…o variable-length-string-support

kgryte changed the base branch from df-interchange-protocol-staging to mainJune 28, 2021 17:51

kgryte changed the base branch from main to df-interchange-protocol-stagingJune 28, 2021 17:51

Remove colons
89a7996

jorisvandenbossche reviewed Jul 8, 2021
View reviewed changes

Comment threadprotocol/dataframe_protocol_summary.md Outdated

jorisvandenbossche reviewed Jul 8, 2021
View reviewed changes

Comment threadprotocol/pandas_implementation.py
Comment threadprotocol/pandas_implementation.py

Fix grammar
a3ff4e7
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

kgryte added 3 commits July 8, 2021 09:20

Rename methods
ff84e8c

Rename methods
c954f3c

Update describe_null to indicate a byte array for string dtype
ed64fb7

kgryte added 9 commits July 19, 2021 09:20

Return encoding for missing values
9b9aecf

Update test
4026900

Use invalid value encoding
87d7143

Update copy
56ee2da

Use Arrow format strings
0035c90

Add get_buffers method to the protocol
91ed6a1

Remove individual methods
26fb48d

Update copy
0d0e94b

Refactor to return a dictionary of buffers
9ec830c

kgryte added a commit that referenced this pull request Jul 19, 2021
Port changes from#45
31cbf86
This is a fresh port of changes made in order to support variable length strings in order to provide a cleaner merge.

kgryte mentioned this pull request Jul 19, 2021
Add variable-length string support #47
Merged

kgryte added 2 commits July 19, 2021 11:28

Update comments
0dd4e2c

Fix copy
ade0d76

kgryte closed this Jul 19, 2021

kgryte added 12 commits June 23, 2021 13:32

Update protocol to support returning an offsets buffer
5a8d030

Add punctuation
3aac477

Update protocol to return dtype along with buffer
c7728e2

Add string support in various methods and add todos
c8000f7

Add support for resolving an offsets buffer
040f928

Add support for returning a data buffer for string dtypes
a982987

Update offsets buffer accessor
fd4d71b

Add implementation to convert a string column
e40f902

Add tests
c122b3c

Handle missing values
0d04af3

Update typing and docs
58fee89

Add comment
2c4a846

kgryte changed the title ~~WIP: Add variable-length string support~~Add variable-length string supportJun 25, 2021

kgryte requested a review from rgommers June 25, 2021 00:09

rgommers added 3 commits June 25, 2021 21:45

Remove outdated figures
f9f259c

Document that strided buffers do not need to be supported
a545faa

rgommers mentioned this pull request Jun 25, 2021
Tracking issue: dataframe protocol implementation #46
Open
8 tasks

Merge pull request#38from data-apis/protocol-impl
52abf7a
Add a prototype of the dataframe interchange protocol

rgommers added enhancement New feature or request interchange-protocol labels Jun 25, 2021

rgommers reviewed Jun 28, 2021
View reviewed changes

Comment threadprotocol/pandas_implementation.py Outdated
Comment threadprotocol/pandas_implementation.py
Comment threadprotocol/pandas_implementation.py Outdated

kgryte added 2 commits June 28, 2021 10:48

Add todo
6010ae7

Merge branch 'main' ofhttps://github.com/data-apis/dataframe-apiint…
ac1a5ca
…o variable-length-string-support

kgryte changed the base branch from df-interchange-protocol-staging to mainJune 28, 2021 17:51

kgryte changed the base branch from main to df-interchange-protocol-stagingJune 28, 2021 17:51

Remove colons
89a7996

jorisvandenbossche reviewed Jul 8, 2021
View reviewed changes

Comment threadprotocol/dataframe_protocol_summary.md Outdated

jorisvandenbossche reviewed Jul 8, 2021
View reviewed changes

Comment threadprotocol/pandas_implementation.py
Comment threadprotocol/pandas_implementation.py

Fix grammar
a3ff4e7
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

kgryte added 3 commits July 8, 2021 09:20

Rename methods
ff84e8c

Rename methods
c954f3c

Update describe_null to indicate a byte array for string dtype
ed64fb7

kgryte added 9 commits July 19, 2021 09:20

Return encoding for missing values
9b9aecf

Update test
4026900

Use invalid value encoding
87d7143

Update copy
56ee2da

Use Arrow format strings
0035c90

Add get_buffers method to the protocol
91ed6a1

Remove individual methods
26fb48d

Update copy
0d0e94b

Refactor to return a dictionary of buffers
9ec830c

kgryte added a commit that referenced this pull request Jul 19, 2021
Port changes from#45
31cbf86
This is a fresh port of changes made in order to support variable length strings in order to provide a cleaner merge.

kgryte mentioned this pull request Jul 19, 2021
Add variable-length string support #47
Merged

kgryte added 2 commits July 19, 2021 11:28

Update comments
0dd4e2c

Fix copy
ade0d76

kgryte closed this Jul 19, 2021

kgryte added 12 commits June 23, 2021 13:32

Update protocol to support returning an offsets buffer
5a8d030

Add punctuation
3aac477

Update protocol to return dtype along with buffer
c7728e2

Add string support in various methods and add todos
c8000f7

Add support for resolving an offsets buffer
040f928

Add support for returning a data buffer for string dtypes
a982987

Update offsets buffer accessor
fd4d71b

Add implementation to convert a string column
e40f902

Add tests
c122b3c

Handle missing values
0d04af3

Update typing and docs
58fee89

Add comment
2c4a846

kgryte changed the title ~~WIP: Add variable-length string support~~Add variable-length string supportJun 25, 2021

kgryte requested a review from rgommers June 25, 2021 00:09

rgommers added 3 commits June 25, 2021 21:45

Remove outdated figures
f9f259c

Document that strided buffers do not need to be supported
a545faa

rgommers mentioned this pull request Jun 25, 2021
Tracking issue: dataframe protocol implementation #46
Open
8 tasks

Merge pull request#38from data-apis/protocol-impl
52abf7a
Add a prototype of the dataframe interchange protocol

rgommers added enhancement New feature or request interchange-protocol labels Jun 25, 2021

rgommers reviewed Jun 28, 2021
View reviewed changes

Comment threadprotocol/pandas_implementation.py Outdated
Comment threadprotocol/pandas_implementation.py
Comment threadprotocol/pandas_implementation.py Outdated

kgryte added 2 commits June 28, 2021 10:48

Add todo
6010ae7

Merge branch 'main' ofhttps://github.com/data-apis/dataframe-apiint…
ac1a5ca
…o variable-length-string-support

kgryte changed the base branch from df-interchange-protocol-staging to mainJune 28, 2021 17:51

kgryte changed the base branch from main to df-interchange-protocol-stagingJune 28, 2021 17:51

Remove colons
89a7996

jorisvandenbossche reviewed Jul 8, 2021
View reviewed changes

Comment threadprotocol/dataframe_protocol_summary.md Outdated

jorisvandenbossche reviewed Jul 8, 2021
View reviewed changes

Comment threadprotocol/pandas_implementation.py
Comment threadprotocol/pandas_implementation.py

Fix grammar
a3ff4e7
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

kgryte added 3 commits July 8, 2021 09:20

Rename methods
ff84e8c

Rename methods
c954f3c

Update describe_null to indicate a byte array for string dtype
ed64fb7

kgryte added 9 commits July 19, 2021 09:20

Return encoding for missing values
9b9aecf

Update test
4026900

Use invalid value encoding
87d7143

Update copy
56ee2da

Use Arrow format strings
0035c90

Add get_buffers method to the protocol
91ed6a1

Remove individual methods
26fb48d

Update copy
0d0e94b

Refactor to return a dictionary of buffers
9ec830c

kgryte added a commit that referenced this pull request Jul 19, 2021
Port changes from#45
31cbf86
This is a fresh port of changes made in order to support variable length strings in order to provide a cleaner merge.

kgryte mentioned this pull request Jul 19, 2021
Add variable-length string support #47
Merged

kgryte added 2 commits July 19, 2021 11:28

Update comments
0dd4e2c

Fix copy
ade0d76

kgryte closed this Jul 19, 2021

kgryte added 12 commits June 23, 2021 13:32

Update protocol to support returning an offsets buffer
5a8d030

Add punctuation
3aac477

Update protocol to return dtype along with buffer
c7728e2

Add string support in various methods and add todos
c8000f7

Add support for resolving an offsets buffer
040f928

Add support for returning a data buffer for string dtypes
a982987

Update offsets buffer accessor
fd4d71b

Add implementation to convert a string column
e40f902

Add tests
c122b3c

Handle missing values
0d04af3

Update typing and docs
58fee89

Add comment
2c4a846

kgryte changed the title ~~WIP: Add variable-length string support~~Add variable-length string supportJun 25, 2021

kgryte requested a review from rgommers June 25, 2021 00:09

rgommers added 3 commits June 25, 2021 21:45

Remove outdated figures
f9f259c

Document that strided buffers do not need to be supported
a545faa

rgommers mentioned this pull request Jun 25, 2021
Tracking issue: dataframe protocol implementation #46
Open
8 tasks

Merge pull request#38from data-apis/protocol-impl
52abf7a
Add a prototype of the dataframe interchange protocol

rgommers added enhancement New feature or request interchange-protocol labels Jun 25, 2021

rgommers reviewed Jun 28, 2021
View reviewed changes

Comment threadprotocol/pandas_implementation.py Outdated
Comment threadprotocol/pandas_implementation.py
Comment threadprotocol/pandas_implementation.py Outdated

kgryte added 2 commits June 28, 2021 10:48

Add todo
6010ae7

Merge branch 'main' ofhttps://github.com/data-apis/dataframe-apiint…
ac1a5ca
…o variable-length-string-support

kgryte changed the base branch from df-interchange-protocol-staging to mainJune 28, 2021 17:51

kgryte changed the base branch from main to df-interchange-protocol-stagingJune 28, 2021 17:51

Remove colons
89a7996

jorisvandenbossche reviewed Jul 8, 2021
View reviewed changes

Comment threadprotocol/dataframe_protocol_summary.md Outdated

jorisvandenbossche reviewed Jul 8, 2021
View reviewed changes

Comment threadprotocol/pandas_implementation.py
Comment threadprotocol/pandas_implementation.py

Fix grammar
a3ff4e7
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

kgryte added 3 commits July 8, 2021 09:20

Rename methods
ff84e8c

Rename methods
c954f3c

Update describe_null to indicate a byte array for string dtype
ed64fb7

kgryte added 9 commits July 19, 2021 09:20

Return encoding for missing values
9b9aecf

Update test
4026900

Use invalid value encoding
87d7143

Update copy
56ee2da

Use Arrow format strings
0035c90

Add get_buffers method to the protocol
91ed6a1

Remove individual methods
26fb48d

Update copy
0d0e94b

Refactor to return a dictionary of buffers
9ec830c

kgryte added a commit that referenced this pull request Jul 19, 2021
Port changes from#45
31cbf86
This is a fresh port of changes made in order to support variable length strings in order to provide a cleaner merge.

kgryte mentioned this pull request Jul 19, 2021
Add variable-length string support #47
Merged

kgryte added 2 commits July 19, 2021 11:28

Update comments
0dd4e2c

Fix copy
ade0d76

kgryte closed this Jul 19, 2021

kgryte added 12 commits June 23, 2021 13:32

Update protocol to support returning an offsets buffer
5a8d030

Add punctuation
3aac477

Update protocol to return dtype along with buffer
c7728e2

Add string support in various methods and add todos
c8000f7

Add support for resolving an offsets buffer
040f928

Add support for returning a data buffer for string dtypes
a982987

Update offsets buffer accessor
fd4d71b

Add implementation to convert a string column
e40f902

Add tests
c122b3c

Handle missing values
0d04af3

Update typing and docs
58fee89

Add comment
2c4a846

kgryte changed the title ~~WIP: Add variable-length string support~~Add variable-length string supportJun 25, 2021

kgryte requested a review from rgommers June 25, 2021 00:09

rgommers added 3 commits June 25, 2021 21:45

Remove outdated figures
f9f259c

Document that strided buffers do not need to be supported
a545faa

rgommers mentioned this pull request Jun 25, 2021
Tracking issue: dataframe protocol implementation #46
Open
8 tasks

Merge pull request#38from data-apis/protocol-impl
52abf7a
Add a prototype of the dataframe interchange protocol

rgommers added enhancement New feature or request interchange-protocol labels Jun 25, 2021

rgommers reviewed Jun 28, 2021
View reviewed changes

Comment threadprotocol/pandas_implementation.py Outdated
Comment threadprotocol/pandas_implementation.py
Comment threadprotocol/pandas_implementation.py Outdated

kgryte added 2 commits June 28, 2021 10:48

Add todo
6010ae7

Merge branch 'main' ofhttps://github.com/data-apis/dataframe-apiint…
ac1a5ca
…o variable-length-string-support

kgryte changed the base branch from df-interchange-protocol-staging to mainJune 28, 2021 17:51

kgryte changed the base branch from main to df-interchange-protocol-stagingJune 28, 2021 17:51

Remove colons
89a7996

jorisvandenbossche reviewed Jul 8, 2021
View reviewed changes

Comment threadprotocol/dataframe_protocol_summary.md Outdated

jorisvandenbossche reviewed Jul 8, 2021
View reviewed changes

Comment threadprotocol/pandas_implementation.py
Comment threadprotocol/pandas_implementation.py

Fix grammar
a3ff4e7
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

kgryte added 3 commits July 8, 2021 09:20

Rename methods
ff84e8c

Rename methods
c954f3c

Update describe_null to indicate a byte array for string dtype
ed64fb7

kgryte added 9 commits July 19, 2021 09:20

Return encoding for missing values
9b9aecf

Update test
4026900

Use invalid value encoding
87d7143

Update copy
56ee2da

Use Arrow format strings
0035c90

Add get_buffers method to the protocol
91ed6a1

Remove individual methods
26fb48d

Update copy
0d0e94b

Refactor to return a dictionary of buffers
9ec830c

kgryte added a commit that referenced this pull request Jul 19, 2021
Port changes from#45
31cbf86
This is a fresh port of changes made in order to support variable length strings in order to provide a cleaner merge.

kgryte mentioned this pull request Jul 19, 2021
Add variable-length string support #47
Merged

kgryte added 2 commits July 19, 2021 11:28

Update comments
0dd4e2c

Fix copy
ade0d76

kgryte closed this Jul 19, 2021

kgryte added 12 commits June 23, 2021 13:32

Update protocol to support returning an offsets buffer
5a8d030

Add punctuation
3aac477

Update protocol to return dtype along with buffer
c7728e2

Add string support in various methods and add todos
c8000f7

Add support for resolving an offsets buffer
040f928

Add support for returning a data buffer for string dtypes
a982987

Update offsets buffer accessor
fd4d71b

Add implementation to convert a string column
e40f902

Add tests
c122b3c

Handle missing values
0d04af3

Update typing and docs
58fee89

Add comment
2c4a846

kgryte changed the title ~~WIP: Add variable-length string support~~Add variable-length string supportJun 25, 2021

kgryte requested a review from rgommers June 25, 2021 00:09

rgommers added 3 commits June 25, 2021 21:45

Remove outdated figures
f9f259c

Document that strided buffers do not need to be supported
a545faa

rgommers mentioned this pull request Jun 25, 2021
Tracking issue: dataframe protocol implementation #46
Open
8 tasks

Merge pull request#38from data-apis/protocol-impl
52abf7a
Add a prototype of the dataframe interchange protocol

rgommers added enhancement New feature or request interchange-protocol labels Jun 25, 2021

rgommers reviewed Jun 28, 2021
View reviewed changes

Comment threadprotocol/pandas_implementation.py Outdated
Comment threadprotocol/pandas_implementation.py
Comment threadprotocol/pandas_implementation.py Outdated

kgryte added 2 commits June 28, 2021 10:48

Add todo
6010ae7

Merge branch 'main' ofhttps://github.com/data-apis/dataframe-apiint…
ac1a5ca
…o variable-length-string-support

kgryte changed the base branch from df-interchange-protocol-staging to mainJune 28, 2021 17:51

kgryte changed the base branch from main to df-interchange-protocol-stagingJune 28, 2021 17:51

Remove colons
89a7996

jorisvandenbossche reviewed Jul 8, 2021
View reviewed changes

Comment threadprotocol/dataframe_protocol_summary.md Outdated

jorisvandenbossche reviewed Jul 8, 2021
View reviewed changes

Comment threadprotocol/pandas_implementation.py
Comment threadprotocol/pandas_implementation.py

Fix grammar
a3ff4e7
Co-authored-by: Joris Van den Bossche <jorisvandenbossche@gmail.com>

kgryte added 3 commits July 8, 2021 09:20

Rename methods
ff84e8c

Rename methods
c954f3c

Update describe_null to indicate a byte array for string dtype
ed64fb7

kgryte added 9 commits July 19, 2021 09:20

Return encoding for missing values
9b9aecf

Update test
4026900

Use invalid value encoding
87d7143

Update copy
56ee2da

Use Arrow format strings
0035c90

Add get_buffers method to the protocol
91ed6a1

Remove individual methods
26fb48d

Update copy
0d0e94b

Refactor to return a dictionary of buffers
9ec830c

kgryte added a commit that referenced this pull request Jul 19, 2021
Port changes from#45
31cbf86
This is a fresh port of changes made in order to support variable length strings in order to provide a cleaner merge.

kgryte mentioned this pull request Jul 19, 2021
Add variable-length string support #47
Merged

kgryte added 2 commits July 19, 2021 11:28

Update comments
0dd4e2c

Fix copy
ade0d76

kgryte closed this Jul 19, 2021

Uh oh!

Conversation

kgryte commented Jun 24, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rgommers commented Jun 28, 2021

Uh oh!

rgommers left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kgryte commented Jun 28, 2021

Uh oh!

jorisvandenbossche commented Jul 8, 2021

Uh oh!

Uh oh!

jorisvandenbossche left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kgryte commented Jul 8, 2021

Uh oh!

jorisvandenbossche commented Jul 8, 2021

Uh oh!

kgryte commented Jul 19, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

kgryte commented Jun 24, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rgommers commented Jun 28, 2021

Uh oh!

rgommers left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kgryte commented Jun 28, 2021

Uh oh!

jorisvandenbossche commented Jul 8, 2021

Uh oh!

Uh oh!

jorisvandenbossche left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

kgryte commented Jul 8, 2021

Uh oh!

jorisvandenbossche commented Jul 8, 2021

Uh oh!

kgryte commented Jul 19, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

kgryte commented Jun 24, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

rgommers commented Jun 28, 2021

kgryte commented Jun 24, 2021•
edited
Loading

kgryte commented Jun 24, 2021•
edited
Loading

kgryte commented Jun 24, 2021•
edited
Loading

kgryte commented Jun 24, 2021•
edited
Loading