ARROW-12109: [Rust][DataFusion] Implement SHOW COLUMNS

[alamb]

Rationale

Accessing the list of columns via select * from information_schema.columns (introduced in #9840) is a lot to type

See the doc for background: https://docs.google.com/document/d/12cpZUSNPqVH9Z0BBx6O8REu7TFqL-NPPAYCUPpDls1k/edit#

This is a sister PR to SHOW TABLES here: #9847

Proposal

Add support for SHOW COLUMNS FROM <table> command.

Following the MySQL syntax supported by sqlparser: https://dev.mysql.com/doc/refman/8.0/en/show-columns.html

Example Use

Setup:

echo "1,Foo,44.9" > /tmp/table.csv
echo "2,Bar,22.1" >> /tmp/table.csv
cargo run --bin datafusion-cli

Then run :

> CREATE EXTERNAL TABLE t(a int, b varchar, c float)
STORED AS CSV
LOCATION '/tmp/table.csv';
0 rows in set. Query took 0 seconds.
> show columns from t;
+---------------+--------------+------------+-------------+-----------+-------------+
| table_catalog | table_schema | table_name | column_name | data_type | is_nullable |
+---------------+--------------+------------+-------------+-----------+-------------+
| datafusion | public | t | a | Int32 | NO |
| datafusion | public | t | b | Utf8 | NO |
| datafusion | public | t | c | Float32 | NO |
+---------------+--------------+------------+-------------+-----------+-------------+
3 row in set. Query took 0 seconds.

Commentary

Note that the identifiers are case sensitive (which is a more general
problem that affects all name resolution, not just SHOW COLUMNS). Ideally this should also work:

> show columns from T;
Plan("Unknown relation for SHOW COLUMNS: T")
> select * from T;
Plan("Table or CTE with name \'T\' not found")

[github-actions]

https://issues.apache.org/jira/browse/ARROW-12109

[Dandandan]

Why rev and not listing the items in columns in reverse?

[alamb]

I was thinking because the identifier may not have a table_catalog or table_schema

So you have to handle the case of

table_name (position 0)
table_name (position 1), schema_name (postition 0)
table_name (postition 2), schema_name (postition 1), catalog_name (position 0),

So this formulation was what I could come up with that would match them up to the information_schema column names

I am open to other ways of doing it as well if you have suggestions

[Dandandan]

You are totally right 👍 I don't have a suggestion for an easier way

[Dandandan]

Really like those features @alamb really cool additions to make DataFusion more mature for BI tools / catalogs / tools like data build tool, etc.

[alamb]

FYI @returnString and @seddonm1

[jorgecarleitao]

Sorry for the delay, I had skimmed through it but did not left any feedback :/. Thanks a lot, @alamb for this.

I can only comment on the code part which looks great; I will leave design and functionality to @andygrove , as that is farther from my comfort zone for now.

My only general comment is that we should document this somewhere, e.g. README, together with the other associated functionality (schema, catalog, etc.). It can be a separate PR dedicated to it.

[returnString]

Great stuff! 👍

Agreed with @jorgecarleitao that we should have some kinda documentation available for these metadata operations and the catalog system more generally, I'd be happy to organise/contribute to that.

[alamb]

Thanks. I made a PR with some proposed docs here: #9895

[alamb]

I plan to wait until tomorrow to merge this in case @andygrove has any comments

[andygrove]

LGTM @alamb

[alamb]

Rationale

Accessing the list of columns via select * from information_schema.columns (introduced in #9840) is a lot to type

See the doc for background: https://docs.google.com/document/d/12cpZUSNPqVH9Z0BBx6O8REu7TFqL-NPPAYCUPpDls1k/edit#

This is a sister PR to SHOW TABLES here: #9847

Proposal

Add support for SHOW COLUMNS FROM <table> command.

Following the MySQL syntax supported by sqlparser: https://dev.mysql.com/doc/refman/8.0/en/show-columns.html

Example Use

Setup:

echo "1,Foo,44.9" > /tmp/table.csv
echo "2,Bar,22.1" >> /tmp/table.csv
cargo run --bin datafusion-cli

Then run :

> CREATE EXTERNAL TABLE t(a int, b varchar, c float)
STORED AS CSV
LOCATION '/tmp/table.csv';
0 rows in set. Query took 0 seconds.
> show columns from t;
+---------------+--------------+------------+-------------+-----------+-------------+
| table_catalog | table_schema | table_name | column_name | data_type | is_nullable |
+---------------+--------------+------------+-------------+-----------+-------------+
| datafusion | public | t | a | Int32 | NO |
| datafusion | public | t | b | Utf8 | NO |
| datafusion | public | t | c | Float32 | NO |
+---------------+--------------+------------+-------------+-----------+-------------+
3 row in set. Query took 0 seconds.

Commentary

Note that the identifiers are case sensitive (which is a more general
problem that affects all name resolution, not just SHOW COLUMNS). Ideally this should also work:

> show columns from T;
Plan("Unknown relation for SHOW COLUMNS: T")
> select * from T;
Plan("Table or CTE with name \'T\' not found")

[github-actions]

https://issues.apache.org/jira/browse/ARROW-12109

[Dandandan]

Why rev and not listing the items in columns in reverse?

[alamb]

I was thinking because the identifier may not have a table_catalog or table_schema

So you have to handle the case of

table_name (position 0)
table_name (position 1), schema_name (postition 0)
table_name (postition 2), schema_name (postition 1), catalog_name (position 0),

So this formulation was what I could come up with that would match them up to the information_schema column names

I am open to other ways of doing it as well if you have suggestions

[Dandandan]

You are totally right 👍 I don't have a suggestion for an easier way

[Dandandan]

Really like those features @alamb really cool additions to make DataFusion more mature for BI tools / catalogs / tools like data build tool, etc.

[alamb]

FYI @returnString and @seddonm1

[jorgecarleitao]

Sorry for the delay, I had skimmed through it but did not left any feedback :/. Thanks a lot, @alamb for this.

I can only comment on the code part which looks great; I will leave design and functionality to @andygrove , as that is farther from my comfort zone for now.

My only general comment is that we should document this somewhere, e.g. README, together with the other associated functionality (schema, catalog, etc.). It can be a separate PR dedicated to it.

[returnString]

Great stuff! 👍

Agreed with @jorgecarleitao that we should have some kinda documentation available for these metadata operations and the catalog system more generally, I'd be happy to organise/contribute to that.

[alamb]

Thanks. I made a PR with some proposed docs here: #9895

[alamb]

I plan to wait until tomorrow to merge this in case @andygrove has any comments

[andygrove]

LGTM @alamb

[alamb]

Rationale

Accessing the list of columns via select * from information_schema.columns (introduced in #9840) is a lot to type

See the doc for background: https://docs.google.com/document/d/12cpZUSNPqVH9Z0BBx6O8REu7TFqL-NPPAYCUPpDls1k/edit#

This is a sister PR to SHOW TABLES here: #9847

Proposal

Add support for SHOW COLUMNS FROM <table> command.

Following the MySQL syntax supported by sqlparser: https://dev.mysql.com/doc/refman/8.0/en/show-columns.html

Example Use

Setup:

echo "1,Foo,44.9" > /tmp/table.csv
echo "2,Bar,22.1" >> /tmp/table.csv
cargo run --bin datafusion-cli

Then run :

> CREATE EXTERNAL TABLE t(a int, b varchar, c float)
STORED AS CSV
LOCATION '/tmp/table.csv';
0 rows in set. Query took 0 seconds.
> show columns from t;
+---------------+--------------+------------+-------------+-----------+-------------+
| table_catalog | table_schema | table_name | column_name | data_type | is_nullable |
+---------------+--------------+------------+-------------+-----------+-------------+
| datafusion | public | t | a | Int32 | NO |
| datafusion | public | t | b | Utf8 | NO |
| datafusion | public | t | c | Float32 | NO |
+---------------+--------------+------------+-------------+-----------+-------------+
3 row in set. Query took 0 seconds.

Commentary

Note that the identifiers are case sensitive (which is a more general
problem that affects all name resolution, not just SHOW COLUMNS). Ideally this should also work:

> show columns from T;
Plan("Unknown relation for SHOW COLUMNS: T")
> select * from T;
Plan("Table or CTE with name \'T\' not found")

[github-actions]

https://issues.apache.org/jira/browse/ARROW-12109

[Dandandan]

Why rev and not listing the items in columns in reverse?

[alamb]

I was thinking because the identifier may not have a table_catalog or table_schema

So you have to handle the case of

table_name (position 0)
table_name (position 1), schema_name (postition 0)
table_name (postition 2), schema_name (postition 1), catalog_name (position 0),

So this formulation was what I could come up with that would match them up to the information_schema column names

I am open to other ways of doing it as well if you have suggestions

[Dandandan]

You are totally right 👍 I don't have a suggestion for an easier way

[Dandandan]

Really like those features @alamb really cool additions to make DataFusion more mature for BI tools / catalogs / tools like data build tool, etc.

[alamb]

FYI @returnString and @seddonm1

[jorgecarleitao]

Sorry for the delay, I had skimmed through it but did not left any feedback :/. Thanks a lot, @alamb for this.

I can only comment on the code part which looks great; I will leave design and functionality to @andygrove , as that is farther from my comfort zone for now.

My only general comment is that we should document this somewhere, e.g. README, together with the other associated functionality (schema, catalog, etc.). It can be a separate PR dedicated to it.

[returnString]

Great stuff! 👍

Agreed with @jorgecarleitao that we should have some kinda documentation available for these metadata operations and the catalog system more generally, I'd be happy to organise/contribute to that.

[alamb]

Thanks. I made a PR with some proposed docs here: #9895

[alamb]

I plan to wait until tomorrow to merge this in case @andygrove has any comments

[andygrove]

LGTM @alamb

[alamb]

Rationale

Accessing the list of columns via select * from information_schema.columns (introduced in #9840) is a lot to type

See the doc for background: https://docs.google.com/document/d/12cpZUSNPqVH9Z0BBx6O8REu7TFqL-NPPAYCUPpDls1k/edit#

This is a sister PR to SHOW TABLES here: #9847

Proposal

Add support for SHOW COLUMNS FROM <table> command.

Following the MySQL syntax supported by sqlparser: https://dev.mysql.com/doc/refman/8.0/en/show-columns.html

Example Use

Setup:

echo "1,Foo,44.9" > /tmp/table.csv
echo "2,Bar,22.1" >> /tmp/table.csv
cargo run --bin datafusion-cli

Then run :

> CREATE EXTERNAL TABLE t(a int, b varchar, c float)
STORED AS CSV
LOCATION '/tmp/table.csv';
0 rows in set. Query took 0 seconds.
> show columns from t;
+---------------+--------------+------------+-------------+-----------+-------------+
| table_catalog | table_schema | table_name | column_name | data_type | is_nullable |
+---------------+--------------+------------+-------------+-----------+-------------+
| datafusion | public | t | a | Int32 | NO |
| datafusion | public | t | b | Utf8 | NO |
| datafusion | public | t | c | Float32 | NO |
+---------------+--------------+------------+-------------+-----------+-------------+
3 row in set. Query took 0 seconds.

Commentary

Note that the identifiers are case sensitive (which is a more general
problem that affects all name resolution, not just SHOW COLUMNS). Ideally this should also work:

> show columns from T;
Plan("Unknown relation for SHOW COLUMNS: T")
> select * from T;
Plan("Table or CTE with name \'T\' not found")

[github-actions]

https://issues.apache.org/jira/browse/ARROW-12109

[Dandandan]

Why rev and not listing the items in columns in reverse?

[alamb]

I was thinking because the identifier may not have a table_catalog or table_schema

So you have to handle the case of

table_name (position 0)
table_name (position 1), schema_name (postition 0)
table_name (postition 2), schema_name (postition 1), catalog_name (position 0),

So this formulation was what I could come up with that would match them up to the information_schema column names

I am open to other ways of doing it as well if you have suggestions

[Dandandan]

You are totally right 👍 I don't have a suggestion for an easier way

[Dandandan]

Really like those features @alamb really cool additions to make DataFusion more mature for BI tools / catalogs / tools like data build tool, etc.

[alamb]

FYI @returnString and @seddonm1

[jorgecarleitao]

Sorry for the delay, I had skimmed through it but did not left any feedback :/. Thanks a lot, @alamb for this.

I can only comment on the code part which looks great; I will leave design and functionality to @andygrove , as that is farther from my comfort zone for now.

My only general comment is that we should document this somewhere, e.g. README, together with the other associated functionality (schema, catalog, etc.). It can be a separate PR dedicated to it.

[returnString]

Great stuff! 👍

Agreed with @jorgecarleitao that we should have some kinda documentation available for these metadata operations and the catalog system more generally, I'd be happy to organise/contribute to that.

[alamb]

Thanks. I made a PR with some proposed docs here: #9895

[alamb]

I plan to wait until tomorrow to merge this in case @andygrove has any comments

[andygrove]

LGTM @alamb

[alamb]

Rationale

Accessing the list of columns via select * from information_schema.columns (introduced in #9840) is a lot to type

See the doc for background: https://docs.google.com/document/d/12cpZUSNPqVH9Z0BBx6O8REu7TFqL-NPPAYCUPpDls1k/edit#

This is a sister PR to SHOW TABLES here: #9847

Proposal

Add support for SHOW COLUMNS FROM <table> command.

Following the MySQL syntax supported by sqlparser: https://dev.mysql.com/doc/refman/8.0/en/show-columns.html

Example Use

Setup:

echo "1,Foo,44.9" > /tmp/table.csv
echo "2,Bar,22.1" >> /tmp/table.csv
cargo run --bin datafusion-cli

Then run :

> CREATE EXTERNAL TABLE t(a int, b varchar, c float)
STORED AS CSV
LOCATION '/tmp/table.csv';
0 rows in set. Query took 0 seconds.
> show columns from t;
+---------------+--------------+------------+-------------+-----------+-------------+
| table_catalog | table_schema | table_name | column_name | data_type | is_nullable |
+---------------+--------------+------------+-------------+-----------+-------------+
| datafusion | public | t | a | Int32 | NO |
| datafusion | public | t | b | Utf8 | NO |
| datafusion | public | t | c | Float32 | NO |
+---------------+--------------+------------+-------------+-----------+-------------+
3 row in set. Query took 0 seconds.

Commentary

Note that the identifiers are case sensitive (which is a more general
problem that affects all name resolution, not just SHOW COLUMNS). Ideally this should also work:

> show columns from T;
Plan("Unknown relation for SHOW COLUMNS: T")
> select * from T;
Plan("Table or CTE with name \'T\' not found")

[github-actions]

https://issues.apache.org/jira/browse/ARROW-12109

[Dandandan]

Why rev and not listing the items in columns in reverse?

[alamb]

I was thinking because the identifier may not have a table_catalog or table_schema

So you have to handle the case of

table_name (position 0)
table_name (position 1), schema_name (postition 0)
table_name (postition 2), schema_name (postition 1), catalog_name (position 0),

So this formulation was what I could come up with that would match them up to the information_schema column names

I am open to other ways of doing it as well if you have suggestions

[Dandandan]

You are totally right 👍 I don't have a suggestion for an easier way

[Dandandan]

Really like those features @alamb really cool additions to make DataFusion more mature for BI tools / catalogs / tools like data build tool, etc.

[alamb]

FYI @returnString and @seddonm1

[jorgecarleitao]

Sorry for the delay, I had skimmed through it but did not left any feedback :/. Thanks a lot, @alamb for this.

I can only comment on the code part which looks great; I will leave design and functionality to @andygrove , as that is farther from my comfort zone for now.

My only general comment is that we should document this somewhere, e.g. README, together with the other associated functionality (schema, catalog, etc.). It can be a separate PR dedicated to it.

[returnString]

Great stuff! 👍

Agreed with @jorgecarleitao that we should have some kinda documentation available for these metadata operations and the catalog system more generally, I'd be happy to organise/contribute to that.

[alamb]

Thanks. I made a PR with some proposed docs here: #9895

[alamb]

I plan to wait until tomorrow to merge this in case @andygrove has any comments

[andygrove]

LGTM @alamb

[alamb]

Rationale

Accessing the list of columns via select * from information_schema.columns (introduced in #9840) is a lot to type

See the doc for background: https://docs.google.com/document/d/12cpZUSNPqVH9Z0BBx6O8REu7TFqL-NPPAYCUPpDls1k/edit#

This is a sister PR to SHOW TABLES here: #9847

Proposal

Add support for SHOW COLUMNS FROM <table> command.

Following the MySQL syntax supported by sqlparser: https://dev.mysql.com/doc/refman/8.0/en/show-columns.html

Example Use

Setup:

echo "1,Foo,44.9" > /tmp/table.csv
echo "2,Bar,22.1" >> /tmp/table.csv
cargo run --bin datafusion-cli

Then run :

> CREATE EXTERNAL TABLE t(a int, b varchar, c float)
STORED AS CSV
LOCATION '/tmp/table.csv';
0 rows in set. Query took 0 seconds.
> show columns from t;
+---------------+--------------+------------+-------------+-----------+-------------+
| table_catalog | table_schema | table_name | column_name | data_type | is_nullable |
+---------------+--------------+------------+-------------+-----------+-------------+
| datafusion | public | t | a | Int32 | NO |
| datafusion | public | t | b | Utf8 | NO |
| datafusion | public | t | c | Float32 | NO |
+---------------+--------------+------------+-------------+-----------+-------------+
3 row in set. Query took 0 seconds.

Commentary

Note that the identifiers are case sensitive (which is a more general
problem that affects all name resolution, not just SHOW COLUMNS). Ideally this should also work:

> show columns from T;
Plan("Unknown relation for SHOW COLUMNS: T")
> select * from T;
Plan("Table or CTE with name \'T\' not found")

[github-actions]

https://issues.apache.org/jira/browse/ARROW-12109

[Dandandan]

Why rev and not listing the items in columns in reverse?

[alamb]

I was thinking because the identifier may not have a table_catalog or table_schema

So you have to handle the case of

table_name (position 0)
table_name (position 1), schema_name (postition 0)
table_name (postition 2), schema_name (postition 1), catalog_name (position 0),

So this formulation was what I could come up with that would match them up to the information_schema column names

I am open to other ways of doing it as well if you have suggestions

[Dandandan]

You are totally right 👍 I don't have a suggestion for an easier way

[Dandandan]

Really like those features @alamb really cool additions to make DataFusion more mature for BI tools / catalogs / tools like data build tool, etc.

[alamb]

FYI @returnString and @seddonm1

[jorgecarleitao]

Sorry for the delay, I had skimmed through it but did not left any feedback :/. Thanks a lot, @alamb for this.

I can only comment on the code part which looks great; I will leave design and functionality to @andygrove , as that is farther from my comfort zone for now.

My only general comment is that we should document this somewhere, e.g. README, together with the other associated functionality (schema, catalog, etc.). It can be a separate PR dedicated to it.

[returnString]

Great stuff! 👍

Agreed with @jorgecarleitao that we should have some kinda documentation available for these metadata operations and the catalog system more generally, I'd be happy to organise/contribute to that.

[alamb]

Thanks. I made a PR with some proposed docs here: #9895

[alamb]

I plan to wait until tomorrow to merge this in case @andygrove has any comments

[andygrove]

LGTM @alamb

[alamb]

Rationale

Accessing the list of columns via select * from information_schema.columns (introduced in #9840) is a lot to type

See the doc for background: https://docs.google.com/document/d/12cpZUSNPqVH9Z0BBx6O8REu7TFqL-NPPAYCUPpDls1k/edit#

This is a sister PR to SHOW TABLES here: #9847

Proposal

Add support for SHOW COLUMNS FROM <table> command.

Following the MySQL syntax supported by sqlparser: https://dev.mysql.com/doc/refman/8.0/en/show-columns.html

Example Use

Setup:

echo "1,Foo,44.9" > /tmp/table.csv
echo "2,Bar,22.1" >> /tmp/table.csv
cargo run --bin datafusion-cli

Then run :

> CREATE EXTERNAL TABLE t(a int, b varchar, c float)
STORED AS CSV
LOCATION '/tmp/table.csv';
0 rows in set. Query took 0 seconds.
> show columns from t;
+---------------+--------------+------------+-------------+-----------+-------------+
| table_catalog | table_schema | table_name | column_name | data_type | is_nullable |
+---------------+--------------+------------+-------------+-----------+-------------+
| datafusion | public | t | a | Int32 | NO |
| datafusion | public | t | b | Utf8 | NO |
| datafusion | public | t | c | Float32 | NO |
+---------------+--------------+------------+-------------+-----------+-------------+
3 row in set. Query took 0 seconds.

Commentary

Note that the identifiers are case sensitive (which is a more general
problem that affects all name resolution, not just SHOW COLUMNS). Ideally this should also work:

> show columns from T;
Plan("Unknown relation for SHOW COLUMNS: T")
> select * from T;
Plan("Table or CTE with name \'T\' not found")

[github-actions]

https://issues.apache.org/jira/browse/ARROW-12109

[Dandandan]

Why rev and not listing the items in columns in reverse?

[alamb]

I was thinking because the identifier may not have a table_catalog or table_schema

So you have to handle the case of

table_name (position 0)
table_name (position 1), schema_name (postition 0)
table_name (postition 2), schema_name (postition 1), catalog_name (position 0),

So this formulation was what I could come up with that would match them up to the information_schema column names

I am open to other ways of doing it as well if you have suggestions

[Dandandan]

You are totally right 👍 I don't have a suggestion for an easier way

[Dandandan]

Really like those features @alamb really cool additions to make DataFusion more mature for BI tools / catalogs / tools like data build tool, etc.

[alamb]

FYI @returnString and @seddonm1

[jorgecarleitao]

Sorry for the delay, I had skimmed through it but did not left any feedback :/. Thanks a lot, @alamb for this.

I can only comment on the code part which looks great; I will leave design and functionality to @andygrove , as that is farther from my comfort zone for now.

My only general comment is that we should document this somewhere, e.g. README, together with the other associated functionality (schema, catalog, etc.). It can be a separate PR dedicated to it.

[returnString]

Great stuff! 👍

Agreed with @jorgecarleitao that we should have some kinda documentation available for these metadata operations and the catalog system more generally, I'd be happy to organise/contribute to that.

[alamb]

Thanks. I made a PR with some proposed docs here: #9895

[alamb]

I plan to wait until tomorrow to merge this in case @andygrove has any comments

[andygrove]

LGTM @alamb

[alamb]

Rationale

Accessing the list of columns via select * from information_schema.columns (introduced in #9840) is a lot to type

See the doc for background: https://docs.google.com/document/d/12cpZUSNPqVH9Z0BBx6O8REu7TFqL-NPPAYCUPpDls1k/edit#

This is a sister PR to SHOW TABLES here: #9847

Proposal

Add support for SHOW COLUMNS FROM <table> command.

Following the MySQL syntax supported by sqlparser: https://dev.mysql.com/doc/refman/8.0/en/show-columns.html

Example Use

Setup:

echo "1,Foo,44.9" > /tmp/table.csv
echo "2,Bar,22.1" >> /tmp/table.csv
cargo run --bin datafusion-cli

Then run :

> CREATE EXTERNAL TABLE t(a int, b varchar, c float)
STORED AS CSV
LOCATION '/tmp/table.csv';
0 rows in set. Query took 0 seconds.
> show columns from t;
+---------------+--------------+------------+-------------+-----------+-------------+
| table_catalog | table_schema | table_name | column_name | data_type | is_nullable |
+---------------+--------------+------------+-------------+-----------+-------------+
| datafusion | public | t | a | Int32 | NO |
| datafusion | public | t | b | Utf8 | NO |
| datafusion | public | t | c | Float32 | NO |
+---------------+--------------+------------+-------------+-----------+-------------+
3 row in set. Query took 0 seconds.

Commentary

Note that the identifiers are case sensitive (which is a more general
problem that affects all name resolution, not just SHOW COLUMNS). Ideally this should also work:

> show columns from T;
Plan("Unknown relation for SHOW COLUMNS: T")
> select * from T;
Plan("Table or CTE with name \'T\' not found")

[github-actions]

https://issues.apache.org/jira/browse/ARROW-12109

[Dandandan]

Why rev and not listing the items in columns in reverse?

[alamb]

I was thinking because the identifier may not have a table_catalog or table_schema

So you have to handle the case of

table_name (position 0)
table_name (position 1), schema_name (postition 0)
table_name (postition 2), schema_name (postition 1), catalog_name (position 0),

So this formulation was what I could come up with that would match them up to the information_schema column names

I am open to other ways of doing it as well if you have suggestions

[Dandandan]

You are totally right 👍 I don't have a suggestion for an easier way

[Dandandan]

Really like those features @alamb really cool additions to make DataFusion more mature for BI tools / catalogs / tools like data build tool, etc.

[alamb]

FYI @returnString and @seddonm1

[jorgecarleitao]

Sorry for the delay, I had skimmed through it but did not left any feedback :/. Thanks a lot, @alamb for this.

I can only comment on the code part which looks great; I will leave design and functionality to @andygrove , as that is farther from my comfort zone for now.

My only general comment is that we should document this somewhere, e.g. README, together with the other associated functionality (schema, catalog, etc.). It can be a separate PR dedicated to it.

[returnString]

Great stuff! 👍

Agreed with @jorgecarleitao that we should have some kinda documentation available for these metadata operations and the catalog system more generally, I'd be happy to organise/contribute to that.

[alamb]

Thanks. I made a PR with some proposed docs here: #9895

[alamb]

I plan to wait until tomorrow to merge this in case @andygrove has any comments

[andygrove]

LGTM @alamb