ARROW-12107: [Rust][DataFusion] Support `SELECT * from information_schema.columns` by alamb · Pull Request #9840 · apache/arrow

alamb · 2021-03-29T19:10:45Z

Builds on the code in #9818

Rationale

Provide schema metadata access (so a user can see what columns exist and their type).

See the doc for background: https://docs.google.com/document/d/12cpZUSNPqVH9Z0BBx6O8REu7TFqL-NPPAYCUPpDls1k/edit#

I plan to add support for SHOW COLUMNS possibly as a follow on PR (though I have found out that SHOW COLUMNS and SHOW TABLES are not supported by either MySQL or by Postgres 🤔 )

Changes

I chose to add the first 15 columns from information_schema.columns You can see the full list in Postgres here and SQL Server here.

There are a bunch more columns that say "Applies to features not available in PostgreSQL" and that don't apply to DataFusion either-- since my usecase is to get the basic schema information out I chose not to add a bunch of columns that are always null at this time.

I feel the use of column builders here is somewhat awkward (as it requires many calls to unwrap). I am thinking of a follow on PR to refactor this code to use Vec<String> and Vec<u64> and then create StringArray and UInt64Array directly from them but for now I just want the functionality

Example use

Setup:

echo "1,Foo,44.9" > /tmp/table.csv
echo "2,Bar,22.1" >> /tmp/table.csv
cargo run --bin datafusion-cli

Then run :

> CREATE EXTERNAL TABLE t(a int, b varchar, c float)
STORED AS CSV
LOCATION '/tmp/table.csv';
0 rows in set. Query took 0 seconds.
> select table_name, column_name, ordinal_position, is_nullable, data_type from information_schema.columns;
+------------+-------------+------------------+-------------+-----------+
| table_name | column_name | ordinal_position | is_nullable | data_type |
+------------+-------------+------------------+-------------+-----------+
| t | a | 0 | NO | Int32 |
| t | b | 1 | NO | Utf8 |
| t | c | 2 | NO | Float32 |
+------------+-------------+------------------+-------------+-----------+
3 row in set. Query took 0 seconds.

github-actions · 2021-03-30T01:45:51Z

https://issues.apache.org/jira/browse/ARROW-12107

andygrove

I'm excited to see this. Thanks @alamb

I had a proof-of-concept JDBC driver working against DataFusion at one point. One day I will find time to work on this again. With the information schema support it will really help with making this work with BI tools.

alamb · 2021-03-31T17:57:38Z

FYI @returnString @Dandandan and @jorgecarleitao

jorgecarleitao

LGTM! Thanks a lot, this metadata part is really starting to shine! ✨

jorgecarleitao · 2021-03-31T18:02:10Z

Not it matters much, just more like an FYI: this can cause large binaries as every variation used is compiled individually.

I can change it into just taking &str -- though I think in the case since there is just one callsite there is likely to be just one version of the code

yeah, it was really just if you had not though about it. No need to change anything imo 👍

jorgecarleitao · 2021-03-31T18:03:52Z

it may be more idiomatic to use Into, or From.

Ok. Good idea

In 98f702da90d18d8ae855a5eca6b0d1d6c1809551

(which I now realize I put into #9866 rather than this PR 🤦 ) -- but it should get in that PR hopefully

returnString

Looks great! And selfishly, really happy to see the namespacing stuff being used to power cool new features 😄

alamb · 2021-03-31T20:49:04Z

Looks great! And selfishly, really happy to see the namespacing stuff being used to power cool new features 😄

@returnString -- the namespacing stuff is pretty cool. I didn't realize how much easier it would make my life when it came in!

…ema.columns

# Rationale Accessing the list of columns via `select * from information_schema.columns` (introduced in #9840) is a lot to type See the doc for background: https://docs.google.com/document/d/12cpZUSNPqVH9Z0BBx6O8REu7TFqL-NPPAYCUPpDls1k/edit# This is a sister PR to `SHOW TABLES` here: #9847 # Proposal Add support for `SHOW COLUMNS FROM <table>` command. Following the MySQL syntax supported by sqlparser: https://dev.mysql.com/doc/refman/8.0/en/show-columns.html # Example Use Setup: ``` echo "1,Foo,44.9" > /tmp/table.csv echo "2,Bar,22.1" >> /tmp/table.csv cargo run --bin datafusion-cli ``` Then run : ``` > CREATE EXTERNAL TABLE t(a int, b varchar, c float) STORED AS CSV LOCATION '/tmp/table.csv'; 0 rows in set. Query took 0 seconds. > show columns from t; +---------------+--------------+------------+-------------+-----------+-------------+ | table_catalog | table_schema | table_name | column_name | data_type | is_nullable | +---------------+--------------+------------+-------------+-----------+-------------+ | datafusion | public | t | a | Int32 | NO | | datafusion | public | t | b | Utf8 | NO | | datafusion | public | t | c | Float32 | NO | +---------------+--------------+------------+-------------+-----------+-------------+ 3 row in set. Query took 0 seconds. ``` # Commentary Note that the identifiers are case sensitive (which is a more general problem that affects all name resolution, not just `SHOW COLUMNS`). Ideally this should also work: ``` > show columns from T; Plan("Unknown relation for SHOW COLUMNS: T") > select * from T; Plan("Table or CTE with name \'T\' not found") ``` Closes#9866 from alamb/alamb/show_columns Authored-by: Andrew Lamb <andrew@nerdnetworks.org> Signed-off-by: Andrew Lamb <andrew@nerdnetworks.org>

alamb · 2021-03-29T19:10:45Z

Builds on the code in #9818

Rationale

Provide schema metadata access (so a user can see what columns exist and their type).

See the doc for background: https://docs.google.com/document/d/12cpZUSNPqVH9Z0BBx6O8REu7TFqL-NPPAYCUPpDls1k/edit#

I plan to add support for SHOW COLUMNS possibly as a follow on PR (though I have found out that SHOW COLUMNS and SHOW TABLES are not supported by either MySQL or by Postgres 🤔 )

Changes

I chose to add the first 15 columns from information_schema.columns You can see the full list in Postgres here and SQL Server here.

There are a bunch more columns that say "Applies to features not available in PostgreSQL" and that don't apply to DataFusion either-- since my usecase is to get the basic schema information out I chose not to add a bunch of columns that are always null at this time.

I feel the use of column builders here is somewhat awkward (as it requires many calls to unwrap). I am thinking of a follow on PR to refactor this code to use Vec<String> and Vec<u64> and then create StringArray and UInt64Array directly from them but for now I just want the functionality

Example use

Setup:

echo "1,Foo,44.9" > /tmp/table.csv
echo "2,Bar,22.1" >> /tmp/table.csv
cargo run --bin datafusion-cli

Then run :

> CREATE EXTERNAL TABLE t(a int, b varchar, c float)
STORED AS CSV
LOCATION '/tmp/table.csv';
0 rows in set. Query took 0 seconds.
> select table_name, column_name, ordinal_position, is_nullable, data_type from information_schema.columns;
+------------+-------------+------------------+-------------+-----------+
| table_name | column_name | ordinal_position | is_nullable | data_type |
+------------+-------------+------------------+-------------+-----------+
| t | a | 0 | NO | Int32 |
| t | b | 1 | NO | Utf8 |
| t | c | 2 | NO | Float32 |
+------------+-------------+------------------+-------------+-----------+
3 row in set. Query took 0 seconds.

github-actions · 2021-03-30T01:45:51Z

https://issues.apache.org/jira/browse/ARROW-12107

andygrove

I'm excited to see this. Thanks @alamb

I had a proof-of-concept JDBC driver working against DataFusion at one point. One day I will find time to work on this again. With the information schema support it will really help with making this work with BI tools.

alamb · 2021-03-31T17:57:38Z

FYI @returnString @Dandandan and @jorgecarleitao

jorgecarleitao

LGTM! Thanks a lot, this metadata part is really starting to shine! ✨

jorgecarleitao · 2021-03-31T18:02:10Z

Not it matters much, just more like an FYI: this can cause large binaries as every variation used is compiled individually.

I can change it into just taking &str -- though I think in the case since there is just one callsite there is likely to be just one version of the code

yeah, it was really just if you had not though about it. No need to change anything imo 👍

jorgecarleitao · 2021-03-31T18:03:52Z

it may be more idiomatic to use Into, or From.

Ok. Good idea

In 98f702da90d18d8ae855a5eca6b0d1d6c1809551

(which I now realize I put into #9866 rather than this PR 🤦 ) -- but it should get in that PR hopefully

returnString

Looks great! And selfishly, really happy to see the namespacing stuff being used to power cool new features 😄

alamb · 2021-03-31T20:49:04Z

Looks great! And selfishly, really happy to see the namespacing stuff being used to power cool new features 😄

@returnString -- the namespacing stuff is pretty cool. I didn't realize how much easier it would make my life when it came in!

…ema.columns

# Rationale Accessing the list of columns via `select * from information_schema.columns` (introduced in #9840) is a lot to type See the doc for background: https://docs.google.com/document/d/12cpZUSNPqVH9Z0BBx6O8REu7TFqL-NPPAYCUPpDls1k/edit# This is a sister PR to `SHOW TABLES` here: #9847 # Proposal Add support for `SHOW COLUMNS FROM <table>` command. Following the MySQL syntax supported by sqlparser: https://dev.mysql.com/doc/refman/8.0/en/show-columns.html # Example Use Setup: ``` echo "1,Foo,44.9" > /tmp/table.csv echo "2,Bar,22.1" >> /tmp/table.csv cargo run --bin datafusion-cli ``` Then run : ``` > CREATE EXTERNAL TABLE t(a int, b varchar, c float) STORED AS CSV LOCATION '/tmp/table.csv'; 0 rows in set. Query took 0 seconds. > show columns from t; +---------------+--------------+------------+-------------+-----------+-------------+ | table_catalog | table_schema | table_name | column_name | data_type | is_nullable | +---------------+--------------+------------+-------------+-----------+-------------+ | datafusion | public | t | a | Int32 | NO | | datafusion | public | t | b | Utf8 | NO | | datafusion | public | t | c | Float32 | NO | +---------------+--------------+------------+-------------+-----------+-------------+ 3 row in set. Query took 0 seconds. ``` # Commentary Note that the identifiers are case sensitive (which is a more general problem that affects all name resolution, not just `SHOW COLUMNS`). Ideally this should also work: ``` > show columns from T; Plan("Unknown relation for SHOW COLUMNS: T") > select * from T; Plan("Table or CTE with name \'T\' not found") ``` Closes#9866 from alamb/alamb/show_columns Authored-by: Andrew Lamb <andrew@nerdnetworks.org> Signed-off-by: Andrew Lamb <andrew@nerdnetworks.org>

alamb · 2021-03-29T19:10:45Z

Builds on the code in #9818

Rationale

Provide schema metadata access (so a user can see what columns exist and their type).

See the doc for background: https://docs.google.com/document/d/12cpZUSNPqVH9Z0BBx6O8REu7TFqL-NPPAYCUPpDls1k/edit#

I plan to add support for SHOW COLUMNS possibly as a follow on PR (though I have found out that SHOW COLUMNS and SHOW TABLES are not supported by either MySQL or by Postgres 🤔 )

Changes

I chose to add the first 15 columns from information_schema.columns You can see the full list in Postgres here and SQL Server here.

There are a bunch more columns that say "Applies to features not available in PostgreSQL" and that don't apply to DataFusion either-- since my usecase is to get the basic schema information out I chose not to add a bunch of columns that are always null at this time.

I feel the use of column builders here is somewhat awkward (as it requires many calls to unwrap). I am thinking of a follow on PR to refactor this code to use Vec<String> and Vec<u64> and then create StringArray and UInt64Array directly from them but for now I just want the functionality

Example use

Setup:

echo "1,Foo,44.9" > /tmp/table.csv
echo "2,Bar,22.1" >> /tmp/table.csv
cargo run --bin datafusion-cli

Then run :

> CREATE EXTERNAL TABLE t(a int, b varchar, c float)
STORED AS CSV
LOCATION '/tmp/table.csv';
0 rows in set. Query took 0 seconds.
> select table_name, column_name, ordinal_position, is_nullable, data_type from information_schema.columns;
+------------+-------------+------------------+-------------+-----------+
| table_name | column_name | ordinal_position | is_nullable | data_type |
+------------+-------------+------------------+-------------+-----------+
| t | a | 0 | NO | Int32 |
| t | b | 1 | NO | Utf8 |
| t | c | 2 | NO | Float32 |
+------------+-------------+------------------+-------------+-----------+
3 row in set. Query took 0 seconds.

github-actions · 2021-03-30T01:45:51Z

https://issues.apache.org/jira/browse/ARROW-12107

andygrove

I'm excited to see this. Thanks @alamb

I had a proof-of-concept JDBC driver working against DataFusion at one point. One day I will find time to work on this again. With the information schema support it will really help with making this work with BI tools.

alamb · 2021-03-31T17:57:38Z

FYI @returnString @Dandandan and @jorgecarleitao

jorgecarleitao

LGTM! Thanks a lot, this metadata part is really starting to shine! ✨

jorgecarleitao · 2021-03-31T18:02:10Z

Not it matters much, just more like an FYI: this can cause large binaries as every variation used is compiled individually.

I can change it into just taking &str -- though I think in the case since there is just one callsite there is likely to be just one version of the code

yeah, it was really just if you had not though about it. No need to change anything imo 👍

jorgecarleitao · 2021-03-31T18:03:52Z

it may be more idiomatic to use Into, or From.

Ok. Good idea

In 98f702da90d18d8ae855a5eca6b0d1d6c1809551

(which I now realize I put into #9866 rather than this PR 🤦 ) -- but it should get in that PR hopefully

returnString

Looks great! And selfishly, really happy to see the namespacing stuff being used to power cool new features 😄

alamb · 2021-03-31T20:49:04Z

Looks great! And selfishly, really happy to see the namespacing stuff being used to power cool new features 😄

@returnString -- the namespacing stuff is pretty cool. I didn't realize how much easier it would make my life when it came in!

…ema.columns

# Rationale Accessing the list of columns via `select * from information_schema.columns` (introduced in #9840) is a lot to type See the doc for background: https://docs.google.com/document/d/12cpZUSNPqVH9Z0BBx6O8REu7TFqL-NPPAYCUPpDls1k/edit# This is a sister PR to `SHOW TABLES` here: #9847 # Proposal Add support for `SHOW COLUMNS FROM <table>` command. Following the MySQL syntax supported by sqlparser: https://dev.mysql.com/doc/refman/8.0/en/show-columns.html # Example Use Setup: ``` echo "1,Foo,44.9" > /tmp/table.csv echo "2,Bar,22.1" >> /tmp/table.csv cargo run --bin datafusion-cli ``` Then run : ``` > CREATE EXTERNAL TABLE t(a int, b varchar, c float) STORED AS CSV LOCATION '/tmp/table.csv'; 0 rows in set. Query took 0 seconds. > show columns from t; +---------------+--------------+------------+-------------+-----------+-------------+ | table_catalog | table_schema | table_name | column_name | data_type | is_nullable | +---------------+--------------+------------+-------------+-----------+-------------+ | datafusion | public | t | a | Int32 | NO | | datafusion | public | t | b | Utf8 | NO | | datafusion | public | t | c | Float32 | NO | +---------------+--------------+------------+-------------+-----------+-------------+ 3 row in set. Query took 0 seconds. ``` # Commentary Note that the identifiers are case sensitive (which is a more general problem that affects all name resolution, not just `SHOW COLUMNS`). Ideally this should also work: ``` > show columns from T; Plan("Unknown relation for SHOW COLUMNS: T") > select * from T; Plan("Table or CTE with name \'T\' not found") ``` Closes#9866 from alamb/alamb/show_columns Authored-by: Andrew Lamb <andrew@nerdnetworks.org> Signed-off-by: Andrew Lamb <andrew@nerdnetworks.org>

alamb · 2021-03-29T19:10:45Z

Builds on the code in #9818

Rationale

Provide schema metadata access (so a user can see what columns exist and their type).

See the doc for background: https://docs.google.com/document/d/12cpZUSNPqVH9Z0BBx6O8REu7TFqL-NPPAYCUPpDls1k/edit#

I plan to add support for SHOW COLUMNS possibly as a follow on PR (though I have found out that SHOW COLUMNS and SHOW TABLES are not supported by either MySQL or by Postgres 🤔 )

Changes

I chose to add the first 15 columns from information_schema.columns You can see the full list in Postgres here and SQL Server here.

There are a bunch more columns that say "Applies to features not available in PostgreSQL" and that don't apply to DataFusion either-- since my usecase is to get the basic schema information out I chose not to add a bunch of columns that are always null at this time.

I feel the use of column builders here is somewhat awkward (as it requires many calls to unwrap). I am thinking of a follow on PR to refactor this code to use Vec<String> and Vec<u64> and then create StringArray and UInt64Array directly from them but for now I just want the functionality

Example use

Setup:

echo "1,Foo,44.9" > /tmp/table.csv
echo "2,Bar,22.1" >> /tmp/table.csv
cargo run --bin datafusion-cli

Then run :

> CREATE EXTERNAL TABLE t(a int, b varchar, c float)
STORED AS CSV
LOCATION '/tmp/table.csv';
0 rows in set. Query took 0 seconds.
> select table_name, column_name, ordinal_position, is_nullable, data_type from information_schema.columns;
+------------+-------------+------------------+-------------+-----------+
| table_name | column_name | ordinal_position | is_nullable | data_type |
+------------+-------------+------------------+-------------+-----------+
| t | a | 0 | NO | Int32 |
| t | b | 1 | NO | Utf8 |
| t | c | 2 | NO | Float32 |
+------------+-------------+------------------+-------------+-----------+
3 row in set. Query took 0 seconds.

github-actions · 2021-03-30T01:45:51Z

https://issues.apache.org/jira/browse/ARROW-12107

andygrove

I'm excited to see this. Thanks @alamb

I had a proof-of-concept JDBC driver working against DataFusion at one point. One day I will find time to work on this again. With the information schema support it will really help with making this work with BI tools.

alamb · 2021-03-31T17:57:38Z

FYI @returnString @Dandandan and @jorgecarleitao

jorgecarleitao

LGTM! Thanks a lot, this metadata part is really starting to shine! ✨

jorgecarleitao · 2021-03-31T18:02:10Z

Not it matters much, just more like an FYI: this can cause large binaries as every variation used is compiled individually.

I can change it into just taking &str -- though I think in the case since there is just one callsite there is likely to be just one version of the code

yeah, it was really just if you had not though about it. No need to change anything imo 👍

jorgecarleitao · 2021-03-31T18:03:52Z

it may be more idiomatic to use Into, or From.

Ok. Good idea

In 98f702da90d18d8ae855a5eca6b0d1d6c1809551

(which I now realize I put into #9866 rather than this PR 🤦 ) -- but it should get in that PR hopefully

returnString

Looks great! And selfishly, really happy to see the namespacing stuff being used to power cool new features 😄

alamb · 2021-03-31T20:49:04Z

Looks great! And selfishly, really happy to see the namespacing stuff being used to power cool new features 😄

@returnString -- the namespacing stuff is pretty cool. I didn't realize how much easier it would make my life when it came in!

…ema.columns

# Rationale Accessing the list of columns via `select * from information_schema.columns` (introduced in #9840) is a lot to type See the doc for background: https://docs.google.com/document/d/12cpZUSNPqVH9Z0BBx6O8REu7TFqL-NPPAYCUPpDls1k/edit# This is a sister PR to `SHOW TABLES` here: #9847 # Proposal Add support for `SHOW COLUMNS FROM <table>` command. Following the MySQL syntax supported by sqlparser: https://dev.mysql.com/doc/refman/8.0/en/show-columns.html # Example Use Setup: ``` echo "1,Foo,44.9" > /tmp/table.csv echo "2,Bar,22.1" >> /tmp/table.csv cargo run --bin datafusion-cli ``` Then run : ``` > CREATE EXTERNAL TABLE t(a int, b varchar, c float) STORED AS CSV LOCATION '/tmp/table.csv'; 0 rows in set. Query took 0 seconds. > show columns from t; +---------------+--------------+------------+-------------+-----------+-------------+ | table_catalog | table_schema | table_name | column_name | data_type | is_nullable | +---------------+--------------+------------+-------------+-----------+-------------+ | datafusion | public | t | a | Int32 | NO | | datafusion | public | t | b | Utf8 | NO | | datafusion | public | t | c | Float32 | NO | +---------------+--------------+------------+-------------+-----------+-------------+ 3 row in set. Query took 0 seconds. ``` # Commentary Note that the identifiers are case sensitive (which is a more general problem that affects all name resolution, not just `SHOW COLUMNS`). Ideally this should also work: ``` > show columns from T; Plan("Unknown relation for SHOW COLUMNS: T") > select * from T; Plan("Table or CTE with name \'T\' not found") ``` Closes#9866 from alamb/alamb/show_columns Authored-by: Andrew Lamb <andrew@nerdnetworks.org> Signed-off-by: Andrew Lamb <andrew@nerdnetworks.org>

alamb · 2021-03-29T19:10:45Z

Builds on the code in #9818

Rationale

Provide schema metadata access (so a user can see what columns exist and their type).

See the doc for background: https://docs.google.com/document/d/12cpZUSNPqVH9Z0BBx6O8REu7TFqL-NPPAYCUPpDls1k/edit#

I plan to add support for SHOW COLUMNS possibly as a follow on PR (though I have found out that SHOW COLUMNS and SHOW TABLES are not supported by either MySQL or by Postgres 🤔 )

Changes

I chose to add the first 15 columns from information_schema.columns You can see the full list in Postgres here and SQL Server here.

There are a bunch more columns that say "Applies to features not available in PostgreSQL" and that don't apply to DataFusion either-- since my usecase is to get the basic schema information out I chose not to add a bunch of columns that are always null at this time.

I feel the use of column builders here is somewhat awkward (as it requires many calls to unwrap). I am thinking of a follow on PR to refactor this code to use Vec<String> and Vec<u64> and then create StringArray and UInt64Array directly from them but for now I just want the functionality

Example use

Setup:

echo "1,Foo,44.9" > /tmp/table.csv
echo "2,Bar,22.1" >> /tmp/table.csv
cargo run --bin datafusion-cli

Then run :

> CREATE EXTERNAL TABLE t(a int, b varchar, c float)
STORED AS CSV
LOCATION '/tmp/table.csv';
0 rows in set. Query took 0 seconds.
> select table_name, column_name, ordinal_position, is_nullable, data_type from information_schema.columns;
+------------+-------------+------------------+-------------+-----------+
| table_name | column_name | ordinal_position | is_nullable | data_type |
+------------+-------------+------------------+-------------+-----------+
| t | a | 0 | NO | Int32 |
| t | b | 1 | NO | Utf8 |
| t | c | 2 | NO | Float32 |
+------------+-------------+------------------+-------------+-----------+
3 row in set. Query took 0 seconds.

github-actions · 2021-03-30T01:45:51Z

https://issues.apache.org/jira/browse/ARROW-12107

andygrove

I'm excited to see this. Thanks @alamb

I had a proof-of-concept JDBC driver working against DataFusion at one point. One day I will find time to work on this again. With the information schema support it will really help with making this work with BI tools.

alamb · 2021-03-31T17:57:38Z

FYI @returnString @Dandandan and @jorgecarleitao

jorgecarleitao

LGTM! Thanks a lot, this metadata part is really starting to shine! ✨

jorgecarleitao · 2021-03-31T18:02:10Z

Not it matters much, just more like an FYI: this can cause large binaries as every variation used is compiled individually.

I can change it into just taking &str -- though I think in the case since there is just one callsite there is likely to be just one version of the code

yeah, it was really just if you had not though about it. No need to change anything imo 👍

jorgecarleitao · 2021-03-31T18:03:52Z

it may be more idiomatic to use Into, or From.

Ok. Good idea

In 98f702da90d18d8ae855a5eca6b0d1d6c1809551

(which I now realize I put into #9866 rather than this PR 🤦 ) -- but it should get in that PR hopefully

returnString

Looks great! And selfishly, really happy to see the namespacing stuff being used to power cool new features 😄

alamb · 2021-03-31T20:49:04Z

Looks great! And selfishly, really happy to see the namespacing stuff being used to power cool new features 😄

@returnString -- the namespacing stuff is pretty cool. I didn't realize how much easier it would make my life when it came in!

…ema.columns

# Rationale Accessing the list of columns via `select * from information_schema.columns` (introduced in #9840) is a lot to type See the doc for background: https://docs.google.com/document/d/12cpZUSNPqVH9Z0BBx6O8REu7TFqL-NPPAYCUPpDls1k/edit# This is a sister PR to `SHOW TABLES` here: #9847 # Proposal Add support for `SHOW COLUMNS FROM <table>` command. Following the MySQL syntax supported by sqlparser: https://dev.mysql.com/doc/refman/8.0/en/show-columns.html # Example Use Setup: ``` echo "1,Foo,44.9" > /tmp/table.csv echo "2,Bar,22.1" >> /tmp/table.csv cargo run --bin datafusion-cli ``` Then run : ``` > CREATE EXTERNAL TABLE t(a int, b varchar, c float) STORED AS CSV LOCATION '/tmp/table.csv'; 0 rows in set. Query took 0 seconds. > show columns from t; +---------------+--------------+------------+-------------+-----------+-------------+ | table_catalog | table_schema | table_name | column_name | data_type | is_nullable | +---------------+--------------+------------+-------------+-----------+-------------+ | datafusion | public | t | a | Int32 | NO | | datafusion | public | t | b | Utf8 | NO | | datafusion | public | t | c | Float32 | NO | +---------------+--------------+------------+-------------+-----------+-------------+ 3 row in set. Query took 0 seconds. ``` # Commentary Note that the identifiers are case sensitive (which is a more general problem that affects all name resolution, not just `SHOW COLUMNS`). Ideally this should also work: ``` > show columns from T; Plan("Unknown relation for SHOW COLUMNS: T") > select * from T; Plan("Table or CTE with name \'T\' not found") ``` Closes#9866 from alamb/alamb/show_columns Authored-by: Andrew Lamb <andrew@nerdnetworks.org> Signed-off-by: Andrew Lamb <andrew@nerdnetworks.org>

alamb · 2021-03-29T19:10:45Z

Builds on the code in #9818

Rationale

Provide schema metadata access (so a user can see what columns exist and their type).

See the doc for background: https://docs.google.com/document/d/12cpZUSNPqVH9Z0BBx6O8REu7TFqL-NPPAYCUPpDls1k/edit#

I plan to add support for SHOW COLUMNS possibly as a follow on PR (though I have found out that SHOW COLUMNS and SHOW TABLES are not supported by either MySQL or by Postgres 🤔 )

Changes

I chose to add the first 15 columns from information_schema.columns You can see the full list in Postgres here and SQL Server here.

There are a bunch more columns that say "Applies to features not available in PostgreSQL" and that don't apply to DataFusion either-- since my usecase is to get the basic schema information out I chose not to add a bunch of columns that are always null at this time.

I feel the use of column builders here is somewhat awkward (as it requires many calls to unwrap). I am thinking of a follow on PR to refactor this code to use Vec<String> and Vec<u64> and then create StringArray and UInt64Array directly from them but for now I just want the functionality

Example use

Setup:

echo "1,Foo,44.9" > /tmp/table.csv
echo "2,Bar,22.1" >> /tmp/table.csv
cargo run --bin datafusion-cli

Then run :

> CREATE EXTERNAL TABLE t(a int, b varchar, c float)
STORED AS CSV
LOCATION '/tmp/table.csv';
0 rows in set. Query took 0 seconds.
> select table_name, column_name, ordinal_position, is_nullable, data_type from information_schema.columns;
+------------+-------------+------------------+-------------+-----------+
| table_name | column_name | ordinal_position | is_nullable | data_type |
+------------+-------------+------------------+-------------+-----------+
| t | a | 0 | NO | Int32 |
| t | b | 1 | NO | Utf8 |
| t | c | 2 | NO | Float32 |
+------------+-------------+------------------+-------------+-----------+
3 row in set. Query took 0 seconds.

github-actions · 2021-03-30T01:45:51Z

https://issues.apache.org/jira/browse/ARROW-12107

andygrove

I'm excited to see this. Thanks @alamb

I had a proof-of-concept JDBC driver working against DataFusion at one point. One day I will find time to work on this again. With the information schema support it will really help with making this work with BI tools.

alamb · 2021-03-31T17:57:38Z

FYI @returnString @Dandandan and @jorgecarleitao

jorgecarleitao

LGTM! Thanks a lot, this metadata part is really starting to shine! ✨

jorgecarleitao · 2021-03-31T18:02:10Z

Not it matters much, just more like an FYI: this can cause large binaries as every variation used is compiled individually.

I can change it into just taking &str -- though I think in the case since there is just one callsite there is likely to be just one version of the code

yeah, it was really just if you had not though about it. No need to change anything imo 👍

jorgecarleitao · 2021-03-31T18:03:52Z

it may be more idiomatic to use Into, or From.

Ok. Good idea

In 98f702da90d18d8ae855a5eca6b0d1d6c1809551

(which I now realize I put into #9866 rather than this PR 🤦 ) -- but it should get in that PR hopefully

returnString

Looks great! And selfishly, really happy to see the namespacing stuff being used to power cool new features 😄

alamb · 2021-03-31T20:49:04Z

Looks great! And selfishly, really happy to see the namespacing stuff being used to power cool new features 😄

@returnString -- the namespacing stuff is pretty cool. I didn't realize how much easier it would make my life when it came in!

…ema.columns

# Rationale Accessing the list of columns via `select * from information_schema.columns` (introduced in #9840) is a lot to type See the doc for background: https://docs.google.com/document/d/12cpZUSNPqVH9Z0BBx6O8REu7TFqL-NPPAYCUPpDls1k/edit# This is a sister PR to `SHOW TABLES` here: #9847 # Proposal Add support for `SHOW COLUMNS FROM <table>` command. Following the MySQL syntax supported by sqlparser: https://dev.mysql.com/doc/refman/8.0/en/show-columns.html # Example Use Setup: ``` echo "1,Foo,44.9" > /tmp/table.csv echo "2,Bar,22.1" >> /tmp/table.csv cargo run --bin datafusion-cli ``` Then run : ``` > CREATE EXTERNAL TABLE t(a int, b varchar, c float) STORED AS CSV LOCATION '/tmp/table.csv'; 0 rows in set. Query took 0 seconds. > show columns from t; +---------------+--------------+------------+-------------+-----------+-------------+ | table_catalog | table_schema | table_name | column_name | data_type | is_nullable | +---------------+--------------+------------+-------------+-----------+-------------+ | datafusion | public | t | a | Int32 | NO | | datafusion | public | t | b | Utf8 | NO | | datafusion | public | t | c | Float32 | NO | +---------------+--------------+------------+-------------+-----------+-------------+ 3 row in set. Query took 0 seconds. ``` # Commentary Note that the identifiers are case sensitive (which is a more general problem that affects all name resolution, not just `SHOW COLUMNS`). Ideally this should also work: ``` > show columns from T; Plan("Unknown relation for SHOW COLUMNS: T") > select * from T; Plan("Table or CTE with name \'T\' not found") ``` Closes#9866 from alamb/alamb/show_columns Authored-by: Andrew Lamb <andrew@nerdnetworks.org> Signed-off-by: Andrew Lamb <andrew@nerdnetworks.org>

alamb · 2021-03-29T19:10:45Z

Builds on the code in #9818

Rationale

Provide schema metadata access (so a user can see what columns exist and their type).

See the doc for background: https://docs.google.com/document/d/12cpZUSNPqVH9Z0BBx6O8REu7TFqL-NPPAYCUPpDls1k/edit#

I plan to add support for SHOW COLUMNS possibly as a follow on PR (though I have found out that SHOW COLUMNS and SHOW TABLES are not supported by either MySQL or by Postgres 🤔 )

Changes

I chose to add the first 15 columns from information_schema.columns You can see the full list in Postgres here and SQL Server here.

There are a bunch more columns that say "Applies to features not available in PostgreSQL" and that don't apply to DataFusion either-- since my usecase is to get the basic schema information out I chose not to add a bunch of columns that are always null at this time.

I feel the use of column builders here is somewhat awkward (as it requires many calls to unwrap). I am thinking of a follow on PR to refactor this code to use Vec<String> and Vec<u64> and then create StringArray and UInt64Array directly from them but for now I just want the functionality

Example use

Setup:

echo "1,Foo,44.9" > /tmp/table.csv
echo "2,Bar,22.1" >> /tmp/table.csv
cargo run --bin datafusion-cli

Then run :

> CREATE EXTERNAL TABLE t(a int, b varchar, c float)
STORED AS CSV
LOCATION '/tmp/table.csv';
0 rows in set. Query took 0 seconds.
> select table_name, column_name, ordinal_position, is_nullable, data_type from information_schema.columns;
+------------+-------------+------------------+-------------+-----------+
| table_name | column_name | ordinal_position | is_nullable | data_type |
+------------+-------------+------------------+-------------+-----------+
| t | a | 0 | NO | Int32 |
| t | b | 1 | NO | Utf8 |
| t | c | 2 | NO | Float32 |
+------------+-------------+------------------+-------------+-----------+
3 row in set. Query took 0 seconds.

github-actions · 2021-03-30T01:45:51Z

https://issues.apache.org/jira/browse/ARROW-12107

andygrove

I'm excited to see this. Thanks @alamb

I had a proof-of-concept JDBC driver working against DataFusion at one point. One day I will find time to work on this again. With the information schema support it will really help with making this work with BI tools.

alamb · 2021-03-31T17:57:38Z

FYI @returnString @Dandandan and @jorgecarleitao

jorgecarleitao

LGTM! Thanks a lot, this metadata part is really starting to shine! ✨

jorgecarleitao · 2021-03-31T18:02:10Z

Not it matters much, just more like an FYI: this can cause large binaries as every variation used is compiled individually.

I can change it into just taking &str -- though I think in the case since there is just one callsite there is likely to be just one version of the code

yeah, it was really just if you had not though about it. No need to change anything imo 👍

jorgecarleitao · 2021-03-31T18:03:52Z

it may be more idiomatic to use Into, or From.

Ok. Good idea

In 98f702da90d18d8ae855a5eca6b0d1d6c1809551

(which I now realize I put into #9866 rather than this PR 🤦 ) -- but it should get in that PR hopefully

returnString

Looks great! And selfishly, really happy to see the namespacing stuff being used to power cool new features 😄

alamb · 2021-03-31T20:49:04Z

Looks great! And selfishly, really happy to see the namespacing stuff being used to power cool new features 😄

@returnString -- the namespacing stuff is pretty cool. I didn't realize how much easier it would make my life when it came in!

…ema.columns

# Rationale Accessing the list of columns via `select * from information_schema.columns` (introduced in #9840) is a lot to type See the doc for background: https://docs.google.com/document/d/12cpZUSNPqVH9Z0BBx6O8REu7TFqL-NPPAYCUPpDls1k/edit# This is a sister PR to `SHOW TABLES` here: #9847 # Proposal Add support for `SHOW COLUMNS FROM <table>` command. Following the MySQL syntax supported by sqlparser: https://dev.mysql.com/doc/refman/8.0/en/show-columns.html # Example Use Setup: ``` echo "1,Foo,44.9" > /tmp/table.csv echo "2,Bar,22.1" >> /tmp/table.csv cargo run --bin datafusion-cli ``` Then run : ``` > CREATE EXTERNAL TABLE t(a int, b varchar, c float) STORED AS CSV LOCATION '/tmp/table.csv'; 0 rows in set. Query took 0 seconds. > show columns from t; +---------------+--------------+------------+-------------+-----------+-------------+ | table_catalog | table_schema | table_name | column_name | data_type | is_nullable | +---------------+--------------+------------+-------------+-----------+-------------+ | datafusion | public | t | a | Int32 | NO | | datafusion | public | t | b | Utf8 | NO | | datafusion | public | t | c | Float32 | NO | +---------------+--------------+------------+-------------+-----------+-------------+ 3 row in set. Query took 0 seconds. ``` # Commentary Note that the identifiers are case sensitive (which is a more general problem that affects all name resolution, not just `SHOW COLUMNS`). Ideally this should also work: ``` > show columns from T; Plan("Unknown relation for SHOW COLUMNS: T") > select * from T; Plan("Table or CTE with name \'T\' not found") ``` Closes#9866 from alamb/alamb/show_columns Authored-by: Andrew Lamb <andrew@nerdnetworks.org> Signed-off-by: Andrew Lamb <andrew@nerdnetworks.org>

alamb · 2021-03-29T19:10:45Z

Builds on the code in #9818

Rationale

Provide schema metadata access (so a user can see what columns exist and their type).

See the doc for background: https://docs.google.com/document/d/12cpZUSNPqVH9Z0BBx6O8REu7TFqL-NPPAYCUPpDls1k/edit#

I plan to add support for SHOW COLUMNS possibly as a follow on PR (though I have found out that SHOW COLUMNS and SHOW TABLES are not supported by either MySQL or by Postgres 🤔 )

Changes

I chose to add the first 15 columns from information_schema.columns You can see the full list in Postgres here and SQL Server here.

There are a bunch more columns that say "Applies to features not available in PostgreSQL" and that don't apply to DataFusion either-- since my usecase is to get the basic schema information out I chose not to add a bunch of columns that are always null at this time.

I feel the use of column builders here is somewhat awkward (as it requires many calls to unwrap). I am thinking of a follow on PR to refactor this code to use Vec<String> and Vec<u64> and then create StringArray and UInt64Array directly from them but for now I just want the functionality

Example use

Setup:

echo "1,Foo,44.9" > /tmp/table.csv
echo "2,Bar,22.1" >> /tmp/table.csv
cargo run --bin datafusion-cli

Then run :

> CREATE EXTERNAL TABLE t(a int, b varchar, c float)
STORED AS CSV
LOCATION '/tmp/table.csv';
0 rows in set. Query took 0 seconds.
> select table_name, column_name, ordinal_position, is_nullable, data_type from information_schema.columns;
+------------+-------------+------------------+-------------+-----------+
| table_name | column_name | ordinal_position | is_nullable | data_type |
+------------+-------------+------------------+-------------+-----------+
| t | a | 0 | NO | Int32 |
| t | b | 1 | NO | Utf8 |
| t | c | 2 | NO | Float32 |
+------------+-------------+------------------+-------------+-----------+
3 row in set. Query took 0 seconds.

github-actions · 2021-03-30T01:45:51Z

https://issues.apache.org/jira/browse/ARROW-12107

andygrove

I'm excited to see this. Thanks @alamb

I had a proof-of-concept JDBC driver working against DataFusion at one point. One day I will find time to work on this again. With the information schema support it will really help with making this work with BI tools.

alamb · 2021-03-31T17:57:38Z

FYI @returnString @Dandandan and @jorgecarleitao

jorgecarleitao

LGTM! Thanks a lot, this metadata part is really starting to shine! ✨

jorgecarleitao · 2021-03-31T18:02:10Z

Not it matters much, just more like an FYI: this can cause large binaries as every variation used is compiled individually.

I can change it into just taking &str -- though I think in the case since there is just one callsite there is likely to be just one version of the code

yeah, it was really just if you had not though about it. No need to change anything imo 👍

jorgecarleitao · 2021-03-31T18:03:52Z

it may be more idiomatic to use Into, or From.

Ok. Good idea

In 98f702da90d18d8ae855a5eca6b0d1d6c1809551

(which I now realize I put into #9866 rather than this PR 🤦 ) -- but it should get in that PR hopefully

returnString

Looks great! And selfishly, really happy to see the namespacing stuff being used to power cool new features 😄

alamb · 2021-03-31T20:49:04Z

Looks great! And selfishly, really happy to see the namespacing stuff being used to power cool new features 😄

@returnString -- the namespacing stuff is pretty cool. I didn't realize how much easier it would make my life when it came in!

…ema.columns

# Rationale Accessing the list of columns via `select * from information_schema.columns` (introduced in #9840) is a lot to type See the doc for background: https://docs.google.com/document/d/12cpZUSNPqVH9Z0BBx6O8REu7TFqL-NPPAYCUPpDls1k/edit# This is a sister PR to `SHOW TABLES` here: #9847 # Proposal Add support for `SHOW COLUMNS FROM <table>` command. Following the MySQL syntax supported by sqlparser: https://dev.mysql.com/doc/refman/8.0/en/show-columns.html # Example Use Setup: ``` echo "1,Foo,44.9" > /tmp/table.csv echo "2,Bar,22.1" >> /tmp/table.csv cargo run --bin datafusion-cli ``` Then run : ``` > CREATE EXTERNAL TABLE t(a int, b varchar, c float) STORED AS CSV LOCATION '/tmp/table.csv'; 0 rows in set. Query took 0 seconds. > show columns from t; +---------------+--------------+------------+-------------+-----------+-------------+ | table_catalog | table_schema | table_name | column_name | data_type | is_nullable | +---------------+--------------+------------+-------------+-----------+-------------+ | datafusion | public | t | a | Int32 | NO | | datafusion | public | t | b | Utf8 | NO | | datafusion | public | t | c | Float32 | NO | +---------------+--------------+------------+-------------+-----------+-------------+ 3 row in set. Query took 0 seconds. ``` # Commentary Note that the identifiers are case sensitive (which is a more general problem that affects all name resolution, not just `SHOW COLUMNS`). Ideally this should also work: ``` > show columns from T; Plan("Unknown relation for SHOW COLUMNS: T") > select * from T; Plan("Table or CTE with name \'T\' not found") ``` Closes#9866 from alamb/alamb/show_columns Authored-by: Andrew Lamb <andrew@nerdnetworks.org> Signed-off-by: Andrew Lamb <andrew@nerdnetworks.org>

github-actionsBot added Component: Rust - DataFusion Component: Rust labels Mar 30, 2021

alamb changed the title ~~ARROW-12107: [Rust][DataFusion] Support SELECT * from information_schema.columns~~ARROW-12107: [Rust][DataFusion] Support SELECT * from information_schema.columnsMar 30, 2021

alamb force-pushed the alamn/information_schema_columns branch from ad1935d to 0a3663bCompare March 30, 2021 11:58

alamb marked this pull request as ready for review March 30, 2021 11:58

andygrove approved these changes Mar 31, 2021
View reviewed changes

alamb force-pushed the alamn/information_schema_columns branch from 0a3663b to ec17d43Compare March 31, 2021 17:57

jorgecarleitao approved these changes Mar 31, 2021
View reviewed changes

returnString approved these changes Mar 31, 2021
View reviewed changes

alamb mentioned this pull request Mar 31, 2021
ARROW-12109: [Rust][DataFusion] Implement SHOW COLUMNS #9866
Closed

alamb added 5 commits March 31, 2021 17:02

ARROW-12171: clean up clippy lints
3b06907

ARROW-12107: [Rust][DataFusion] Support SELECT * from information_sch…
2b9e073
…ema.columns

Update docs
981fc3a

fixup
f1d3294

clippy
c18f0ae

alamb force-pushed the alamn/information_schema_columns branch from ec17d43 to c18f0aeCompare March 31, 2021 21:07

github-actionsBot added the Component: Parquet label Apr 1, 2021

alamb closed this in caa8950Apr 1, 2021

alamb deleted the alamn/information_schema_columns branch April 1, 2021 12:00

asfimport mentioned this pull request Apr 1, 2021
[Rust][DataFusion] Support SELECT * from information_schema.columns#27930
Closed