spark: supplement spark-queries.md doc by KnightChess · Pull Request #3482 · apache/iceberg

KnightChess · 2021-11-06T14:10:20Z

#3465
@jackye1995

jackye1995 · 2021-11-06T18:29:07Z

+### All Data Files
+
+To show a table's valid data files and each file's metadata, run:
+> tip: a valid data file is one that is readable from any snapshot currently tracked by the table.


notes in the doc are in the form of !!! Note, example: https://github.com/apache/iceberg/blob/master/site/docs/spark-writes.md?plain=1#L298-L301
Also we should add the warning that this table may return duplicate rows.

jackye1995 · 2021-11-06T18:29:51Z

+### All Manifests
+
+To show a table's valid file manifests and each file's metadata, run:
+> tip: a valid manifest file is one that is referenced from any snapshot currently tracked by the table.


similar comment as above, this table may return duplicate rows.

jackye1995 · 2021-11-06T18:30:21Z

+### All Entries
+
+To show a table's valid manifest entries as rows, for both delete and data files, run:
+> tip: a valid manifest file is one that is referenced from any snapshot currently tracked by the table.


similar comment as above, this table may return duplicate rows.

KnightChess · 2021-11-07T02:49:30Z

I have modify it

rdblue · 2021-11-07T18:22:11Z

 ```

-Note: 
+!!! Note


Doesn't this create an empty note?

yeah, I will resole it like this:

I don't think this should be in a note callout block. The previous formatting was just a paragraph, "Note:" with a list is better.

@KnightChess, can you fix this?

rdblue · 2021-11-07T18:22:45Z

+ a valid manifest file is one that is referenced from any snapshot currently tracked by the table.
+
+!!! WARNING
+ this table may return **duplicate** rows


Documentation should use correct capitalization. In this case, sentences should start with a capital letter.

Looks like there are several cases of sentences that don't start with capitals, so be sure to fix all of them, please!

rdblue · 2021-11-07T18:24:20Z

+----------------+------------+----------+
+```
+
+### Entries


While we support this, I'm not sure that we want this in the standard documentation for Spark users. The entries table will show all entries in manifests, for both deleted and live files in the table. I think interpreting this table is more of an advanced use.
What do you think @samredai?

+1 I totally agree. I can't see a reason for a typical user to use this table and having it in this section of the docs could be a signal that they should be using it. There's also an instinct of data engineers coming from Hive tables to want to find the location of the data files and start messing around there so it's beneficial if we strengthen that abstraction by example in the docs (maybe even explicitly call it out)

Sorry I am late. I personally use this and have pointed others to build on this, because it's currently the only way to track when is the last time data is added, or how much data is added, to a partition.
But that being said, do understand it's a bit too complicated to document in the standard guide here, though just wondering what way advanced users can ever figure this out.

rdblue · 2021-11-07T18:26:03Z

 +-------------------------------------------------------------------------+-------------+--------------+--------------------+--------------------+------------------+-------------------+------------------+-----------------+-----------------+--------------+---------------+
 ```

+### All Data Files


I think it makes sense to have a separate section for the "all" tables. These aren't needed nearly as often and require warnings, as you have in these sections.

rdblue · 2021-11-07T18:26:47Z


+### All Data Files
+
+To show a table's valid data files and each file's metadata, run:


I don't think it is a good idea to end a paragraph with "run:" and then put two callouts between it and the SQL that it is referencing.

rdblue · 2021-11-07T18:28:18Z

+| 0|s3:.../dt=20210102/xxx.parquet| PARQUET|{20210102}| 14| 2444|{1 -> 94, 2 -> 17}|{1 -> 14, 2 -> 14}| {1 -> 0, 2 -> 0}| {}|{1 -> 1, 2 -> 20210102}|{1 -> 2, 2 -> 20210102}| null| [4]| null| 0|
+| 0|s3:.../dt=20210103/xxx.parquet| PARQUET|{20210103}| 14| 2444|{1 -> 94, 2 -> 17}|{1 -> 14, 2 -> 14}| {1 -> 0, 2 -> 0}| {}|{1 -> 1, 2 -> 20210103}|{1 -> 3, 2 -> 20210103}| null| [4]| null| 0|
+| 0|s3:.../dt=20210104/xxx.parquet| PARQUET|{20210104}| 14| 2444|{1 -> 94, 2 -> 17}|{1 -> 14, 2 -> 14}| {1 -> 0, 2 -> 0}| {}|{1 -> 1, 2 -> 20210104}|{1 -> 3, 2 -> 20210104}| null| [4]| null| 0|
+-------+------------------------------+-----------+----------+------------+------------------+------------------+------------------+------------------+----------------+-----------------------+-----------------------+------------+-------------+------------+-------------+


Can you fix the formatting of this table? You can see my recommendations on how to get a nicely formatted table here: #3422 (comment)

KnightChess · 2021-11-08T12:42:35Z

@rdblue

I have update all table format
use three-level title all metadata to split it
other fix

rdblue · 2021-11-08T16:30:29Z

+ The metadata is readable from any snapshot currently tracked by the table
+
+!!! WARNING
+ The table may return **duplicate** rows


What is "the table"?

my mistake, the table's metadata

KnightChess · 2021-11-10T05:13:52Z

 SELECT * FROM prod.db.table.history
 ```
-```text
-+-------------------------+---------------------+---------------------+---------------------+
-| made_current_at | snapshot_id | parent_id | is_current_ancestor |
-+-------------------------+---------------------+---------------------+---------------------+
+
+<div class="markdown-table-container" markdown="block">
+| made_current_at | snapshot_id | parent_id | is_current_ancestor |
+| -- | -- | -- | -- |
 | 2019-02-08 03:29:51.215 | 5781947118336215154 | NULL | true |
 | 2019-02-08 03:47:55.948 | 5179299526185056830 | 5781947118336215154 | true |
 | 2019-02-09 16:24:30.13 | 296410040247533544 | 5179299526185056830 | false |
 | 2019-02-09 16:32:47.336 | 2999875608062437330 | 5179299526185056830 | true |
 | 2019-02-09 19:42:03.919 | 8924558786060583479 | 2999875608062437330 | true |
 | 2019-02-09 19:49:16.343 | 6536733823181975045 | 8924558786060583479 | true |
-+-------------------------+---------------------+---------------------+---------------------+
-```
+</div>


@szehon-ho here, and I use the css test in my local branch
.markdown-table-container { width:780px; overflow-x: scroll; }

Thanks for pointing this out @KnightChess ! If we're going to use the container class for all markdown tables, we can use overflow-x: auto; instead and it will only show the bar when there's overflow.

To prevent a merge conflict, I'll mention this over in PR #3422 so the css can be updated there.

jackye1995 · 2021-11-22T21:06:54Z

+</div>
+
+!!! Note
+ Diffrent from metadata table [files](#files), This table exposes internal details, like files that have been deleted.


nit: this table (lower case)

jackye1995 · 2021-11-22T21:08:22Z

+</div>
+
+!!! Note
+ Like metadata table [entries](#entries), contain delete data files too, different from [all_data_files](#all-data-files)


this sentence sounds a bit awkward, what about:
Like metadata table [entries](#entries), this metadata table is different from [all_data_files](#all-data-files) and contains deleted data files.

jackye1995 · 2021-12-06T21:17:59Z

@KnightChess any updates?

KnightChess · 2021-12-07T01:28:45Z

@KnightChess any updates?

sorry, my fault. There was no change in the email I received so I did not confirm on github. I will do it today

KnightChess · 2021-12-07T01:31:04Z

@jackye1995 cc

jackye1995

thanks for the quick update, looks good to me

rdblue · 2021-12-09T00:26:58Z

+!!! Note
+ Like metadata table [entries](#entries), this metadata table is different from [all_data_files](#all-data-files) and contains deleted data files.
+
+!!! WARNING


This should be "Warning" rather than all caps.

KnightChess · 2021-12-24T08:21:36Z

update and fix conflict from #3422 , the style for markdown table look good.
cc: @rdblue @samredai

rdblue · 2021-12-29T21:58:48Z

+### All Metadata
+
+!!! Note
+ The metadata is readable from any snapshot currently tracked by the table


This doesn't make sense to me.

rdblue · 2021-12-29T22:00:01Z

+| {20211002, 10}| 1| 1|
+</div>
+
+### All Metadata


I think this should be "All Metadata Tables" (note plural tables)
And this should have a paragraph explaining that these tables are unions of the metadata tables that are specific to a snapshot.

szehon-ho · 2022-03-18T07:27:14Z

+----------------+------------+----------+
+```
+
+### Entries


Sorry I am late. I personally use this and have pointed others to build on this, because it's currently the only way to track when is the last time data is added, or how much data is added, to a partition.
But that being said, do understand it's a bit too complicated to document in the standard guide here, though just wondering what way advanced users can ever figure this out.

szehon-ho

Hi @KnightChess , if you still have time, could you rebase and address the comments? I would like to move forward on it, I think it's a valuable change.

ajantha-bhat · 2022-03-24T03:01:27Z

Hi @KnightChess , if you still have time, could you rebase and address the comments? I would like to move forward on it, I think it's a valuable change.

Agree, I thought of adding this doc few months ago, but because this PR was already there, I was waiting for it to getting merged.

@KnightChess : If you don't have time, you can let us know. I can add you as a coauthor and take this PR forward.

KnightChess · 2022-03-24T03:18:36Z

@szehon-ho @ajantha-bhat sorry for the late reply, I will take this PR forward in the next few days.

ajantha-bhat · 2022-03-24T03:40:56Z

@szehon-ho @ajantha-bhat sorry for the late reply, I will take this PR forward in the next few days.

Awesome. Looking forward to it.

szehon-ho

Thanks @KnightChess for continuing it, replied to some of the comments, and look forward to the changes. I'll be out next week, but can look if i have time, or after that.

KnightChess · 2022-03-26T10:02:30Z

@szehon-ho update other except Advanced Usage, and adapt new doc styple for hugo

szehon-ho

Sorry for delay, I just got back from vacation. Thanks @KnightChess for rebase/fix, and others for confirming the earlier comment.

I made another round of review and some smaller comments, can you please take a look? The change is getting close, and should be good soon. Thanks again.

szehon-ho

Thanks @KnightChess I built and found just one broken link in the new doc, should be good after this.

szehon-ho · 2022-04-08T16:55:53Z

Merged, thanks @KnightChess for your patience and contribution

KnightChess · 2022-04-09T00:45:22Z

thanks @szehon-ho @rdblue @jackye1995 @samredai a lot for instruct patiently

samredai · 2022-04-09T17:15:33Z

Thanks @KnightChess!

github-actionsBot added the docs label Nov 6, 2021

jackye1995 reviewed Nov 6, 2021
View reviewed changes

rdblue reviewed Nov 7, 2021
View reviewed changes

KnightChess mentioned this pull request Nov 8, 2021
Doc: Update Files metadata table #3422
Merged

rdblue reviewed Nov 8, 2021
View reviewed changes

KnightChess force-pushed the doc-for-system-tables branch from b5e6fc2 to 369f55cCompare November 10, 2021 05:04

KnightChess commented Nov 10, 2021
View reviewed changes

KnightChess commented Nov 11, 2021
View reviewed changes

Comment threadsite/docs/spark-queries.md Outdated

KnightChess force-pushed the doc-for-system-tables branch from 19e3035 to f0b198cCompare November 17, 2021 02:46

jackye1995 added this to the Iceberg 0.13.0 Release milestone Nov 18, 2021

jackye1995 reviewed Nov 22, 2021
View reviewed changes

jackye1995 approved these changes Dec 7, 2021
View reviewed changes

jackye1995 requested review from rdblue and samredai December 7, 2021 02:34

rdblue reviewed Dec 9, 2021
View reviewed changes

Comment threadsite/docs/spark-queries.md Outdated

rdblue reviewed Dec 9, 2021
View reviewed changes

Comment threadsite/docs/spark-queries.md Outdated

KnightChess force-pushed the doc-for-system-tables branch from 3814a2b to aa6570bCompare December 24, 2021 08:12

samredai reviewed Dec 28, 2021
View reviewed changes

Comment threadsite/docs/spark-queries.md Outdated

rdblue reviewed Dec 29, 2021
View reviewed changes

Comment threadsite/docs/spark-queries.md Outdated

szehon-ho reviewed Mar 18, 2022
View reviewed changes

spark: supplement spark-queries.md doc
2309c74

szehon-ho reviewed Mar 25, 2022
View reviewed changes

Comment threadsite/docs/spark-queries.md Outdated
Comment threadsite/docs/spark-queries.md Outdated

spark: update spark-queries according to advice
998f271

KnightChess force-pushed the doc-for-system-tables branch from b821ae0 to 998f271Compare March 26, 2022 09:57

spark: spark-queries remove advanced usage
c3f0555

szehon-ho reviewed Apr 4, 2022
View reviewed changes

KnightChess added 2 commits April 7, 2022 10:02

spark: update nit
3e36a56

spark: update nit
be553ce

szehon-ho reviewed Apr 7, 2022
View reviewed changes

Comment threaddocs/spark/spark-queries.md Outdated

spark: update nit
fab949a

szehon-ho approved these changes Apr 8, 2022
View reviewed changes

szehon-ho merged commit 49dad0c into apache:masterApr 8, 2022


		### All Data Files

		To show a table's valid data files and each file's metadata, run:

Uh oh!

Conversation

KnightChess commented Nov 6, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

KnightChess commented Nov 7, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

KnightChess commented Nov 8, 2021

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

samredaiNov 10, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jackye1995Nov 22, 2021• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jackye1995 commented Dec 6, 2021

Uh oh!

KnightChess commented Dec 7, 2021

Uh oh!

KnightChess commented Dec 7, 2021

Uh oh!

jackye1995 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

KnightChess commented Dec 24, 2021

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

samredaiNov 10, 2021•
edited
Loading

jackye1995Nov 22, 2021•
edited
Loading

szehon-ho left a comment •
edited
Loading

szehon-ho left a comment •
edited
Loading