Doc: Update Files metadata table

[szehon-ho]

Files metadata table was missing some columns, and also missing an example for partitioned tables in which the new partition column is added (users may want to know how to check what data files they have for a given partition)

[kbendick]

cc @samredai as the docs refactor is underway 👍

[rdblue]

Is it possible to not revert the formatting changes? I think it is less readable with the initial space removed.

Maybe we should replace this with a real HTML table?

Here's a snippet of code that we use in our notebooks to format PySpark dataframes as nicer tables:

fromprettytableimportPrettyTablefromIPython.core.magicimportregister_line_cell_magicclassDFTable(PrettyTable):
def__repr__(self):
returnself.get_string()
def_repr_html_(self):
returnself.get_html_string()
defto_table(df, num_rows=100):
cols=df.columnst=DFTable()
t.field_names=colst.align="r"forrowindf.limit(num_rows).collect():
d=row.asDict()
t.add_row([ d[col] forcolincols ])
returnt

That will produce both HTML and text tables that have reasonable formatting.

[szehon-ho]

Makes sense, let me look at this.

[kbendick]

Somewhat unrelated, but I noticed that lower_bounds has no value for some keys. Is this possibly confusing for readers of the docs?

[1 -> , 2 -> c]

Notice that 1 doesn't point to anything. Is this intentional and do we think this might confuse readers? I can open a separate issue if so.

[rdblue]

@kbendick, I think that we should convert lower and upper bounds into human-readable strings. Right now, I think we pass them to Spark as a map of id to binary.

[szehon-ho]

@rdblue@kbendick@samredai i made a markdown table for files, if you guys click 'View File' in github it should show what it looks like. If you want i can extend this to the other ones too, or do it separately.

By the way i also found a possible problem in the spec_id column added in #3015 , it should probably get hidden just like partition column if table is unpartitioned, I could fix that in another PR.

[samredai]

GitHub injects some overflow css to add the scroll bar, if we switch to markdown tables we'd have to add that as well. Here's how the table currently would look:

But adding the following:

extra.css

.markdown-table-container {
width:780px;
overflow-x: scroll;
}

and then surrounding your markdown table with:

<divclass="markdown-table-container" markdown="block">
...markdown table here...
</div>

will make it look like this with the scrollbar:

[rdblue]

+1 to adding the scroll bar. Thanks @samredai!

[rdblue]

The markdown table looks good to me. The only problem is that now just this table is markdown. Anyone want to follow up with an update for the other tables?

[KnightChess]

The markdown table looks good to me. The only problem is that now just this table is markdown. Anyone want to follow up with an update for the other tables?

@rdblue I can update other tables in this pr #3482

[szehon-ho]

Thanks @samredai and @rdblue for review , added the scroll bar using div and css as suggested. Attached screenshot

[KnightChess]

use scroll bar, there will be a empty line under the form when there is not overflow. did I not use it or optimize the scroll bar @szehon-ho

[szehon-ho]

@KnightChess I'm not sure to be honest, can you put up your change on your pr? We can continue discussion on that pr instead of this one?

[samredai]

@szehon-ho can you update the CSS to the below? If we're using this for all markdown tables, auto will hide the scrollbar space that @KnightChess is seeing when the table width is below the container width.

.markdown-table-container {
width:780px;
overflow-x: auto;
}

[szehon-ho]

Done, thanks @KnightChess for finding the issue that would occur for the other table, and @samredai for figuring it out

[szehon-ho]

Hey @samredai@rdblue , would we able to continue moving this forward? Was out on vacation, but my understanding is that we still keep the markdown files in this repo?

[samredai]

@szehon-ho yes the plan is to keep the markdown files in this repo. There are some changes coming soon but any merged docs PR will be included. This looks good to me, I'll let @rdblue comment on if it's good to merge.

[rdblue]

Merged. Thanks, @szehon-ho!

[szehon-ho]

Thanks for fast response !