From a204f31fac9fcb4ffae3edc0e139419d1b808ff7 Mon Sep 17 00:00:00 2001 From: Kyle Bendickson Date: Tue, 12 Oct 2021 10:48:51 -0700 Subject: [PATCH 1/4] Spec - Add -1 to data_file.record_count to indicate unknown record count, such as files imported from Avro tables --- site/docs/spec.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/docs/spec.md b/site/docs/spec.md index 5627df919572..6d8c224c96d1 100644 --- a/site/docs/spec.md +++ b/site/docs/spec.md @@ -379,7 +379,7 @@ The schema of a manifest file is a struct called `manifest_entry` with the follo | _required_ | _required_ | **`100 file_path`** | `string` | Full URI for the file with FS scheme | | _required_ | _required_ | **`101 file_format`** | `string` | String file format name, avro, orc or parquet | | _required_ | _required_ | **`102 partition`** | `struct<...>` | Partition data tuple, schema based on the partition spec output using partition field ids for the struct field ids | -| _required_ | _required_ | **`103 record_count`** | `long` | Number of records in this file | +| _required_ | _required_ | **`103 record_count`** | `long` with special value: `-1: Record count unknown` | Number of records in this file. | | _required_ | _required_ | **`104 file_size_in_bytes`** | `long` | Total file size in bytes | | _required_ | | ~~**`105 block_size_in_bytes`**~~ | `long` | **Deprecated. Always write a default in v1. Do not write in v2.** | | _optional_ | | ~~**`106 file_ordinal`**~~ | `int` | **Deprecated. Do not write.** | From 492466222eb5d65e0c1a9c408425af4d44499b7b Mon Sep 17 00:00:00 2001 From: Kyle Bendickson Date: Tue, 12 Oct 2021 12:58:56 -0700 Subject: [PATCH 2/4] Remove added period --- site/docs/spec.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/docs/spec.md b/site/docs/spec.md index 6d8c224c96d1..14b14ce21d9c 100644 --- a/site/docs/spec.md +++ b/site/docs/spec.md @@ -379,7 +379,7 @@ The schema of a manifest file is a struct called `manifest_entry` with the follo | _required_ | _required_ | **`100 file_path`** | `string` | Full URI for the file with FS scheme | | _required_ | _required_ | **`101 file_format`** | `string` | String file format name, avro, orc or parquet | | _required_ | _required_ | **`102 partition`** | `struct<...>` | Partition data tuple, schema based on the partition spec output using partition field ids for the struct field ids | -| _required_ | _required_ | **`103 record_count`** | `long` with special value: `-1: Record count unknown` | Number of records in this file. | +| _required_ | _required_ | **`103 record_count`** | `long` with special value: `-1: Record count unknown` | Number of records in this file | | _required_ | _required_ | **`104 file_size_in_bytes`** | `long` | Total file size in bytes | | _required_ | | ~~**`105 block_size_in_bytes`**~~ | `long` | **Deprecated. Always write a default in v1. Do not write in v2.** | | _optional_ | | ~~**`106 file_ordinal`**~~ | `int` | **Deprecated. Do not write.** | From fd501ab1a3c351e73168c8ce69fb9417a3b49751 Mon Sep 17 00:00:00 2001 From: Kyle Bendickson Date: Tue, 12 Oct 2021 13:06:22 -0700 Subject: [PATCH 3/4] Move -1 to be in description and match the language of row-oriented format (Avro) used a few lines down --- site/docs/spec.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/docs/spec.md b/site/docs/spec.md index 14b14ce21d9c..34e9fcef6166 100644 --- a/site/docs/spec.md +++ b/site/docs/spec.md @@ -379,7 +379,7 @@ The schema of a manifest file is a struct called `manifest_entry` with the follo | _required_ | _required_ | **`100 file_path`** | `string` | Full URI for the file with FS scheme | | _required_ | _required_ | **`101 file_format`** | `string` | String file format name, avro, orc or parquet | | _required_ | _required_ | **`102 partition`** | `struct<...>` | Partition data tuple, schema based on the partition spec output using partition field ids for the struct field ids | -| _required_ | _required_ | **`103 record_count`** | `long` with special value: `-1: Record count unknown` | Number of records in this file | +| _required_ | _required_ | **`103 record_count`** | `long` | Number of records in this file. -1 if the count is unknown, for example when importing files directly from row-oriented formats (Avro) | | _required_ | _required_ | **`104 file_size_in_bytes`** | `long` | Total file size in bytes | | _required_ | | ~~**`105 block_size_in_bytes`**~~ | `long` | **Deprecated. Always write a default in v1. Do not write in v2.** | | _optional_ | | ~~**`106 file_ordinal`**~~ | `int` | **Deprecated. Do not write.** | From 20b9cc3f61f3f08f2e1aa2f27fcaed7d3d8212be Mon Sep 17 00:00:00 2001 From: Kyle Bendickson Date: Wed, 13 Oct 2021 14:52:57 -0700 Subject: [PATCH 4/4] Simply mention that -1 is unknown without mentioning details --- site/docs/spec.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/site/docs/spec.md b/site/docs/spec.md index 34e9fcef6166..a0417c41b14d 100644 --- a/site/docs/spec.md +++ b/site/docs/spec.md @@ -379,7 +379,7 @@ The schema of a manifest file is a struct called `manifest_entry` with the follo | _required_ | _required_ | **`100 file_path`** | `string` | Full URI for the file with FS scheme | | _required_ | _required_ | **`101 file_format`** | `string` | String file format name, avro, orc or parquet | | _required_ | _required_ | **`102 partition`** | `struct<...>` | Partition data tuple, schema based on the partition spec output using partition field ids for the struct field ids | -| _required_ | _required_ | **`103 record_count`** | `long` | Number of records in this file. -1 if the count is unknown, for example when importing files directly from row-oriented formats (Avro) | +| _required_ | _required_ | **`103 record_count`** | `long` | Number of records in this file. -1 if the count is unknown | | _required_ | _required_ | **`104 file_size_in_bytes`** | `long` | Total file size in bytes | | _required_ | | ~~**`105 block_size_in_bytes`**~~ | `long` | **Deprecated. Always write a default in v1. Do not write in v2.** | | _optional_ | | ~~**`106 file_ordinal`**~~ | `int` | **Deprecated. Do not write.** |