Python: Reassign schema/partition-spec/sort-order ids by Fokko · Pull Request #5627 · apache/iceberg

Fokko · 2022-08-24T12:13:54Z

When creating a new schema.

Also created a type alias called TableMetadata that replaces the Union[TableMetadataV1, TableMetadataV2] annotation.

Resolves#5468

When creating a new schema Resolvesapache#5468

rdblue · 2022-08-24T18:50:02Z

+def assign_fresh_partition_spec_ids(spec: PartitionSpec, schema: Schema) -> PartitionSpec:
+ partition_fields = []
+ for pos, field in enumerate(spec.fields):
+ schema_field = schema.find_field(field.name)


This is the partition field name, not a schema field name. The schema field must be looked up by source_id. This method needs both the original schema and the fresh schema. The original schema is used to get field names and then the fresh schema is used to look up the new source ID.

Great catch! 👍🏻

@Fokko, looks like this hasn't been fixed yet, so I'm reopening the thread.

Sorry, that slipped through somehow

…esh-ids-when-creating-a-table

rdblue · 2022-08-28T22:04:11Z

 """Visit a PrimitiveType"""


+class PreOrderSchemaVisitor(Generic[T], ABC):


Is this pre-order? In Java we called it CustomOrder because you can choose when to visit children by accessing the callable.

It is pre-order traversal since we start at the root and then move to the leaves. In order is a bit less intuitive since it is not a binary tree. You could also do a reverse in-order, but not sure if we need that. We can also call it CustomOrder if you have a strong preference, but I think pre-order is the most logical way of using this visitor.

rdblue · 2022-08-28T22:08:33Z

+ return next(self.counter)
+
+ def schema(self, schema: Schema, struct_result: Callable[[], StructType]) -> Schema:
+ return Schema(*struct_result().fields, identifier_field_ids=schema.identifier_field_ids)


Shouldn't this re-map the identifier field IDs since it is returning a new schema?

Yes, we do that in the function itself:
defassign_fresh_schema_ids(schema: Schema) ->Schema: """Traverses the schema, and sets new IDs"""schema_struct=pre_order_visit(schema.as_struct(), _SetFreshIDs()) fresh_identifier_field_ids= [] new_schema=Schema(*schema_struct.fields) forfield_idinschema.identifier_field_ids: original_field_name=schema.find_column_name(field_id) iforiginal_field_nameisNone: raiseValueError(f"Could not find field: {field_id}") fresh_field=new_schema.find_field(original_field_name) iffresh_fieldisNone: raiseValueError(f"Could not lookup field in new schema: {original_field_name}") fresh_identifier_field_ids.append(fresh_field.field_id) returnnew_schema.copy(update={"identifier_field_ids": fresh_identifier_field_ids})
This is because we first want to know all the IDs

Ah, I've refactored this because we need to build a map anyway 👍🏻

rdblue · 2022-08-28T22:11:03Z

+
+ def field(self, field: NestedField, field_result: Callable[[], IcebergType]) -> IcebergType:
+ return NestedField(
+ field_id=self._get_and_increment(), name=field.name, field_type=field_result(), required=field.required, doc=field.doc


This is going to visit children before visiting the next field. If you're trying to match the behavior of assignment in Java, you'd need to increment the counter for each field and then visit children.

Missed that one, thanks! Just updated the code and tests

rdblue · 2022-08-28T22:13:11Z

+) -> TableMetadata:
+ fresh_schema = assign_fresh_schema_ids(schema)
+ fresh_partition_spec = assign_fresh_partition_spec_ids(partition_spec, fresh_schema)
+ fresh_sort_order = assign_fresh_sort_order_ids(sort_order, schema, fresh_schema)


Do the "fresh" methods always reset schema_id, spec_id, and order_id?

Only when you create TableMetadata out of it (when creating a new table). And it resets if it isn't 1.

…esh-ids-when-creating-a-table

Fokko

I'll move this forward. Let me know if there is anything that you would like to see changed. There are two followups that I'd like to do:

Remove the pre-validators because they are confusing and error prone
Smooth out the API for the docs

Fokko · 2022-08-24T12:13:54Z

When creating a new schema.

Also created a type alias called TableMetadata that replaces the Union[TableMetadataV1, TableMetadataV2] annotation.

Resolves#5468

When creating a new schema Resolvesapache#5468

rdblue · 2022-08-24T18:50:02Z

+def assign_fresh_partition_spec_ids(spec: PartitionSpec, schema: Schema) -> PartitionSpec:
+ partition_fields = []
+ for pos, field in enumerate(spec.fields):
+ schema_field = schema.find_field(field.name)


This is the partition field name, not a schema field name. The schema field must be looked up by source_id. This method needs both the original schema and the fresh schema. The original schema is used to get field names and then the fresh schema is used to look up the new source ID.

Great catch! 👍🏻

@Fokko, looks like this hasn't been fixed yet, so I'm reopening the thread.

Sorry, that slipped through somehow

…esh-ids-when-creating-a-table

rdblue · 2022-08-28T22:04:11Z

 """Visit a PrimitiveType"""


+class PreOrderSchemaVisitor(Generic[T], ABC):


Is this pre-order? In Java we called it CustomOrder because you can choose when to visit children by accessing the callable.

It is pre-order traversal since we start at the root and then move to the leaves. In order is a bit less intuitive since it is not a binary tree. You could also do a reverse in-order, but not sure if we need that. We can also call it CustomOrder if you have a strong preference, but I think pre-order is the most logical way of using this visitor.

rdblue · 2022-08-28T22:08:33Z

+ return next(self.counter)
+
+ def schema(self, schema: Schema, struct_result: Callable[[], StructType]) -> Schema:
+ return Schema(*struct_result().fields, identifier_field_ids=schema.identifier_field_ids)


Shouldn't this re-map the identifier field IDs since it is returning a new schema?

Yes, we do that in the function itself:
defassign_fresh_schema_ids(schema: Schema) ->Schema: """Traverses the schema, and sets new IDs"""schema_struct=pre_order_visit(schema.as_struct(), _SetFreshIDs()) fresh_identifier_field_ids= [] new_schema=Schema(*schema_struct.fields) forfield_idinschema.identifier_field_ids: original_field_name=schema.find_column_name(field_id) iforiginal_field_nameisNone: raiseValueError(f"Could not find field: {field_id}") fresh_field=new_schema.find_field(original_field_name) iffresh_fieldisNone: raiseValueError(f"Could not lookup field in new schema: {original_field_name}") fresh_identifier_field_ids.append(fresh_field.field_id) returnnew_schema.copy(update={"identifier_field_ids": fresh_identifier_field_ids})
This is because we first want to know all the IDs

Ah, I've refactored this because we need to build a map anyway 👍🏻

rdblue · 2022-08-28T22:11:03Z

+
+ def field(self, field: NestedField, field_result: Callable[[], IcebergType]) -> IcebergType:
+ return NestedField(
+ field_id=self._get_and_increment(), name=field.name, field_type=field_result(), required=field.required, doc=field.doc


This is going to visit children before visiting the next field. If you're trying to match the behavior of assignment in Java, you'd need to increment the counter for each field and then visit children.

Missed that one, thanks! Just updated the code and tests

rdblue · 2022-08-28T22:13:11Z

+) -> TableMetadata:
+ fresh_schema = assign_fresh_schema_ids(schema)
+ fresh_partition_spec = assign_fresh_partition_spec_ids(partition_spec, fresh_schema)
+ fresh_sort_order = assign_fresh_sort_order_ids(sort_order, schema, fresh_schema)


Do the "fresh" methods always reset schema_id, spec_id, and order_id?

Only when you create TableMetadata out of it (when creating a new table). And it resets if it isn't 1.

…esh-ids-when-creating-a-table

Fokko

I'll move this forward. Let me know if there is anything that you would like to see changed. There are two followups that I'd like to do:

Remove the pre-validators because they are confusing and error prone
Smooth out the API for the docs

Fokko · 2022-08-24T12:13:54Z

When creating a new schema.

Also created a type alias called TableMetadata that replaces the Union[TableMetadataV1, TableMetadataV2] annotation.

Resolves#5468

When creating a new schema Resolvesapache#5468

rdblue · 2022-08-24T18:50:02Z

+def assign_fresh_partition_spec_ids(spec: PartitionSpec, schema: Schema) -> PartitionSpec:
+ partition_fields = []
+ for pos, field in enumerate(spec.fields):
+ schema_field = schema.find_field(field.name)


This is the partition field name, not a schema field name. The schema field must be looked up by source_id. This method needs both the original schema and the fresh schema. The original schema is used to get field names and then the fresh schema is used to look up the new source ID.

Great catch! 👍🏻

@Fokko, looks like this hasn't been fixed yet, so I'm reopening the thread.

Sorry, that slipped through somehow

…esh-ids-when-creating-a-table

rdblue · 2022-08-28T22:04:11Z

 """Visit a PrimitiveType"""


+class PreOrderSchemaVisitor(Generic[T], ABC):


Is this pre-order? In Java we called it CustomOrder because you can choose when to visit children by accessing the callable.

It is pre-order traversal since we start at the root and then move to the leaves. In order is a bit less intuitive since it is not a binary tree. You could also do a reverse in-order, but not sure if we need that. We can also call it CustomOrder if you have a strong preference, but I think pre-order is the most logical way of using this visitor.

rdblue · 2022-08-28T22:08:33Z

+ return next(self.counter)
+
+ def schema(self, schema: Schema, struct_result: Callable[[], StructType]) -> Schema:
+ return Schema(*struct_result().fields, identifier_field_ids=schema.identifier_field_ids)


Shouldn't this re-map the identifier field IDs since it is returning a new schema?

Yes, we do that in the function itself:
defassign_fresh_schema_ids(schema: Schema) ->Schema: """Traverses the schema, and sets new IDs"""schema_struct=pre_order_visit(schema.as_struct(), _SetFreshIDs()) fresh_identifier_field_ids= [] new_schema=Schema(*schema_struct.fields) forfield_idinschema.identifier_field_ids: original_field_name=schema.find_column_name(field_id) iforiginal_field_nameisNone: raiseValueError(f"Could not find field: {field_id}") fresh_field=new_schema.find_field(original_field_name) iffresh_fieldisNone: raiseValueError(f"Could not lookup field in new schema: {original_field_name}") fresh_identifier_field_ids.append(fresh_field.field_id) returnnew_schema.copy(update={"identifier_field_ids": fresh_identifier_field_ids})
This is because we first want to know all the IDs

Ah, I've refactored this because we need to build a map anyway 👍🏻

rdblue · 2022-08-28T22:11:03Z

+
+ def field(self, field: NestedField, field_result: Callable[[], IcebergType]) -> IcebergType:
+ return NestedField(
+ field_id=self._get_and_increment(), name=field.name, field_type=field_result(), required=field.required, doc=field.doc


This is going to visit children before visiting the next field. If you're trying to match the behavior of assignment in Java, you'd need to increment the counter for each field and then visit children.

Missed that one, thanks! Just updated the code and tests

rdblue · 2022-08-28T22:13:11Z

+) -> TableMetadata:
+ fresh_schema = assign_fresh_schema_ids(schema)
+ fresh_partition_spec = assign_fresh_partition_spec_ids(partition_spec, fresh_schema)
+ fresh_sort_order = assign_fresh_sort_order_ids(sort_order, schema, fresh_schema)


Do the "fresh" methods always reset schema_id, spec_id, and order_id?

Only when you create TableMetadata out of it (when creating a new table). And it resets if it isn't 1.

…esh-ids-when-creating-a-table

Fokko

I'll move this forward. Let me know if there is anything that you would like to see changed. There are two followups that I'd like to do:

Remove the pre-validators because they are confusing and error prone
Smooth out the API for the docs

Fokko · 2022-08-24T12:13:54Z

When creating a new schema.

Also created a type alias called TableMetadata that replaces the Union[TableMetadataV1, TableMetadataV2] annotation.

Resolves#5468

When creating a new schema Resolvesapache#5468

rdblue · 2022-08-24T18:50:02Z

+def assign_fresh_partition_spec_ids(spec: PartitionSpec, schema: Schema) -> PartitionSpec:
+ partition_fields = []
+ for pos, field in enumerate(spec.fields):
+ schema_field = schema.find_field(field.name)


This is the partition field name, not a schema field name. The schema field must be looked up by source_id. This method needs both the original schema and the fresh schema. The original schema is used to get field names and then the fresh schema is used to look up the new source ID.

Great catch! 👍🏻

@Fokko, looks like this hasn't been fixed yet, so I'm reopening the thread.

Sorry, that slipped through somehow

…esh-ids-when-creating-a-table

rdblue · 2022-08-28T22:04:11Z

 """Visit a PrimitiveType"""


+class PreOrderSchemaVisitor(Generic[T], ABC):


Is this pre-order? In Java we called it CustomOrder because you can choose when to visit children by accessing the callable.

It is pre-order traversal since we start at the root and then move to the leaves. In order is a bit less intuitive since it is not a binary tree. You could also do a reverse in-order, but not sure if we need that. We can also call it CustomOrder if you have a strong preference, but I think pre-order is the most logical way of using this visitor.

rdblue · 2022-08-28T22:08:33Z

+ return next(self.counter)
+
+ def schema(self, schema: Schema, struct_result: Callable[[], StructType]) -> Schema:
+ return Schema(*struct_result().fields, identifier_field_ids=schema.identifier_field_ids)


Shouldn't this re-map the identifier field IDs since it is returning a new schema?

Yes, we do that in the function itself:
defassign_fresh_schema_ids(schema: Schema) ->Schema: """Traverses the schema, and sets new IDs"""schema_struct=pre_order_visit(schema.as_struct(), _SetFreshIDs()) fresh_identifier_field_ids= [] new_schema=Schema(*schema_struct.fields) forfield_idinschema.identifier_field_ids: original_field_name=schema.find_column_name(field_id) iforiginal_field_nameisNone: raiseValueError(f"Could not find field: {field_id}") fresh_field=new_schema.find_field(original_field_name) iffresh_fieldisNone: raiseValueError(f"Could not lookup field in new schema: {original_field_name}") fresh_identifier_field_ids.append(fresh_field.field_id) returnnew_schema.copy(update={"identifier_field_ids": fresh_identifier_field_ids})
This is because we first want to know all the IDs

Ah, I've refactored this because we need to build a map anyway 👍🏻

rdblue · 2022-08-28T22:11:03Z

+
+ def field(self, field: NestedField, field_result: Callable[[], IcebergType]) -> IcebergType:
+ return NestedField(
+ field_id=self._get_and_increment(), name=field.name, field_type=field_result(), required=field.required, doc=field.doc


This is going to visit children before visiting the next field. If you're trying to match the behavior of assignment in Java, you'd need to increment the counter for each field and then visit children.

Missed that one, thanks! Just updated the code and tests

rdblue · 2022-08-28T22:13:11Z

+) -> TableMetadata:
+ fresh_schema = assign_fresh_schema_ids(schema)
+ fresh_partition_spec = assign_fresh_partition_spec_ids(partition_spec, fresh_schema)
+ fresh_sort_order = assign_fresh_sort_order_ids(sort_order, schema, fresh_schema)


Do the "fresh" methods always reset schema_id, spec_id, and order_id?

Only when you create TableMetadata out of it (when creating a new table). And it resets if it isn't 1.

…esh-ids-when-creating-a-table

Fokko

I'll move this forward. Let me know if there is anything that you would like to see changed. There are two followups that I'd like to do:

Remove the pre-validators because they are confusing and error prone
Smooth out the API for the docs

Fokko · 2022-08-24T12:13:54Z

When creating a new schema.

Also created a type alias called TableMetadata that replaces the Union[TableMetadataV1, TableMetadataV2] annotation.

Resolves#5468

When creating a new schema Resolvesapache#5468

rdblue · 2022-08-24T18:50:02Z

+def assign_fresh_partition_spec_ids(spec: PartitionSpec, schema: Schema) -> PartitionSpec:
+ partition_fields = []
+ for pos, field in enumerate(spec.fields):
+ schema_field = schema.find_field(field.name)


This is the partition field name, not a schema field name. The schema field must be looked up by source_id. This method needs both the original schema and the fresh schema. The original schema is used to get field names and then the fresh schema is used to look up the new source ID.

Great catch! 👍🏻

@Fokko, looks like this hasn't been fixed yet, so I'm reopening the thread.

Sorry, that slipped through somehow

…esh-ids-when-creating-a-table

rdblue · 2022-08-28T22:04:11Z

 """Visit a PrimitiveType"""


+class PreOrderSchemaVisitor(Generic[T], ABC):


Is this pre-order? In Java we called it CustomOrder because you can choose when to visit children by accessing the callable.

It is pre-order traversal since we start at the root and then move to the leaves. In order is a bit less intuitive since it is not a binary tree. You could also do a reverse in-order, but not sure if we need that. We can also call it CustomOrder if you have a strong preference, but I think pre-order is the most logical way of using this visitor.

rdblue · 2022-08-28T22:08:33Z

+ return next(self.counter)
+
+ def schema(self, schema: Schema, struct_result: Callable[[], StructType]) -> Schema:
+ return Schema(*struct_result().fields, identifier_field_ids=schema.identifier_field_ids)


Shouldn't this re-map the identifier field IDs since it is returning a new schema?

Yes, we do that in the function itself:
defassign_fresh_schema_ids(schema: Schema) ->Schema: """Traverses the schema, and sets new IDs"""schema_struct=pre_order_visit(schema.as_struct(), _SetFreshIDs()) fresh_identifier_field_ids= [] new_schema=Schema(*schema_struct.fields) forfield_idinschema.identifier_field_ids: original_field_name=schema.find_column_name(field_id) iforiginal_field_nameisNone: raiseValueError(f"Could not find field: {field_id}") fresh_field=new_schema.find_field(original_field_name) iffresh_fieldisNone: raiseValueError(f"Could not lookup field in new schema: {original_field_name}") fresh_identifier_field_ids.append(fresh_field.field_id) returnnew_schema.copy(update={"identifier_field_ids": fresh_identifier_field_ids})
This is because we first want to know all the IDs

Ah, I've refactored this because we need to build a map anyway 👍🏻

rdblue · 2022-08-28T22:11:03Z

+
+ def field(self, field: NestedField, field_result: Callable[[], IcebergType]) -> IcebergType:
+ return NestedField(
+ field_id=self._get_and_increment(), name=field.name, field_type=field_result(), required=field.required, doc=field.doc


This is going to visit children before visiting the next field. If you're trying to match the behavior of assignment in Java, you'd need to increment the counter for each field and then visit children.

Missed that one, thanks! Just updated the code and tests

rdblue · 2022-08-28T22:13:11Z

+) -> TableMetadata:
+ fresh_schema = assign_fresh_schema_ids(schema)
+ fresh_partition_spec = assign_fresh_partition_spec_ids(partition_spec, fresh_schema)
+ fresh_sort_order = assign_fresh_sort_order_ids(sort_order, schema, fresh_schema)


Do the "fresh" methods always reset schema_id, spec_id, and order_id?

Only when you create TableMetadata out of it (when creating a new table). And it resets if it isn't 1.

…esh-ids-when-creating-a-table

Fokko

I'll move this forward. Let me know if there is anything that you would like to see changed. There are two followups that I'd like to do:

Remove the pre-validators because they are confusing and error prone
Smooth out the API for the docs

Fokko · 2022-08-24T12:13:54Z

When creating a new schema.

Also created a type alias called TableMetadata that replaces the Union[TableMetadataV1, TableMetadataV2] annotation.

Resolves#5468

When creating a new schema Resolvesapache#5468

rdblue · 2022-08-24T18:50:02Z

+def assign_fresh_partition_spec_ids(spec: PartitionSpec, schema: Schema) -> PartitionSpec:
+ partition_fields = []
+ for pos, field in enumerate(spec.fields):
+ schema_field = schema.find_field(field.name)


This is the partition field name, not a schema field name. The schema field must be looked up by source_id. This method needs both the original schema and the fresh schema. The original schema is used to get field names and then the fresh schema is used to look up the new source ID.

Great catch! 👍🏻

@Fokko, looks like this hasn't been fixed yet, so I'm reopening the thread.

Sorry, that slipped through somehow

…esh-ids-when-creating-a-table

rdblue · 2022-08-28T22:04:11Z

 """Visit a PrimitiveType"""


+class PreOrderSchemaVisitor(Generic[T], ABC):


Is this pre-order? In Java we called it CustomOrder because you can choose when to visit children by accessing the callable.

It is pre-order traversal since we start at the root and then move to the leaves. In order is a bit less intuitive since it is not a binary tree. You could also do a reverse in-order, but not sure if we need that. We can also call it CustomOrder if you have a strong preference, but I think pre-order is the most logical way of using this visitor.

rdblue · 2022-08-28T22:08:33Z

+ return next(self.counter)
+
+ def schema(self, schema: Schema, struct_result: Callable[[], StructType]) -> Schema:
+ return Schema(*struct_result().fields, identifier_field_ids=schema.identifier_field_ids)


Shouldn't this re-map the identifier field IDs since it is returning a new schema?

Yes, we do that in the function itself:
defassign_fresh_schema_ids(schema: Schema) ->Schema: """Traverses the schema, and sets new IDs"""schema_struct=pre_order_visit(schema.as_struct(), _SetFreshIDs()) fresh_identifier_field_ids= [] new_schema=Schema(*schema_struct.fields) forfield_idinschema.identifier_field_ids: original_field_name=schema.find_column_name(field_id) iforiginal_field_nameisNone: raiseValueError(f"Could not find field: {field_id}") fresh_field=new_schema.find_field(original_field_name) iffresh_fieldisNone: raiseValueError(f"Could not lookup field in new schema: {original_field_name}") fresh_identifier_field_ids.append(fresh_field.field_id) returnnew_schema.copy(update={"identifier_field_ids": fresh_identifier_field_ids})
This is because we first want to know all the IDs

Ah, I've refactored this because we need to build a map anyway 👍🏻

rdblue · 2022-08-28T22:11:03Z

+
+ def field(self, field: NestedField, field_result: Callable[[], IcebergType]) -> IcebergType:
+ return NestedField(
+ field_id=self._get_and_increment(), name=field.name, field_type=field_result(), required=field.required, doc=field.doc


This is going to visit children before visiting the next field. If you're trying to match the behavior of assignment in Java, you'd need to increment the counter for each field and then visit children.

Missed that one, thanks! Just updated the code and tests

rdblue · 2022-08-28T22:13:11Z

+) -> TableMetadata:
+ fresh_schema = assign_fresh_schema_ids(schema)
+ fresh_partition_spec = assign_fresh_partition_spec_ids(partition_spec, fresh_schema)
+ fresh_sort_order = assign_fresh_sort_order_ids(sort_order, schema, fresh_schema)


Do the "fresh" methods always reset schema_id, spec_id, and order_id?

Only when you create TableMetadata out of it (when creating a new table). And it resets if it isn't 1.

…esh-ids-when-creating-a-table

Fokko

I'll move this forward. Let me know if there is anything that you would like to see changed. There are two followups that I'd like to do:

Remove the pre-validators because they are confusing and error prone
Smooth out the API for the docs

Fokko · 2022-08-24T12:13:54Z

When creating a new schema.

Also created a type alias called TableMetadata that replaces the Union[TableMetadataV1, TableMetadataV2] annotation.

Resolves#5468

When creating a new schema Resolvesapache#5468

rdblue · 2022-08-24T18:50:02Z

+def assign_fresh_partition_spec_ids(spec: PartitionSpec, schema: Schema) -> PartitionSpec:
+ partition_fields = []
+ for pos, field in enumerate(spec.fields):
+ schema_field = schema.find_field(field.name)


This is the partition field name, not a schema field name. The schema field must be looked up by source_id. This method needs both the original schema and the fresh schema. The original schema is used to get field names and then the fresh schema is used to look up the new source ID.

Great catch! 👍🏻

@Fokko, looks like this hasn't been fixed yet, so I'm reopening the thread.

Sorry, that slipped through somehow

…esh-ids-when-creating-a-table

rdblue · 2022-08-28T22:04:11Z

 """Visit a PrimitiveType"""


+class PreOrderSchemaVisitor(Generic[T], ABC):


Is this pre-order? In Java we called it CustomOrder because you can choose when to visit children by accessing the callable.

It is pre-order traversal since we start at the root and then move to the leaves. In order is a bit less intuitive since it is not a binary tree. You could also do a reverse in-order, but not sure if we need that. We can also call it CustomOrder if you have a strong preference, but I think pre-order is the most logical way of using this visitor.

rdblue · 2022-08-28T22:08:33Z

+ return next(self.counter)
+
+ def schema(self, schema: Schema, struct_result: Callable[[], StructType]) -> Schema:
+ return Schema(*struct_result().fields, identifier_field_ids=schema.identifier_field_ids)


Shouldn't this re-map the identifier field IDs since it is returning a new schema?

Yes, we do that in the function itself:
defassign_fresh_schema_ids(schema: Schema) ->Schema: """Traverses the schema, and sets new IDs"""schema_struct=pre_order_visit(schema.as_struct(), _SetFreshIDs()) fresh_identifier_field_ids= [] new_schema=Schema(*schema_struct.fields) forfield_idinschema.identifier_field_ids: original_field_name=schema.find_column_name(field_id) iforiginal_field_nameisNone: raiseValueError(f"Could not find field: {field_id}") fresh_field=new_schema.find_field(original_field_name) iffresh_fieldisNone: raiseValueError(f"Could not lookup field in new schema: {original_field_name}") fresh_identifier_field_ids.append(fresh_field.field_id) returnnew_schema.copy(update={"identifier_field_ids": fresh_identifier_field_ids})
This is because we first want to know all the IDs

Ah, I've refactored this because we need to build a map anyway 👍🏻

rdblue · 2022-08-28T22:11:03Z

+
+ def field(self, field: NestedField, field_result: Callable[[], IcebergType]) -> IcebergType:
+ return NestedField(
+ field_id=self._get_and_increment(), name=field.name, field_type=field_result(), required=field.required, doc=field.doc


This is going to visit children before visiting the next field. If you're trying to match the behavior of assignment in Java, you'd need to increment the counter for each field and then visit children.

Missed that one, thanks! Just updated the code and tests

rdblue · 2022-08-28T22:13:11Z

+) -> TableMetadata:
+ fresh_schema = assign_fresh_schema_ids(schema)
+ fresh_partition_spec = assign_fresh_partition_spec_ids(partition_spec, fresh_schema)
+ fresh_sort_order = assign_fresh_sort_order_ids(sort_order, schema, fresh_schema)


Do the "fresh" methods always reset schema_id, spec_id, and order_id?

Only when you create TableMetadata out of it (when creating a new table). And it resets if it isn't 1.

…esh-ids-when-creating-a-table

Fokko

I'll move this forward. Let me know if there is anything that you would like to see changed. There are two followups that I'd like to do:

Remove the pre-validators because they are confusing and error prone
Smooth out the API for the docs

Fokko · 2022-08-24T12:13:54Z

When creating a new schema.

Also created a type alias called TableMetadata that replaces the Union[TableMetadataV1, TableMetadataV2] annotation.

Resolves#5468

When creating a new schema Resolvesapache#5468

rdblue · 2022-08-24T18:50:02Z

+def assign_fresh_partition_spec_ids(spec: PartitionSpec, schema: Schema) -> PartitionSpec:
+ partition_fields = []
+ for pos, field in enumerate(spec.fields):
+ schema_field = schema.find_field(field.name)


This is the partition field name, not a schema field name. The schema field must be looked up by source_id. This method needs both the original schema and the fresh schema. The original schema is used to get field names and then the fresh schema is used to look up the new source ID.

Great catch! 👍🏻

@Fokko, looks like this hasn't been fixed yet, so I'm reopening the thread.

Sorry, that slipped through somehow

…esh-ids-when-creating-a-table

rdblue · 2022-08-28T22:04:11Z

 """Visit a PrimitiveType"""


+class PreOrderSchemaVisitor(Generic[T], ABC):


Is this pre-order? In Java we called it CustomOrder because you can choose when to visit children by accessing the callable.

It is pre-order traversal since we start at the root and then move to the leaves. In order is a bit less intuitive since it is not a binary tree. You could also do a reverse in-order, but not sure if we need that. We can also call it CustomOrder if you have a strong preference, but I think pre-order is the most logical way of using this visitor.

rdblue · 2022-08-28T22:08:33Z

+ return next(self.counter)
+
+ def schema(self, schema: Schema, struct_result: Callable[[], StructType]) -> Schema:
+ return Schema(*struct_result().fields, identifier_field_ids=schema.identifier_field_ids)


Shouldn't this re-map the identifier field IDs since it is returning a new schema?

Yes, we do that in the function itself:
defassign_fresh_schema_ids(schema: Schema) ->Schema: """Traverses the schema, and sets new IDs"""schema_struct=pre_order_visit(schema.as_struct(), _SetFreshIDs()) fresh_identifier_field_ids= [] new_schema=Schema(*schema_struct.fields) forfield_idinschema.identifier_field_ids: original_field_name=schema.find_column_name(field_id) iforiginal_field_nameisNone: raiseValueError(f"Could not find field: {field_id}") fresh_field=new_schema.find_field(original_field_name) iffresh_fieldisNone: raiseValueError(f"Could not lookup field in new schema: {original_field_name}") fresh_identifier_field_ids.append(fresh_field.field_id) returnnew_schema.copy(update={"identifier_field_ids": fresh_identifier_field_ids})
This is because we first want to know all the IDs

Ah, I've refactored this because we need to build a map anyway 👍🏻

rdblue · 2022-08-28T22:11:03Z

+
+ def field(self, field: NestedField, field_result: Callable[[], IcebergType]) -> IcebergType:
+ return NestedField(
+ field_id=self._get_and_increment(), name=field.name, field_type=field_result(), required=field.required, doc=field.doc


This is going to visit children before visiting the next field. If you're trying to match the behavior of assignment in Java, you'd need to increment the counter for each field and then visit children.

Missed that one, thanks! Just updated the code and tests

rdblue · 2022-08-28T22:13:11Z

+) -> TableMetadata:
+ fresh_schema = assign_fresh_schema_ids(schema)
+ fresh_partition_spec = assign_fresh_partition_spec_ids(partition_spec, fresh_schema)
+ fresh_sort_order = assign_fresh_sort_order_ids(sort_order, schema, fresh_schema)


Do the "fresh" methods always reset schema_id, spec_id, and order_id?

Only when you create TableMetadata out of it (when creating a new table). And it resets if it isn't 1.

…esh-ids-when-creating-a-table

Fokko

I'll move this forward. Let me know if there is anything that you would like to see changed. There are two followups that I'd like to do:

Remove the pre-validators because they are confusing and error prone
Smooth out the API for the docs

Python: Reassign schema/partition-spec/sort-order ids
178659c
When creating a new schema Resolvesapache#5468

github-actionsBot added the python label Aug 24, 2022

Fokko commented Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/metadata.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/metadata.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/metadata.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/sorting.py Outdated

Fokko added 4 commits August 24, 2022 23:37

Convert into pre-order
6482984

Merge branch 'master' ofhttps://github.com/apache/iceberginto fd-fr…
2bdf233
…esh-ids-when-creating-a-table

Small docstring improvements
076b6e0

Merge branch 'master' ofhttps://github.com/apache/iceberginto fd-fr…
2df8cee
…esh-ids-when-creating-a-table

Fokko force-pushed the fd-fresh-ids-when-creating-a-table branch from dc0dbc5 to 2df8ceeCompare August 25, 2022 08:09

Fokko commented Aug 25, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/metadata.py

rdblue reviewed Aug 28, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py

rdblue reviewed Aug 28, 2022
View reviewed changes

Fokko added 2 commits August 30, 2022 20:23

Merge branch 'master' ofhttps://github.com/apache/iceberginto fd-fr…
1091859
…esh-ids-when-creating-a-table

Fix order of the structs/lists/maps
bee1c81

Fokko force-pushed the fd-fresh-ids-when-creating-a-table branch from ad95028 to bee1c81Compare August 30, 2022 19:45

Fokko mentioned this pull request Aug 30, 2022
Python: Fine-tune the API #5672
Merged

Fokko commented Aug 31, 2022
View reviewed changes

Fokko merged commit 08bb3e2 into apache:masterAug 31, 2022

Fokko deleted the fd-fresh-ids-when-creating-a-table branch August 31, 2022 17:28

HonahX mentioned this pull request Jan 29, 2024
create_table with a PyArrow Schema apache/iceberg-python#305
Merged

Python: Reassign schema/partition-spec/sort-order ids
178659c
When creating a new schema Resolvesapache#5468

github-actionsBot added the python label Aug 24, 2022

Fokko commented Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/metadata.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/metadata.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/metadata.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/sorting.py Outdated

Fokko added 4 commits August 24, 2022 23:37

Convert into pre-order
6482984

Merge branch 'master' ofhttps://github.com/apache/iceberginto fd-fr…
2bdf233
…esh-ids-when-creating-a-table

Small docstring improvements
076b6e0

Merge branch 'master' ofhttps://github.com/apache/iceberginto fd-fr…
2df8cee
…esh-ids-when-creating-a-table

Fokko force-pushed the fd-fresh-ids-when-creating-a-table branch from dc0dbc5 to 2df8ceeCompare August 25, 2022 08:09

Fokko commented Aug 25, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/metadata.py

rdblue reviewed Aug 28, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py

rdblue reviewed Aug 28, 2022
View reviewed changes

Fokko added 2 commits August 30, 2022 20:23

Merge branch 'master' ofhttps://github.com/apache/iceberginto fd-fr…
1091859
…esh-ids-when-creating-a-table

Fix order of the structs/lists/maps
bee1c81

Fokko force-pushed the fd-fresh-ids-when-creating-a-table branch from ad95028 to bee1c81Compare August 30, 2022 19:45

Fokko mentioned this pull request Aug 30, 2022
Python: Fine-tune the API #5672
Merged

Fokko commented Aug 31, 2022
View reviewed changes

Fokko merged commit 08bb3e2 into apache:masterAug 31, 2022

Fokko deleted the fd-fresh-ids-when-creating-a-table branch August 31, 2022 17:28

HonahX mentioned this pull request Jan 29, 2024
create_table with a PyArrow Schema apache/iceberg-python#305
Merged

Python: Reassign schema/partition-spec/sort-order ids
178659c
When creating a new schema Resolvesapache#5468

github-actionsBot added the python label Aug 24, 2022

Fokko commented Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/metadata.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/metadata.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/metadata.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/sorting.py Outdated

Fokko added 4 commits August 24, 2022 23:37

Convert into pre-order
6482984

Merge branch 'master' ofhttps://github.com/apache/iceberginto fd-fr…
2bdf233
…esh-ids-when-creating-a-table

Small docstring improvements
076b6e0

Merge branch 'master' ofhttps://github.com/apache/iceberginto fd-fr…
2df8cee
…esh-ids-when-creating-a-table

Fokko force-pushed the fd-fresh-ids-when-creating-a-table branch from dc0dbc5 to 2df8ceeCompare August 25, 2022 08:09

Fokko commented Aug 25, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/metadata.py

rdblue reviewed Aug 28, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py

rdblue reviewed Aug 28, 2022
View reviewed changes

Fokko added 2 commits August 30, 2022 20:23

Merge branch 'master' ofhttps://github.com/apache/iceberginto fd-fr…
1091859
…esh-ids-when-creating-a-table

Fix order of the structs/lists/maps
bee1c81

Fokko force-pushed the fd-fresh-ids-when-creating-a-table branch from ad95028 to bee1c81Compare August 30, 2022 19:45

Fokko mentioned this pull request Aug 30, 2022
Python: Fine-tune the API #5672
Merged

Fokko commented Aug 31, 2022
View reviewed changes

Fokko merged commit 08bb3e2 into apache:masterAug 31, 2022

Fokko deleted the fd-fresh-ids-when-creating-a-table branch August 31, 2022 17:28

HonahX mentioned this pull request Jan 29, 2024
create_table with a PyArrow Schema apache/iceberg-python#305
Merged

Python: Reassign schema/partition-spec/sort-order ids
178659c
When creating a new schema Resolvesapache#5468

github-actionsBot added the python label Aug 24, 2022

Fokko commented Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/metadata.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/metadata.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/metadata.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/sorting.py Outdated

Fokko added 4 commits August 24, 2022 23:37

Convert into pre-order
6482984

Merge branch 'master' ofhttps://github.com/apache/iceberginto fd-fr…
2bdf233
…esh-ids-when-creating-a-table

Small docstring improvements
076b6e0

Merge branch 'master' ofhttps://github.com/apache/iceberginto fd-fr…
2df8cee
…esh-ids-when-creating-a-table

Fokko force-pushed the fd-fresh-ids-when-creating-a-table branch from dc0dbc5 to 2df8ceeCompare August 25, 2022 08:09

Fokko commented Aug 25, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/metadata.py

rdblue reviewed Aug 28, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py

rdblue reviewed Aug 28, 2022
View reviewed changes

Fokko added 2 commits August 30, 2022 20:23

Merge branch 'master' ofhttps://github.com/apache/iceberginto fd-fr…
1091859
…esh-ids-when-creating-a-table

Fix order of the structs/lists/maps
bee1c81

Fokko force-pushed the fd-fresh-ids-when-creating-a-table branch from ad95028 to bee1c81Compare August 30, 2022 19:45

Fokko mentioned this pull request Aug 30, 2022
Python: Fine-tune the API #5672
Merged

Fokko commented Aug 31, 2022
View reviewed changes

Fokko merged commit 08bb3e2 into apache:masterAug 31, 2022

Fokko deleted the fd-fresh-ids-when-creating-a-table branch August 31, 2022 17:28

HonahX mentioned this pull request Jan 29, 2024
create_table with a PyArrow Schema apache/iceberg-python#305
Merged

Python: Reassign schema/partition-spec/sort-order ids
178659c
When creating a new schema Resolvesapache#5468

github-actionsBot added the python label Aug 24, 2022

Fokko commented Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/metadata.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/metadata.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/metadata.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/sorting.py Outdated

Fokko added 4 commits August 24, 2022 23:37

Convert into pre-order
6482984

Merge branch 'master' ofhttps://github.com/apache/iceberginto fd-fr…
2bdf233
…esh-ids-when-creating-a-table

Small docstring improvements
076b6e0

Merge branch 'master' ofhttps://github.com/apache/iceberginto fd-fr…
2df8cee
…esh-ids-when-creating-a-table

Fokko force-pushed the fd-fresh-ids-when-creating-a-table branch from dc0dbc5 to 2df8ceeCompare August 25, 2022 08:09

Fokko commented Aug 25, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/metadata.py

rdblue reviewed Aug 28, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py

rdblue reviewed Aug 28, 2022
View reviewed changes

Fokko added 2 commits August 30, 2022 20:23

Merge branch 'master' ofhttps://github.com/apache/iceberginto fd-fr…
1091859
…esh-ids-when-creating-a-table

Fix order of the structs/lists/maps
bee1c81

Fokko force-pushed the fd-fresh-ids-when-creating-a-table branch from ad95028 to bee1c81Compare August 30, 2022 19:45

Fokko mentioned this pull request Aug 30, 2022
Python: Fine-tune the API #5672
Merged

Fokko commented Aug 31, 2022
View reviewed changes

Fokko merged commit 08bb3e2 into apache:masterAug 31, 2022

Fokko deleted the fd-fresh-ids-when-creating-a-table branch August 31, 2022 17:28

HonahX mentioned this pull request Jan 29, 2024
create_table with a PyArrow Schema apache/iceberg-python#305
Merged

Python: Reassign schema/partition-spec/sort-order ids
178659c
When creating a new schema Resolvesapache#5468

github-actionsBot added the python label Aug 24, 2022

Fokko commented Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/metadata.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/metadata.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/metadata.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/sorting.py Outdated

Fokko added 4 commits August 24, 2022 23:37

Convert into pre-order
6482984

Merge branch 'master' ofhttps://github.com/apache/iceberginto fd-fr…
2bdf233
…esh-ids-when-creating-a-table

Small docstring improvements
076b6e0

Merge branch 'master' ofhttps://github.com/apache/iceberginto fd-fr…
2df8cee
…esh-ids-when-creating-a-table

Fokko force-pushed the fd-fresh-ids-when-creating-a-table branch from dc0dbc5 to 2df8ceeCompare August 25, 2022 08:09

Fokko commented Aug 25, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/metadata.py

rdblue reviewed Aug 28, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py

rdblue reviewed Aug 28, 2022
View reviewed changes

Fokko added 2 commits August 30, 2022 20:23

Merge branch 'master' ofhttps://github.com/apache/iceberginto fd-fr…
1091859
…esh-ids-when-creating-a-table

Fix order of the structs/lists/maps
bee1c81

Fokko force-pushed the fd-fresh-ids-when-creating-a-table branch from ad95028 to bee1c81Compare August 30, 2022 19:45

Fokko mentioned this pull request Aug 30, 2022
Python: Fine-tune the API #5672
Merged

Fokko commented Aug 31, 2022
View reviewed changes

Fokko merged commit 08bb3e2 into apache:masterAug 31, 2022

Fokko deleted the fd-fresh-ids-when-creating-a-table branch August 31, 2022 17:28

HonahX mentioned this pull request Jan 29, 2024
create_table with a PyArrow Schema apache/iceberg-python#305
Merged

Python: Reassign schema/partition-spec/sort-order ids
178659c
When creating a new schema Resolvesapache#5468

github-actionsBot added the python label Aug 24, 2022

Fokko commented Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/metadata.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/metadata.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/metadata.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/sorting.py Outdated

Fokko added 4 commits August 24, 2022 23:37

Convert into pre-order
6482984

Merge branch 'master' ofhttps://github.com/apache/iceberginto fd-fr…
2bdf233
…esh-ids-when-creating-a-table

Small docstring improvements
076b6e0

Merge branch 'master' ofhttps://github.com/apache/iceberginto fd-fr…
2df8cee
…esh-ids-when-creating-a-table

Fokko force-pushed the fd-fresh-ids-when-creating-a-table branch from dc0dbc5 to 2df8ceeCompare August 25, 2022 08:09

Fokko commented Aug 25, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/metadata.py

rdblue reviewed Aug 28, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py

rdblue reviewed Aug 28, 2022
View reviewed changes

Fokko added 2 commits August 30, 2022 20:23

Merge branch 'master' ofhttps://github.com/apache/iceberginto fd-fr…
1091859
…esh-ids-when-creating-a-table

Fix order of the structs/lists/maps
bee1c81

Fokko force-pushed the fd-fresh-ids-when-creating-a-table branch from ad95028 to bee1c81Compare August 30, 2022 19:45

Fokko mentioned this pull request Aug 30, 2022
Python: Fine-tune the API #5672
Merged

Fokko commented Aug 31, 2022
View reviewed changes

Fokko merged commit 08bb3e2 into apache:masterAug 31, 2022

Fokko deleted the fd-fresh-ids-when-creating-a-table branch August 31, 2022 17:28

HonahX mentioned this pull request Jan 29, 2024
create_table with a PyArrow Schema apache/iceberg-python#305
Merged

Python: Reassign schema/partition-spec/sort-order ids
178659c
When creating a new schema Resolvesapache#5468

github-actionsBot added the python label Aug 24, 2022

Fokko commented Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/metadata.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/metadata.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/metadata.py Outdated

rdblue reviewed Aug 24, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/sorting.py Outdated

Fokko added 4 commits August 24, 2022 23:37

Convert into pre-order
6482984

Merge branch 'master' ofhttps://github.com/apache/iceberginto fd-fr…
2bdf233
…esh-ids-when-creating-a-table

Small docstring improvements
076b6e0

Merge branch 'master' ofhttps://github.com/apache/iceberginto fd-fr…
2df8cee
…esh-ids-when-creating-a-table

Fokko force-pushed the fd-fresh-ids-when-creating-a-table branch from dc0dbc5 to 2df8ceeCompare August 25, 2022 08:09

Fokko commented Aug 25, 2022
View reviewed changes

Comment threadpython/pyiceberg/table/metadata.py

rdblue reviewed Aug 28, 2022
View reviewed changes

Comment threadpython/pyiceberg/schema.py

rdblue reviewed Aug 28, 2022
View reviewed changes

Fokko added 2 commits August 30, 2022 20:23

Merge branch 'master' ofhttps://github.com/apache/iceberginto fd-fr…
1091859
…esh-ids-when-creating-a-table

Fix order of the structs/lists/maps
bee1c81

Fokko force-pushed the fd-fresh-ids-when-creating-a-table branch from ad95028 to bee1c81Compare August 30, 2022 19:45

Fokko mentioned this pull request Aug 30, 2022
Python: Fine-tune the API #5672
Merged

Fokko commented Aug 31, 2022
View reviewed changes

Fokko merged commit 08bb3e2 into apache:masterAug 31, 2022

Fokko deleted the fd-fresh-ids-when-creating-a-table branch August 31, 2022 17:28

HonahX mentioned this pull request Jan 29, 2024
create_table with a PyArrow Schema apache/iceberg-python#305
Merged

		"""Visit a PrimitiveType"""


		class PreOrderSchemaVisitor(Generic[T], ABC):

Uh oh!

Conversation

Fokko commented Aug 24, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rdblueAug 24, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Fokko left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Fokko commented Aug 24, 2022

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

rdblueAug 24, 2022• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

rdblueAug 24, 2022•
edited
Loading

rdblueAug 24, 2022•
edited
Loading