Use Pydantic's model_copy for model modification when updating table metadata

[HonahX]

Fixes#179

This PR uses Pydantic's model_copy to apply table updates to metadata. Specifically:

• When applying AddSchema, SetCurrentSchema, AddSnapshot, SetSnapshotRef, we use base_metadata.model_copy(update=...) to get the updated metadata.
• When applying UpgradeFormatVersion, we still need to rebuild the whole model from TableMetadataV1 to TableMetadataV2

@Fokko Could you please take a look at this when you have a chance? Thanks!

[HonahX]

Since model_copy performs a shallow copy by default, I believe we need to execute a deep copy before returning the final new_metadata. Otherwise, base_metadata might be inadvertently altered due to any improper updates applied to new_metadata subsequently.

Furthermore, as indicated by pydantic/pydantic#418 and by some local tests, model_copy(update=) does not validate the contents of update. I think it might be good to reconstruct the metadata at this point to initiate the validation process.

(Alternatively, we could perform a deep copy here and incorporate the validation into our unit tests. Open to discussion on this approach.)

[Fokko]

I think we're okay to skip the validation on the Pydantic side, we should validate the input anyway to generate a more meaningful error.

We could set it to deep-copy using the deep=True kwarg. Doing this in Pydantic is probably an order of magnitude more efficient than doing this in Python.

[HonahX]

Thanks for the explanation! I changed to model_copy(deep=True) here and moved the validation to a unit test

[Fokko]

This is great @HonahX! Thanks for fixing this right away.

One thing I wondered if it is better to do a deep-copy when returning the final metadata, or just deep-copy in the steps where we update the metadata. But I think the current approach is more efficient when there are several updates.

[HonahX]

Fixes#179

This PR uses Pydantic's model_copy to apply table updates to metadata. Specifically:

• When applying AddSchema, SetCurrentSchema, AddSnapshot, SetSnapshotRef, we use base_metadata.model_copy(update=...) to get the updated metadata.
• When applying UpgradeFormatVersion, we still need to rebuild the whole model from TableMetadataV1 to TableMetadataV2

@Fokko Could you please take a look at this when you have a chance? Thanks!

[HonahX]

Since model_copy performs a shallow copy by default, I believe we need to execute a deep copy before returning the final new_metadata. Otherwise, base_metadata might be inadvertently altered due to any improper updates applied to new_metadata subsequently.

Furthermore, as indicated by pydantic/pydantic#418 and by some local tests, model_copy(update=) does not validate the contents of update. I think it might be good to reconstruct the metadata at this point to initiate the validation process.

(Alternatively, we could perform a deep copy here and incorporate the validation into our unit tests. Open to discussion on this approach.)

[Fokko]

I think we're okay to skip the validation on the Pydantic side, we should validate the input anyway to generate a more meaningful error.

We could set it to deep-copy using the deep=True kwarg. Doing this in Pydantic is probably an order of magnitude more efficient than doing this in Python.

[HonahX]

Thanks for the explanation! I changed to model_copy(deep=True) here and moved the validation to a unit test

[Fokko]

This is great @HonahX! Thanks for fixing this right away.

One thing I wondered if it is better to do a deep-copy when returning the final metadata, or just deep-copy in the steps where we update the metadata. But I think the current approach is more efficient when there are several updates.

[HonahX]

Fixes#179

This PR uses Pydantic's model_copy to apply table updates to metadata. Specifically:

• When applying AddSchema, SetCurrentSchema, AddSnapshot, SetSnapshotRef, we use base_metadata.model_copy(update=...) to get the updated metadata.
• When applying UpgradeFormatVersion, we still need to rebuild the whole model from TableMetadataV1 to TableMetadataV2

@Fokko Could you please take a look at this when you have a chance? Thanks!

[HonahX]

Since model_copy performs a shallow copy by default, I believe we need to execute a deep copy before returning the final new_metadata. Otherwise, base_metadata might be inadvertently altered due to any improper updates applied to new_metadata subsequently.

Furthermore, as indicated by pydantic/pydantic#418 and by some local tests, model_copy(update=) does not validate the contents of update. I think it might be good to reconstruct the metadata at this point to initiate the validation process.

(Alternatively, we could perform a deep copy here and incorporate the validation into our unit tests. Open to discussion on this approach.)

[Fokko]

I think we're okay to skip the validation on the Pydantic side, we should validate the input anyway to generate a more meaningful error.

We could set it to deep-copy using the deep=True kwarg. Doing this in Pydantic is probably an order of magnitude more efficient than doing this in Python.

[HonahX]

Thanks for the explanation! I changed to model_copy(deep=True) here and moved the validation to a unit test

[Fokko]

This is great @HonahX! Thanks for fixing this right away.

One thing I wondered if it is better to do a deep-copy when returning the final metadata, or just deep-copy in the steps where we update the metadata. But I think the current approach is more efficient when there are several updates.

[HonahX]

Fixes#179

This PR uses Pydantic's model_copy to apply table updates to metadata. Specifically:

• When applying AddSchema, SetCurrentSchema, AddSnapshot, SetSnapshotRef, we use base_metadata.model_copy(update=...) to get the updated metadata.
• When applying UpgradeFormatVersion, we still need to rebuild the whole model from TableMetadataV1 to TableMetadataV2

@Fokko Could you please take a look at this when you have a chance? Thanks!

[HonahX]

Since model_copy performs a shallow copy by default, I believe we need to execute a deep copy before returning the final new_metadata. Otherwise, base_metadata might be inadvertently altered due to any improper updates applied to new_metadata subsequently.

Furthermore, as indicated by pydantic/pydantic#418 and by some local tests, model_copy(update=) does not validate the contents of update. I think it might be good to reconstruct the metadata at this point to initiate the validation process.

(Alternatively, we could perform a deep copy here and incorporate the validation into our unit tests. Open to discussion on this approach.)

[Fokko]

I think we're okay to skip the validation on the Pydantic side, we should validate the input anyway to generate a more meaningful error.

We could set it to deep-copy using the deep=True kwarg. Doing this in Pydantic is probably an order of magnitude more efficient than doing this in Python.

[HonahX]

Thanks for the explanation! I changed to model_copy(deep=True) here and moved the validation to a unit test

[Fokko]

This is great @HonahX! Thanks for fixing this right away.

One thing I wondered if it is better to do a deep-copy when returning the final metadata, or just deep-copy in the steps where we update the metadata. But I think the current approach is more efficient when there are several updates.

[HonahX]

Fixes#179

This PR uses Pydantic's model_copy to apply table updates to metadata. Specifically:

• When applying AddSchema, SetCurrentSchema, AddSnapshot, SetSnapshotRef, we use base_metadata.model_copy(update=...) to get the updated metadata.
• When applying UpgradeFormatVersion, we still need to rebuild the whole model from TableMetadataV1 to TableMetadataV2

@Fokko Could you please take a look at this when you have a chance? Thanks!

[HonahX]

Since model_copy performs a shallow copy by default, I believe we need to execute a deep copy before returning the final new_metadata. Otherwise, base_metadata might be inadvertently altered due to any improper updates applied to new_metadata subsequently.

Furthermore, as indicated by pydantic/pydantic#418 and by some local tests, model_copy(update=) does not validate the contents of update. I think it might be good to reconstruct the metadata at this point to initiate the validation process.

(Alternatively, we could perform a deep copy here and incorporate the validation into our unit tests. Open to discussion on this approach.)

[Fokko]

I think we're okay to skip the validation on the Pydantic side, we should validate the input anyway to generate a more meaningful error.

We could set it to deep-copy using the deep=True kwarg. Doing this in Pydantic is probably an order of magnitude more efficient than doing this in Python.

[HonahX]

Thanks for the explanation! I changed to model_copy(deep=True) here and moved the validation to a unit test

[Fokko]

This is great @HonahX! Thanks for fixing this right away.

One thing I wondered if it is better to do a deep-copy when returning the final metadata, or just deep-copy in the steps where we update the metadata. But I think the current approach is more efficient when there are several updates.

[HonahX]

Fixes#179

This PR uses Pydantic's model_copy to apply table updates to metadata. Specifically:

• When applying AddSchema, SetCurrentSchema, AddSnapshot, SetSnapshotRef, we use base_metadata.model_copy(update=...) to get the updated metadata.
• When applying UpgradeFormatVersion, we still need to rebuild the whole model from TableMetadataV1 to TableMetadataV2

@Fokko Could you please take a look at this when you have a chance? Thanks!

[HonahX]

Since model_copy performs a shallow copy by default, I believe we need to execute a deep copy before returning the final new_metadata. Otherwise, base_metadata might be inadvertently altered due to any improper updates applied to new_metadata subsequently.

Furthermore, as indicated by pydantic/pydantic#418 and by some local tests, model_copy(update=) does not validate the contents of update. I think it might be good to reconstruct the metadata at this point to initiate the validation process.

(Alternatively, we could perform a deep copy here and incorporate the validation into our unit tests. Open to discussion on this approach.)

[Fokko]

I think we're okay to skip the validation on the Pydantic side, we should validate the input anyway to generate a more meaningful error.

We could set it to deep-copy using the deep=True kwarg. Doing this in Pydantic is probably an order of magnitude more efficient than doing this in Python.

[HonahX]

Thanks for the explanation! I changed to model_copy(deep=True) here and moved the validation to a unit test

[Fokko]

This is great @HonahX! Thanks for fixing this right away.

One thing I wondered if it is better to do a deep-copy when returning the final metadata, or just deep-copy in the steps where we update the metadata. But I think the current approach is more efficient when there are several updates.

[HonahX]

Fixes#179

This PR uses Pydantic's model_copy to apply table updates to metadata. Specifically:

• When applying AddSchema, SetCurrentSchema, AddSnapshot, SetSnapshotRef, we use base_metadata.model_copy(update=...) to get the updated metadata.
• When applying UpgradeFormatVersion, we still need to rebuild the whole model from TableMetadataV1 to TableMetadataV2

@Fokko Could you please take a look at this when you have a chance? Thanks!

[HonahX]

Since model_copy performs a shallow copy by default, I believe we need to execute a deep copy before returning the final new_metadata. Otherwise, base_metadata might be inadvertently altered due to any improper updates applied to new_metadata subsequently.

Furthermore, as indicated by pydantic/pydantic#418 and by some local tests, model_copy(update=) does not validate the contents of update. I think it might be good to reconstruct the metadata at this point to initiate the validation process.

(Alternatively, we could perform a deep copy here and incorporate the validation into our unit tests. Open to discussion on this approach.)

[Fokko]

I think we're okay to skip the validation on the Pydantic side, we should validate the input anyway to generate a more meaningful error.

We could set it to deep-copy using the deep=True kwarg. Doing this in Pydantic is probably an order of magnitude more efficient than doing this in Python.

[HonahX]

Thanks for the explanation! I changed to model_copy(deep=True) here and moved the validation to a unit test

[Fokko]

This is great @HonahX! Thanks for fixing this right away.

One thing I wondered if it is better to do a deep-copy when returning the final metadata, or just deep-copy in the steps where we update the metadata. But I think the current approach is more efficient when there are several updates.

[HonahX]

Fixes#179

This PR uses Pydantic's model_copy to apply table updates to metadata. Specifically:

• When applying AddSchema, SetCurrentSchema, AddSnapshot, SetSnapshotRef, we use base_metadata.model_copy(update=...) to get the updated metadata.
• When applying UpgradeFormatVersion, we still need to rebuild the whole model from TableMetadataV1 to TableMetadataV2

@Fokko Could you please take a look at this when you have a chance? Thanks!

[HonahX]

Since model_copy performs a shallow copy by default, I believe we need to execute a deep copy before returning the final new_metadata. Otherwise, base_metadata might be inadvertently altered due to any improper updates applied to new_metadata subsequently.

Furthermore, as indicated by pydantic/pydantic#418 and by some local tests, model_copy(update=) does not validate the contents of update. I think it might be good to reconstruct the metadata at this point to initiate the validation process.

(Alternatively, we could perform a deep copy here and incorporate the validation into our unit tests. Open to discussion on this approach.)

[Fokko]

I think we're okay to skip the validation on the Pydantic side, we should validate the input anyway to generate a more meaningful error.

We could set it to deep-copy using the deep=True kwarg. Doing this in Pydantic is probably an order of magnitude more efficient than doing this in Python.

[HonahX]

Thanks for the explanation! I changed to model_copy(deep=True) here and moved the validation to a unit test

[Fokko]

This is great @HonahX! Thanks for fixing this right away.

One thing I wondered if it is better to do a deep-copy when returning the final metadata, or just deep-copy in the steps where we update the metadata. But I think the current approach is more efficient when there are several updates.