features: support encrypted properties

[cportele]

Part of addressing ldproxy/ldproxy#1742.

Properties of the new schema types ENCRYPTED and ENCRYPTED_ARRAY store AES-256-GCM ciphertext (nonce, ciphertext and tag as one binary value) in the database, the logical type of the plaintext is declared in valueType, following the pattern of FEATURE_REF and FEATURE_REF_ARRAY. The symmetric key is configured with the new provider option encryptionKey, typically referencing an environment variable.

• on the read path, values are decrypted in a token transformer and normalized according to the valueType, so that the lexical form of temporal values is canonical; ciphertext is read via JDBC getBytes, which works for all SQL dialects
• on the write path, values are normalized, encrypted and inlined as a bytea literal; the plain literal renderer rejects values of encrypted columns, so that no code path can write plaintext unnoticed
• encrypted properties are neither queryable nor sortable by construction; derived schemas present the logical type
• invalid values in a mutation request are reported as a client error, a stored value that fails to decrypt or normalize is a server error
• providers fail at startup when encrypted properties are misconfigured: properties of type ENCRYPTED or ENCRYPTED_ARRAY on a provider type that does not support them, a missing encryptionKey, or a malformed key (which is also validated when no property uses it). SQL providers, including Oracle and GeoParquet, support encrypted properties; other provider types reject them at startup.

[azahnen]

I moved the encryption to xtraplatform-base. Please check if everything still works as expected, then merge.

[cportele]

@azahnen - I had to make three changes to make it work:

• xtraplatform:
  • add encryption: {} to cfg.base.yml
  • move the code from Encryption.onStartup() to the constructor and remove AppLifeCycle. Otherwise isEnabled() may return false even if I set the priority to a low value. BlobStoreImpl has priority 20, providers are started at priority 210. with async startup Encryption.onStartup() may still not have been executed in some cases and encryption will (silently) fail.
• ldproxy:
  • amend ldproxy-cfg to reflect the encryption changes.

Then everything works as expected. If these changes look right to you, then I will commit them and merge.

[azahnen]

@cportele
I pushed changes to xtraplatform that should address the two points. Throwing exceptions in a constructor is never a good idea, so the setup is still in onStart. But isEnabled() now only depends on key validation, which is done in the constructor.

[cportele]

@azahnen

The problem that now surfaced is that EncryptionImpl.onStart() never runs, since any AppLifeCycle instance in the base layer will not be included in the set that the launcher starts (and I saw no simple way to add it). So I included the startup actions in the constructor, but no exception will be thrown, instead an error will be logged and encryption will be disabled in case of an error.

The other option that I saw would be a lazy initialization, but then the issue would only come up during the first query.

I will merge now, because without the changes in ldproxy-cfg, ldproxy won't build.

If there is a way to keep Encryption as an AppLifeCycle instance, we can still apply that change.

[cportele]

Part of addressing ldproxy/ldproxy#1742.

Properties of the new schema types ENCRYPTED and ENCRYPTED_ARRAY store AES-256-GCM ciphertext (nonce, ciphertext and tag as one binary value) in the database, the logical type of the plaintext is declared in valueType, following the pattern of FEATURE_REF and FEATURE_REF_ARRAY. The symmetric key is configured with the new provider option encryptionKey, typically referencing an environment variable.

• on the read path, values are decrypted in a token transformer and normalized according to the valueType, so that the lexical form of temporal values is canonical; ciphertext is read via JDBC getBytes, which works for all SQL dialects
• on the write path, values are normalized, encrypted and inlined as a bytea literal; the plain literal renderer rejects values of encrypted columns, so that no code path can write plaintext unnoticed
• encrypted properties are neither queryable nor sortable by construction; derived schemas present the logical type
• invalid values in a mutation request are reported as a client error, a stored value that fails to decrypt or normalize is a server error
• providers fail at startup when encrypted properties are misconfigured: properties of type ENCRYPTED or ENCRYPTED_ARRAY on a provider type that does not support them, a missing encryptionKey, or a malformed key (which is also validated when no property uses it). SQL providers, including Oracle and GeoParquet, support encrypted properties; other provider types reject them at startup.

[azahnen]

I moved the encryption to xtraplatform-base. Please check if everything still works as expected, then merge.

[cportele]

@azahnen - I had to make three changes to make it work:

• xtraplatform:
  • add encryption: {} to cfg.base.yml
  • move the code from Encryption.onStartup() to the constructor and remove AppLifeCycle. Otherwise isEnabled() may return false even if I set the priority to a low value. BlobStoreImpl has priority 20, providers are started at priority 210. with async startup Encryption.onStartup() may still not have been executed in some cases and encryption will (silently) fail.
• ldproxy:
  • amend ldproxy-cfg to reflect the encryption changes.

Then everything works as expected. If these changes look right to you, then I will commit them and merge.

[azahnen]

@cportele
I pushed changes to xtraplatform that should address the two points. Throwing exceptions in a constructor is never a good idea, so the setup is still in onStart. But isEnabled() now only depends on key validation, which is done in the constructor.

[cportele]

@azahnen

The problem that now surfaced is that EncryptionImpl.onStart() never runs, since any AppLifeCycle instance in the base layer will not be included in the set that the launcher starts (and I saw no simple way to add it). So I included the startup actions in the constructor, but no exception will be thrown, instead an error will be logged and encryption will be disabled in case of an error.

The other option that I saw would be a lazy initialization, but then the issue would only come up during the first query.

I will merge now, because without the changes in ldproxy-cfg, ldproxy won't build.

If there is a way to keep Encryption as an AppLifeCycle instance, we can still apply that change.

[cportele]

Part of addressing ldproxy/ldproxy#1742.

Properties of the new schema types ENCRYPTED and ENCRYPTED_ARRAY store AES-256-GCM ciphertext (nonce, ciphertext and tag as one binary value) in the database, the logical type of the plaintext is declared in valueType, following the pattern of FEATURE_REF and FEATURE_REF_ARRAY. The symmetric key is configured with the new provider option encryptionKey, typically referencing an environment variable.

• on the read path, values are decrypted in a token transformer and normalized according to the valueType, so that the lexical form of temporal values is canonical; ciphertext is read via JDBC getBytes, which works for all SQL dialects
• on the write path, values are normalized, encrypted and inlined as a bytea literal; the plain literal renderer rejects values of encrypted columns, so that no code path can write plaintext unnoticed
• encrypted properties are neither queryable nor sortable by construction; derived schemas present the logical type
• invalid values in a mutation request are reported as a client error, a stored value that fails to decrypt or normalize is a server error
• providers fail at startup when encrypted properties are misconfigured: properties of type ENCRYPTED or ENCRYPTED_ARRAY on a provider type that does not support them, a missing encryptionKey, or a malformed key (which is also validated when no property uses it). SQL providers, including Oracle and GeoParquet, support encrypted properties; other provider types reject them at startup.

[azahnen]

I moved the encryption to xtraplatform-base. Please check if everything still works as expected, then merge.

[cportele]

@azahnen - I had to make three changes to make it work:

• xtraplatform:
  • add encryption: {} to cfg.base.yml
  • move the code from Encryption.onStartup() to the constructor and remove AppLifeCycle. Otherwise isEnabled() may return false even if I set the priority to a low value. BlobStoreImpl has priority 20, providers are started at priority 210. with async startup Encryption.onStartup() may still not have been executed in some cases and encryption will (silently) fail.
• ldproxy:
  • amend ldproxy-cfg to reflect the encryption changes.

Then everything works as expected. If these changes look right to you, then I will commit them and merge.

[azahnen]

@cportele
I pushed changes to xtraplatform that should address the two points. Throwing exceptions in a constructor is never a good idea, so the setup is still in onStart. But isEnabled() now only depends on key validation, which is done in the constructor.

[cportele]

@azahnen

The problem that now surfaced is that EncryptionImpl.onStart() never runs, since any AppLifeCycle instance in the base layer will not be included in the set that the launcher starts (and I saw no simple way to add it). So I included the startup actions in the constructor, but no exception will be thrown, instead an error will be logged and encryption will be disabled in case of an error.

The other option that I saw would be a lazy initialization, but then the issue would only come up during the first query.

I will merge now, because without the changes in ldproxy-cfg, ldproxy won't build.

If there is a way to keep Encryption as an AppLifeCycle instance, we can still apply that change.

[cportele]

Part of addressing ldproxy/ldproxy#1742.

Properties of the new schema types ENCRYPTED and ENCRYPTED_ARRAY store AES-256-GCM ciphertext (nonce, ciphertext and tag as one binary value) in the database, the logical type of the plaintext is declared in valueType, following the pattern of FEATURE_REF and FEATURE_REF_ARRAY. The symmetric key is configured with the new provider option encryptionKey, typically referencing an environment variable.

• on the read path, values are decrypted in a token transformer and normalized according to the valueType, so that the lexical form of temporal values is canonical; ciphertext is read via JDBC getBytes, which works for all SQL dialects
• on the write path, values are normalized, encrypted and inlined as a bytea literal; the plain literal renderer rejects values of encrypted columns, so that no code path can write plaintext unnoticed
• encrypted properties are neither queryable nor sortable by construction; derived schemas present the logical type
• invalid values in a mutation request are reported as a client error, a stored value that fails to decrypt or normalize is a server error
• providers fail at startup when encrypted properties are misconfigured: properties of type ENCRYPTED or ENCRYPTED_ARRAY on a provider type that does not support them, a missing encryptionKey, or a malformed key (which is also validated when no property uses it). SQL providers, including Oracle and GeoParquet, support encrypted properties; other provider types reject them at startup.

[azahnen]

I moved the encryption to xtraplatform-base. Please check if everything still works as expected, then merge.

[cportele]

@azahnen - I had to make three changes to make it work:

• xtraplatform:
  • add encryption: {} to cfg.base.yml
  • move the code from Encryption.onStartup() to the constructor and remove AppLifeCycle. Otherwise isEnabled() may return false even if I set the priority to a low value. BlobStoreImpl has priority 20, providers are started at priority 210. with async startup Encryption.onStartup() may still not have been executed in some cases and encryption will (silently) fail.
• ldproxy:
  • amend ldproxy-cfg to reflect the encryption changes.

Then everything works as expected. If these changes look right to you, then I will commit them and merge.

[azahnen]

@cportele
I pushed changes to xtraplatform that should address the two points. Throwing exceptions in a constructor is never a good idea, so the setup is still in onStart. But isEnabled() now only depends on key validation, which is done in the constructor.

[cportele]

@azahnen

The problem that now surfaced is that EncryptionImpl.onStart() never runs, since any AppLifeCycle instance in the base layer will not be included in the set that the launcher starts (and I saw no simple way to add it). So I included the startup actions in the constructor, but no exception will be thrown, instead an error will be logged and encryption will be disabled in case of an error.

The other option that I saw would be a lazy initialization, but then the issue would only come up during the first query.

I will merge now, because without the changes in ldproxy-cfg, ldproxy won't build.

If there is a way to keep Encryption as an AppLifeCycle instance, we can still apply that change.

[cportele]

Part of addressing ldproxy/ldproxy#1742.

Properties of the new schema types ENCRYPTED and ENCRYPTED_ARRAY store AES-256-GCM ciphertext (nonce, ciphertext and tag as one binary value) in the database, the logical type of the plaintext is declared in valueType, following the pattern of FEATURE_REF and FEATURE_REF_ARRAY. The symmetric key is configured with the new provider option encryptionKey, typically referencing an environment variable.

• on the read path, values are decrypted in a token transformer and normalized according to the valueType, so that the lexical form of temporal values is canonical; ciphertext is read via JDBC getBytes, which works for all SQL dialects
• on the write path, values are normalized, encrypted and inlined as a bytea literal; the plain literal renderer rejects values of encrypted columns, so that no code path can write plaintext unnoticed
• encrypted properties are neither queryable nor sortable by construction; derived schemas present the logical type
• invalid values in a mutation request are reported as a client error, a stored value that fails to decrypt or normalize is a server error
• providers fail at startup when encrypted properties are misconfigured: properties of type ENCRYPTED or ENCRYPTED_ARRAY on a provider type that does not support them, a missing encryptionKey, or a malformed key (which is also validated when no property uses it). SQL providers, including Oracle and GeoParquet, support encrypted properties; other provider types reject them at startup.

[azahnen]

I moved the encryption to xtraplatform-base. Please check if everything still works as expected, then merge.

[cportele]

@azahnen - I had to make three changes to make it work:

• xtraplatform:
  • add encryption: {} to cfg.base.yml
  • move the code from Encryption.onStartup() to the constructor and remove AppLifeCycle. Otherwise isEnabled() may return false even if I set the priority to a low value. BlobStoreImpl has priority 20, providers are started at priority 210. with async startup Encryption.onStartup() may still not have been executed in some cases and encryption will (silently) fail.
• ldproxy:
  • amend ldproxy-cfg to reflect the encryption changes.

Then everything works as expected. If these changes look right to you, then I will commit them and merge.

[azahnen]

@cportele
I pushed changes to xtraplatform that should address the two points. Throwing exceptions in a constructor is never a good idea, so the setup is still in onStart. But isEnabled() now only depends on key validation, which is done in the constructor.

[cportele]

@azahnen

The problem that now surfaced is that EncryptionImpl.onStart() never runs, since any AppLifeCycle instance in the base layer will not be included in the set that the launcher starts (and I saw no simple way to add it). So I included the startup actions in the constructor, but no exception will be thrown, instead an error will be logged and encryption will be disabled in case of an error.

The other option that I saw would be a lazy initialization, but then the issue would only come up during the first query.

I will merge now, because without the changes in ldproxy-cfg, ldproxy won't build.

If there is a way to keep Encryption as an AppLifeCycle instance, we can still apply that change.

[cportele]

Part of addressing ldproxy/ldproxy#1742.

Properties of the new schema types ENCRYPTED and ENCRYPTED_ARRAY store AES-256-GCM ciphertext (nonce, ciphertext and tag as one binary value) in the database, the logical type of the plaintext is declared in valueType, following the pattern of FEATURE_REF and FEATURE_REF_ARRAY. The symmetric key is configured with the new provider option encryptionKey, typically referencing an environment variable.

• on the read path, values are decrypted in a token transformer and normalized according to the valueType, so that the lexical form of temporal values is canonical; ciphertext is read via JDBC getBytes, which works for all SQL dialects
• on the write path, values are normalized, encrypted and inlined as a bytea literal; the plain literal renderer rejects values of encrypted columns, so that no code path can write plaintext unnoticed
• encrypted properties are neither queryable nor sortable by construction; derived schemas present the logical type
• invalid values in a mutation request are reported as a client error, a stored value that fails to decrypt or normalize is a server error
• providers fail at startup when encrypted properties are misconfigured: properties of type ENCRYPTED or ENCRYPTED_ARRAY on a provider type that does not support them, a missing encryptionKey, or a malformed key (which is also validated when no property uses it). SQL providers, including Oracle and GeoParquet, support encrypted properties; other provider types reject them at startup.

[azahnen]

I moved the encryption to xtraplatform-base. Please check if everything still works as expected, then merge.

[cportele]

@azahnen - I had to make three changes to make it work:

• xtraplatform:
  • add encryption: {} to cfg.base.yml
  • move the code from Encryption.onStartup() to the constructor and remove AppLifeCycle. Otherwise isEnabled() may return false even if I set the priority to a low value. BlobStoreImpl has priority 20, providers are started at priority 210. with async startup Encryption.onStartup() may still not have been executed in some cases and encryption will (silently) fail.
• ldproxy:
  • amend ldproxy-cfg to reflect the encryption changes.

Then everything works as expected. If these changes look right to you, then I will commit them and merge.

[azahnen]

@cportele
I pushed changes to xtraplatform that should address the two points. Throwing exceptions in a constructor is never a good idea, so the setup is still in onStart. But isEnabled() now only depends on key validation, which is done in the constructor.

[cportele]

@azahnen

The problem that now surfaced is that EncryptionImpl.onStart() never runs, since any AppLifeCycle instance in the base layer will not be included in the set that the launcher starts (and I saw no simple way to add it). So I included the startup actions in the constructor, but no exception will be thrown, instead an error will be logged and encryption will be disabled in case of an error.

The other option that I saw would be a lazy initialization, but then the issue would only come up during the first query.

I will merge now, because without the changes in ldproxy-cfg, ldproxy won't build.

If there is a way to keep Encryption as an AppLifeCycle instance, we can still apply that change.

[cportele]

Part of addressing ldproxy/ldproxy#1742.

Properties of the new schema types ENCRYPTED and ENCRYPTED_ARRAY store AES-256-GCM ciphertext (nonce, ciphertext and tag as one binary value) in the database, the logical type of the plaintext is declared in valueType, following the pattern of FEATURE_REF and FEATURE_REF_ARRAY. The symmetric key is configured with the new provider option encryptionKey, typically referencing an environment variable.

• on the read path, values are decrypted in a token transformer and normalized according to the valueType, so that the lexical form of temporal values is canonical; ciphertext is read via JDBC getBytes, which works for all SQL dialects
• on the write path, values are normalized, encrypted and inlined as a bytea literal; the plain literal renderer rejects values of encrypted columns, so that no code path can write plaintext unnoticed
• encrypted properties are neither queryable nor sortable by construction; derived schemas present the logical type
• invalid values in a mutation request are reported as a client error, a stored value that fails to decrypt or normalize is a server error
• providers fail at startup when encrypted properties are misconfigured: properties of type ENCRYPTED or ENCRYPTED_ARRAY on a provider type that does not support them, a missing encryptionKey, or a malformed key (which is also validated when no property uses it). SQL providers, including Oracle and GeoParquet, support encrypted properties; other provider types reject them at startup.

[azahnen]

I moved the encryption to xtraplatform-base. Please check if everything still works as expected, then merge.

[cportele]

@azahnen - I had to make three changes to make it work:

• xtraplatform:
  • add encryption: {} to cfg.base.yml
  • move the code from Encryption.onStartup() to the constructor and remove AppLifeCycle. Otherwise isEnabled() may return false even if I set the priority to a low value. BlobStoreImpl has priority 20, providers are started at priority 210. with async startup Encryption.onStartup() may still not have been executed in some cases and encryption will (silently) fail.
• ldproxy:
  • amend ldproxy-cfg to reflect the encryption changes.

Then everything works as expected. If these changes look right to you, then I will commit them and merge.

[azahnen]

@cportele
I pushed changes to xtraplatform that should address the two points. Throwing exceptions in a constructor is never a good idea, so the setup is still in onStart. But isEnabled() now only depends on key validation, which is done in the constructor.

[cportele]

@azahnen

The problem that now surfaced is that EncryptionImpl.onStart() never runs, since any AppLifeCycle instance in the base layer will not be included in the set that the launcher starts (and I saw no simple way to add it). So I included the startup actions in the constructor, but no exception will be thrown, instead an error will be logged and encryption will be disabled in case of an error.

The other option that I saw would be a lazy initialization, but then the issue would only come up during the first query.

I will merge now, because without the changes in ldproxy-cfg, ldproxy won't build.

If there is a way to keep Encryption as an AppLifeCycle instance, we can still apply that change.

[cportele]

Part of addressing ldproxy/ldproxy#1742.

Properties of the new schema types ENCRYPTED and ENCRYPTED_ARRAY store AES-256-GCM ciphertext (nonce, ciphertext and tag as one binary value) in the database, the logical type of the plaintext is declared in valueType, following the pattern of FEATURE_REF and FEATURE_REF_ARRAY. The symmetric key is configured with the new provider option encryptionKey, typically referencing an environment variable.

• on the read path, values are decrypted in a token transformer and normalized according to the valueType, so that the lexical form of temporal values is canonical; ciphertext is read via JDBC getBytes, which works for all SQL dialects
• on the write path, values are normalized, encrypted and inlined as a bytea literal; the plain literal renderer rejects values of encrypted columns, so that no code path can write plaintext unnoticed
• encrypted properties are neither queryable nor sortable by construction; derived schemas present the logical type
• invalid values in a mutation request are reported as a client error, a stored value that fails to decrypt or normalize is a server error
• providers fail at startup when encrypted properties are misconfigured: properties of type ENCRYPTED or ENCRYPTED_ARRAY on a provider type that does not support them, a missing encryptionKey, or a malformed key (which is also validated when no property uses it). SQL providers, including Oracle and GeoParquet, support encrypted properties; other provider types reject them at startup.

[azahnen]

I moved the encryption to xtraplatform-base. Please check if everything still works as expected, then merge.

[cportele]

@azahnen - I had to make three changes to make it work:

• xtraplatform:
  • add encryption: {} to cfg.base.yml
  • move the code from Encryption.onStartup() to the constructor and remove AppLifeCycle. Otherwise isEnabled() may return false even if I set the priority to a low value. BlobStoreImpl has priority 20, providers are started at priority 210. with async startup Encryption.onStartup() may still not have been executed in some cases and encryption will (silently) fail.
• ldproxy:
  • amend ldproxy-cfg to reflect the encryption changes.

Then everything works as expected. If these changes look right to you, then I will commit them and merge.

[azahnen]

@cportele
I pushed changes to xtraplatform that should address the two points. Throwing exceptions in a constructor is never a good idea, so the setup is still in onStart. But isEnabled() now only depends on key validation, which is done in the constructor.

[cportele]

@azahnen

The problem that now surfaced is that EncryptionImpl.onStart() never runs, since any AppLifeCycle instance in the base layer will not be included in the set that the launcher starts (and I saw no simple way to add it). So I included the startup actions in the constructor, but no exception will be thrown, instead an error will be logged and encryption will be disabled in case of an error.

The other option that I saw would be a lazy initialization, but then the issue would only come up during the first query.

I will merge now, because without the changes in ldproxy-cfg, ldproxy won't build.

If there is a way to keep Encryption as an AppLifeCycle instance, we can still apply that change.