Add custom __reduce__ to kernels

[alexfikl]

This is inspired by a failure in the Stokes branch from pytentialinducer/pytential#162.

It started reproducibly raising exceptions as below. From what I can tell, the issue there was that the _memoize_dict from system_utils_get_base_kernel_matrix_lu_factorization somehow got into the pickling process.

E TypeError: no default __reduce__ due to non-trivial __cinit__
E when serializing dict item 'L'
E when serializing pytential.symbolic.pde.system_utils.LUFactorization state
E when serializing pytential.symbolic.pde.system_utils.LUFactorization object
E when serializing tuple item 0
E when serializing dict item (<class 'pytools._HasKwargs'>, frozenset({('order', None), ('hashable_kernel_arguments', (('mu', 1), ('nu', 0.4)))}))
E when serializing dict item '_memoize_dic_pytential.symbolic.pde.system_utils_get_base_kernel_matrix_lu_factorization'
E when serializing sumpy.kernel.BiharmonicKernel state
E when serializing sumpy.kernel.BiharmonicKernel object
E when serializing tuple item 0
E when serializing tuple item 1
E when serializing tuple item 0

I tried to implement a more generic __reduce__ with fields(self), but pretty much all the kernels have a custom constructor, so went with just copy pasting it a bit. Is there some better fix for this?

[inducer]

I'll try and take a look tomorrow. First I would like to understand how that LU factorization thing got into the pickling in the first place.

[alexfikl]

First I would like to understand how that LU factorization thing got into the pickling in the first place.

The function it's complaining about is here
https://github.com/isuruf/pytential/blob/1446481b5bc88fbfa5e1d15dc3f5a292eb0e5b18/pytential/symbolic/pde/system_utils.py#L560-L566

My understanding is that without a custom __reduce__, pickle just looks at __dict__ and the dict stored by memoize_on_first_arg got in there.

[inducer]

Ah OK, that makes sense. In that case, I think this is probably as good as it gets. I guess what I'd like better is if the kernels were dataclasses, but 🤷 this won't prevent that, and getting them to be dataclasses seems like a somewhat bigger lift.

[alexfikl]

I guess what I'd like better is if the kernels were dataclasses, but 🤷 this won't prevent that, and getting them to be dataclasses seems like a somewhat bigger lift.

All the ones in here are dataclasses, but that doesn't implement any special __reduce__. We could implement a generic one for all of them with a bit of work though?

[inducer]

We could implement a generic one for all of them with a bit of work though?

Don't know if it's worth it. Getting rid of custom constructors would be nice and would fix this, too, though.

[inducer]

I'll go ahead and merge this for the time being; it's easy enough to revert.

[inducer]

Thanks for making this!

[alexfikl]

This is inspired by a failure in the Stokes branch from pytentialinducer/pytential#162.

It started reproducibly raising exceptions as below. From what I can tell, the issue there was that the _memoize_dict from system_utils_get_base_kernel_matrix_lu_factorization somehow got into the pickling process.

E TypeError: no default __reduce__ due to non-trivial __cinit__
E when serializing dict item 'L'
E when serializing pytential.symbolic.pde.system_utils.LUFactorization state
E when serializing pytential.symbolic.pde.system_utils.LUFactorization object
E when serializing tuple item 0
E when serializing dict item (<class 'pytools._HasKwargs'>, frozenset({('order', None), ('hashable_kernel_arguments', (('mu', 1), ('nu', 0.4)))}))
E when serializing dict item '_memoize_dic_pytential.symbolic.pde.system_utils_get_base_kernel_matrix_lu_factorization'
E when serializing sumpy.kernel.BiharmonicKernel state
E when serializing sumpy.kernel.BiharmonicKernel object
E when serializing tuple item 0
E when serializing tuple item 1
E when serializing tuple item 0

I tried to implement a more generic __reduce__ with fields(self), but pretty much all the kernels have a custom constructor, so went with just copy pasting it a bit. Is there some better fix for this?

[inducer]

I'll try and take a look tomorrow. First I would like to understand how that LU factorization thing got into the pickling in the first place.

[alexfikl]

First I would like to understand how that LU factorization thing got into the pickling in the first place.

The function it's complaining about is here
https://github.com/isuruf/pytential/blob/1446481b5bc88fbfa5e1d15dc3f5a292eb0e5b18/pytential/symbolic/pde/system_utils.py#L560-L566

My understanding is that without a custom __reduce__, pickle just looks at __dict__ and the dict stored by memoize_on_first_arg got in there.

[inducer]

Ah OK, that makes sense. In that case, I think this is probably as good as it gets. I guess what I'd like better is if the kernels were dataclasses, but 🤷 this won't prevent that, and getting them to be dataclasses seems like a somewhat bigger lift.

[alexfikl]

I guess what I'd like better is if the kernels were dataclasses, but 🤷 this won't prevent that, and getting them to be dataclasses seems like a somewhat bigger lift.

All the ones in here are dataclasses, but that doesn't implement any special __reduce__. We could implement a generic one for all of them with a bit of work though?

[inducer]

We could implement a generic one for all of them with a bit of work though?

Don't know if it's worth it. Getting rid of custom constructors would be nice and would fix this, too, though.

[inducer]

I'll go ahead and merge this for the time being; it's easy enough to revert.

[inducer]

Thanks for making this!

[alexfikl]

This is inspired by a failure in the Stokes branch from pytentialinducer/pytential#162.

It started reproducibly raising exceptions as below. From what I can tell, the issue there was that the _memoize_dict from system_utils_get_base_kernel_matrix_lu_factorization somehow got into the pickling process.

E TypeError: no default __reduce__ due to non-trivial __cinit__
E when serializing dict item 'L'
E when serializing pytential.symbolic.pde.system_utils.LUFactorization state
E when serializing pytential.symbolic.pde.system_utils.LUFactorization object
E when serializing tuple item 0
E when serializing dict item (<class 'pytools._HasKwargs'>, frozenset({('order', None), ('hashable_kernel_arguments', (('mu', 1), ('nu', 0.4)))}))
E when serializing dict item '_memoize_dic_pytential.symbolic.pde.system_utils_get_base_kernel_matrix_lu_factorization'
E when serializing sumpy.kernel.BiharmonicKernel state
E when serializing sumpy.kernel.BiharmonicKernel object
E when serializing tuple item 0
E when serializing tuple item 1
E when serializing tuple item 0

I tried to implement a more generic __reduce__ with fields(self), but pretty much all the kernels have a custom constructor, so went with just copy pasting it a bit. Is there some better fix for this?

[inducer]

I'll try and take a look tomorrow. First I would like to understand how that LU factorization thing got into the pickling in the first place.

[alexfikl]

First I would like to understand how that LU factorization thing got into the pickling in the first place.

The function it's complaining about is here
https://github.com/isuruf/pytential/blob/1446481b5bc88fbfa5e1d15dc3f5a292eb0e5b18/pytential/symbolic/pde/system_utils.py#L560-L566

My understanding is that without a custom __reduce__, pickle just looks at __dict__ and the dict stored by memoize_on_first_arg got in there.

[inducer]

Ah OK, that makes sense. In that case, I think this is probably as good as it gets. I guess what I'd like better is if the kernels were dataclasses, but 🤷 this won't prevent that, and getting them to be dataclasses seems like a somewhat bigger lift.

[alexfikl]

I guess what I'd like better is if the kernels were dataclasses, but 🤷 this won't prevent that, and getting them to be dataclasses seems like a somewhat bigger lift.

All the ones in here are dataclasses, but that doesn't implement any special __reduce__. We could implement a generic one for all of them with a bit of work though?

[inducer]

We could implement a generic one for all of them with a bit of work though?

Don't know if it's worth it. Getting rid of custom constructors would be nice and would fix this, too, though.

[inducer]

I'll go ahead and merge this for the time being; it's easy enough to revert.

[inducer]

Thanks for making this!

[alexfikl]

This is inspired by a failure in the Stokes branch from pytentialinducer/pytential#162.

It started reproducibly raising exceptions as below. From what I can tell, the issue there was that the _memoize_dict from system_utils_get_base_kernel_matrix_lu_factorization somehow got into the pickling process.

E TypeError: no default __reduce__ due to non-trivial __cinit__
E when serializing dict item 'L'
E when serializing pytential.symbolic.pde.system_utils.LUFactorization state
E when serializing pytential.symbolic.pde.system_utils.LUFactorization object
E when serializing tuple item 0
E when serializing dict item (<class 'pytools._HasKwargs'>, frozenset({('order', None), ('hashable_kernel_arguments', (('mu', 1), ('nu', 0.4)))}))
E when serializing dict item '_memoize_dic_pytential.symbolic.pde.system_utils_get_base_kernel_matrix_lu_factorization'
E when serializing sumpy.kernel.BiharmonicKernel state
E when serializing sumpy.kernel.BiharmonicKernel object
E when serializing tuple item 0
E when serializing tuple item 1
E when serializing tuple item 0

I tried to implement a more generic __reduce__ with fields(self), but pretty much all the kernels have a custom constructor, so went with just copy pasting it a bit. Is there some better fix for this?

[inducer]

I'll try and take a look tomorrow. First I would like to understand how that LU factorization thing got into the pickling in the first place.

[alexfikl]

First I would like to understand how that LU factorization thing got into the pickling in the first place.

The function it's complaining about is here
https://github.com/isuruf/pytential/blob/1446481b5bc88fbfa5e1d15dc3f5a292eb0e5b18/pytential/symbolic/pde/system_utils.py#L560-L566

My understanding is that without a custom __reduce__, pickle just looks at __dict__ and the dict stored by memoize_on_first_arg got in there.

[inducer]

Ah OK, that makes sense. In that case, I think this is probably as good as it gets. I guess what I'd like better is if the kernels were dataclasses, but 🤷 this won't prevent that, and getting them to be dataclasses seems like a somewhat bigger lift.

[alexfikl]

I guess what I'd like better is if the kernels were dataclasses, but 🤷 this won't prevent that, and getting them to be dataclasses seems like a somewhat bigger lift.

All the ones in here are dataclasses, but that doesn't implement any special __reduce__. We could implement a generic one for all of them with a bit of work though?

[inducer]

We could implement a generic one for all of them with a bit of work though?

Don't know if it's worth it. Getting rid of custom constructors would be nice and would fix this, too, though.

[inducer]

I'll go ahead and merge this for the time being; it's easy enough to revert.

[inducer]

Thanks for making this!

[alexfikl]

This is inspired by a failure in the Stokes branch from pytentialinducer/pytential#162.

It started reproducibly raising exceptions as below. From what I can tell, the issue there was that the _memoize_dict from system_utils_get_base_kernel_matrix_lu_factorization somehow got into the pickling process.

E TypeError: no default __reduce__ due to non-trivial __cinit__
E when serializing dict item 'L'
E when serializing pytential.symbolic.pde.system_utils.LUFactorization state
E when serializing pytential.symbolic.pde.system_utils.LUFactorization object
E when serializing tuple item 0
E when serializing dict item (<class 'pytools._HasKwargs'>, frozenset({('order', None), ('hashable_kernel_arguments', (('mu', 1), ('nu', 0.4)))}))
E when serializing dict item '_memoize_dic_pytential.symbolic.pde.system_utils_get_base_kernel_matrix_lu_factorization'
E when serializing sumpy.kernel.BiharmonicKernel state
E when serializing sumpy.kernel.BiharmonicKernel object
E when serializing tuple item 0
E when serializing tuple item 1
E when serializing tuple item 0

I tried to implement a more generic __reduce__ with fields(self), but pretty much all the kernels have a custom constructor, so went with just copy pasting it a bit. Is there some better fix for this?

[inducer]

I'll try and take a look tomorrow. First I would like to understand how that LU factorization thing got into the pickling in the first place.

[alexfikl]

First I would like to understand how that LU factorization thing got into the pickling in the first place.

The function it's complaining about is here
https://github.com/isuruf/pytential/blob/1446481b5bc88fbfa5e1d15dc3f5a292eb0e5b18/pytential/symbolic/pde/system_utils.py#L560-L566

My understanding is that without a custom __reduce__, pickle just looks at __dict__ and the dict stored by memoize_on_first_arg got in there.

[inducer]

Ah OK, that makes sense. In that case, I think this is probably as good as it gets. I guess what I'd like better is if the kernels were dataclasses, but 🤷 this won't prevent that, and getting them to be dataclasses seems like a somewhat bigger lift.

[alexfikl]

I guess what I'd like better is if the kernels were dataclasses, but 🤷 this won't prevent that, and getting them to be dataclasses seems like a somewhat bigger lift.

All the ones in here are dataclasses, but that doesn't implement any special __reduce__. We could implement a generic one for all of them with a bit of work though?

[inducer]

We could implement a generic one for all of them with a bit of work though?

Don't know if it's worth it. Getting rid of custom constructors would be nice and would fix this, too, though.

[inducer]

I'll go ahead and merge this for the time being; it's easy enough to revert.

[inducer]

Thanks for making this!

[alexfikl]

This is inspired by a failure in the Stokes branch from pytentialinducer/pytential#162.

It started reproducibly raising exceptions as below. From what I can tell, the issue there was that the _memoize_dict from system_utils_get_base_kernel_matrix_lu_factorization somehow got into the pickling process.

E TypeError: no default __reduce__ due to non-trivial __cinit__
E when serializing dict item 'L'
E when serializing pytential.symbolic.pde.system_utils.LUFactorization state
E when serializing pytential.symbolic.pde.system_utils.LUFactorization object
E when serializing tuple item 0
E when serializing dict item (<class 'pytools._HasKwargs'>, frozenset({('order', None), ('hashable_kernel_arguments', (('mu', 1), ('nu', 0.4)))}))
E when serializing dict item '_memoize_dic_pytential.symbolic.pde.system_utils_get_base_kernel_matrix_lu_factorization'
E when serializing sumpy.kernel.BiharmonicKernel state
E when serializing sumpy.kernel.BiharmonicKernel object
E when serializing tuple item 0
E when serializing tuple item 1
E when serializing tuple item 0

I tried to implement a more generic __reduce__ with fields(self), but pretty much all the kernels have a custom constructor, so went with just copy pasting it a bit. Is there some better fix for this?

[inducer]

I'll try and take a look tomorrow. First I would like to understand how that LU factorization thing got into the pickling in the first place.

[alexfikl]

First I would like to understand how that LU factorization thing got into the pickling in the first place.

The function it's complaining about is here
https://github.com/isuruf/pytential/blob/1446481b5bc88fbfa5e1d15dc3f5a292eb0e5b18/pytential/symbolic/pde/system_utils.py#L560-L566

My understanding is that without a custom __reduce__, pickle just looks at __dict__ and the dict stored by memoize_on_first_arg got in there.

[inducer]

Ah OK, that makes sense. In that case, I think this is probably as good as it gets. I guess what I'd like better is if the kernels were dataclasses, but 🤷 this won't prevent that, and getting them to be dataclasses seems like a somewhat bigger lift.

[alexfikl]

I guess what I'd like better is if the kernels were dataclasses, but 🤷 this won't prevent that, and getting them to be dataclasses seems like a somewhat bigger lift.

All the ones in here are dataclasses, but that doesn't implement any special __reduce__. We could implement a generic one for all of them with a bit of work though?

[inducer]

We could implement a generic one for all of them with a bit of work though?

Don't know if it's worth it. Getting rid of custom constructors would be nice and would fix this, too, though.

[inducer]

I'll go ahead and merge this for the time being; it's easy enough to revert.

[inducer]

Thanks for making this!

[alexfikl]

This is inspired by a failure in the Stokes branch from pytentialinducer/pytential#162.

It started reproducibly raising exceptions as below. From what I can tell, the issue there was that the _memoize_dict from system_utils_get_base_kernel_matrix_lu_factorization somehow got into the pickling process.

E TypeError: no default __reduce__ due to non-trivial __cinit__
E when serializing dict item 'L'
E when serializing pytential.symbolic.pde.system_utils.LUFactorization state
E when serializing pytential.symbolic.pde.system_utils.LUFactorization object
E when serializing tuple item 0
E when serializing dict item (<class 'pytools._HasKwargs'>, frozenset({('order', None), ('hashable_kernel_arguments', (('mu', 1), ('nu', 0.4)))}))
E when serializing dict item '_memoize_dic_pytential.symbolic.pde.system_utils_get_base_kernel_matrix_lu_factorization'
E when serializing sumpy.kernel.BiharmonicKernel state
E when serializing sumpy.kernel.BiharmonicKernel object
E when serializing tuple item 0
E when serializing tuple item 1
E when serializing tuple item 0

I tried to implement a more generic __reduce__ with fields(self), but pretty much all the kernels have a custom constructor, so went with just copy pasting it a bit. Is there some better fix for this?

[inducer]

I'll try and take a look tomorrow. First I would like to understand how that LU factorization thing got into the pickling in the first place.

[alexfikl]

First I would like to understand how that LU factorization thing got into the pickling in the first place.

The function it's complaining about is here
https://github.com/isuruf/pytential/blob/1446481b5bc88fbfa5e1d15dc3f5a292eb0e5b18/pytential/symbolic/pde/system_utils.py#L560-L566

My understanding is that without a custom __reduce__, pickle just looks at __dict__ and the dict stored by memoize_on_first_arg got in there.

[inducer]

Ah OK, that makes sense. In that case, I think this is probably as good as it gets. I guess what I'd like better is if the kernels were dataclasses, but 🤷 this won't prevent that, and getting them to be dataclasses seems like a somewhat bigger lift.

[alexfikl]

I guess what I'd like better is if the kernels were dataclasses, but 🤷 this won't prevent that, and getting them to be dataclasses seems like a somewhat bigger lift.

All the ones in here are dataclasses, but that doesn't implement any special __reduce__. We could implement a generic one for all of them with a bit of work though?

[inducer]

We could implement a generic one for all of them with a bit of work though?

Don't know if it's worth it. Getting rid of custom constructors would be nice and would fix this, too, though.

[inducer]

I'll go ahead and merge this for the time being; it's easy enough to revert.

[inducer]

Thanks for making this!

[alexfikl]

This is inspired by a failure in the Stokes branch from pytentialinducer/pytential#162.

It started reproducibly raising exceptions as below. From what I can tell, the issue there was that the _memoize_dict from system_utils_get_base_kernel_matrix_lu_factorization somehow got into the pickling process.

E TypeError: no default __reduce__ due to non-trivial __cinit__
E when serializing dict item 'L'
E when serializing pytential.symbolic.pde.system_utils.LUFactorization state
E when serializing pytential.symbolic.pde.system_utils.LUFactorization object
E when serializing tuple item 0
E when serializing dict item (<class 'pytools._HasKwargs'>, frozenset({('order', None), ('hashable_kernel_arguments', (('mu', 1), ('nu', 0.4)))}))
E when serializing dict item '_memoize_dic_pytential.symbolic.pde.system_utils_get_base_kernel_matrix_lu_factorization'
E when serializing sumpy.kernel.BiharmonicKernel state
E when serializing sumpy.kernel.BiharmonicKernel object
E when serializing tuple item 0
E when serializing tuple item 1
E when serializing tuple item 0

I tried to implement a more generic __reduce__ with fields(self), but pretty much all the kernels have a custom constructor, so went with just copy pasting it a bit. Is there some better fix for this?

[inducer]

I'll try and take a look tomorrow. First I would like to understand how that LU factorization thing got into the pickling in the first place.

[alexfikl]

First I would like to understand how that LU factorization thing got into the pickling in the first place.

The function it's complaining about is here
https://github.com/isuruf/pytential/blob/1446481b5bc88fbfa5e1d15dc3f5a292eb0e5b18/pytential/symbolic/pde/system_utils.py#L560-L566

My understanding is that without a custom __reduce__, pickle just looks at __dict__ and the dict stored by memoize_on_first_arg got in there.

[inducer]

Ah OK, that makes sense. In that case, I think this is probably as good as it gets. I guess what I'd like better is if the kernels were dataclasses, but 🤷 this won't prevent that, and getting them to be dataclasses seems like a somewhat bigger lift.

[alexfikl]

I guess what I'd like better is if the kernels were dataclasses, but 🤷 this won't prevent that, and getting them to be dataclasses seems like a somewhat bigger lift.

All the ones in here are dataclasses, but that doesn't implement any special __reduce__. We could implement a generic one for all of them with a bit of work though?

[inducer]

We could implement a generic one for all of them with a bit of work though?

Don't know if it's worth it. Getting rid of custom constructors would be nice and would fix this, too, though.

[inducer]

I'll go ahead and merge this for the time being; it's easy enough to revert.

[inducer]

Thanks for making this!