Support syntax for passing struct in CPython

[ubaidsk]

Related - #1799.

$ cat examples/expr2.py # file: main.py
from lpython import i32, dataclass
from numpy import empty
@dataclass
class Foo:
x: i32
y: i32
def init(foos: Foo[:]) -> None:
foos[0] = Foo(1, 2)
def main() -> None:
foos: Foo[1] = empty(1, dtype=Foo)
init(foos)
print("foos[0].x =", foos[0].x)
main()
$ lpython examples/expr2.py foos[0].x = 1
$ python examples/expr2.py foos[0].x = 1

[czgdp1807]

I am not sure if it fixes #1799 entirely. Could you please add tests from that issue to actually see if they work with your change?

[ubaidsk]

@czgdp1807 do you mean to add the example in #1799 (comment) as an integration_test?

[czgdp1807]

Yes.

[ubaidsk]

Could you please add tests from that issue to actually see if they work with your change?

@czgdp1807 added it. Thank you for the guidance.

[certik]

I would just support the Array[] syntax for arrays of structs, like this:

Suggested change

	foos: Foo[1] =empty(1, dtype=Foo)
	foos: Array[Foo, 1] =empty(1, dtype=Foo)

Then it should be possible to implement this cleanly in lpython.py as well as LPython.

[certik]

I think as a quick fix it is fine. I think it might break if Foo already defines __class_getitem__ or if it supports the [] like syntax. Right now LPython doesn't support it, so it's fine. But in the long run I think we will have to support the Array[Foo] syntax.

[dylon]

This does not work in CPython, right now, because __class_getitem__ returns None, which is not a valid type hint:

File "/home/dylon/Workspace/copperhead/demo-apps/lab4/emulation/gsi_emulation0.py", line 515, in gdl_context_desc_get
c_p_pointer(ctx_desc_ptr, gdl_context_desc[:])
File "/home/dylon/mambaforge/envs/lab4_emulation/lib/python3.10/site-packages/gsi_emulation/ltypes.py", line 488, in c_p_pointer
targettype_ptr = ctypes.POINTER(convert_type_to_ctype(targettype))
File "/home/dylon/mambaforge/envs/lab4_emulation/lib/python3.10/site-packages/gsi_emulation/ltypes.py", line 223, in convert_type_to_ctype
raise NotImplementedError("Type cannot be None")

[ubaidsk]

This does not work in CPython, right now, because class_getitem returns None, which is not a valid type hint:

Please, could you share an example which fails?

[dylon]

Sure, this reminds me of a bug I need to report that I worked around: #1817

With the workaround above, the following fails with the message above:

fromlpythonimportCPtr, c_p_pointer, p_c_pointer, dataclass, empty_c_void_p, pointer, Pointerfromnumpyimportempty@dataclassclassFoo:
passdefbar(foos_ptr: CPtr) ->None:
foos: Pointer[Foo[:]] =c_p_pointer(foos_ptr, Foo[:])
defmain() ->None:
foos: Foo[1] =empty(1, dtype=Foo)
foos_ptr: CPtr=empty_c_void_p()
p_c_pointer(pointer(foos), foos_ptr)
bar(foos_ptr)
main()

$ PYTHONPATH=/home/dylon/Workspace/lpython/src/runtime/lpython python /var/tmp/main.py
Traceback (most recent call last):
File "/var/tmp/main.py", line 18, in <module>
main()
File "/var/tmp/main.py", line 16, in main
bar(foos_ptr)
File "/var/tmp/main.py", line 10, in bar
foos: Pointer[Foo[:]] = c_p_pointer(foos_ptr, Foo[:])
File "/home/dylon/Workspace/lpython/src/runtime/lpython/lpython.py", line 524, in c_p_pointer
targettype_ptr = convert_type_to_ctype(targettype)
File "/home/dylon/Workspace/lpython/src/runtime/lpython/lpython.py", line 255, in convert_type_to_ctype
raise NotImplementedError("Type cannot be None")
NotImplementedError: Type cannot be None

[certik]

It's weird that it works at our CI. I wonder if it is Python version dependent. Or our CI is broken?

[dylon]

This almost works, but leads to another bug (again, branching off my workaround from above): #1818

defdataclass(arg):
def__class_getitem__(idxs):
returnArray(arg, idxs)
arg.__class_getitem__=__class_getitem__returnpy_dataclass(arg)

[certik]

I created a dedicated issue for all the bugs reported in the last few comments above: #1819, so that we don't forget about this. We'll investigate and fix it.

[czgdp1807]

The code posted in #1814 (comment) will not work with CPython because for dataclasses because numpy just creates an array of Python objects. See below,

fromlpythonimportdataclass, i32fromnumpyimportempty@dataclassclassFoo:
x: i32y: i32@dataclassclassFoe:
x: i32y: i32z: i32defmain() ->None:
foos=empty(2, dtype=Foo)
print(foos.dtype)
foos[0] =Foo(1, 2)
foos[1] =Foe(3, 4, 5)
print(foos[0], foos[1])
main()

(lp) 15:11:37:~/lpython_project/lpython % python /Users/czgdp1807/lpython_project/debug.py
object
Foo(x=1, y=2) Foe(x=3, y=4, z=5)

So we won't be able to use ctypes.Structure here because foos[0], foos[1] can easily store objects of different type. Also, foos.dtype is just a Python object. However I can make the following work,

fromlpythonimportCPtr, c_p_pointer, p_c_pointer, dataclass, empty_c_void_p, pointer, Pointerfromnumpyimportempty@dataclassclassFoo:
passdefbar(foos_ptr: CPtr) ->None:
foos: Pointer[Foo[:]] =c_p_pointer(foos_ptr, Foo[:])
defmain() ->None:
foos: Foo[1] =empty(1, dtype=Foo)
foos_ptr: CPtr=empty_c_void_p()
p_c_pointer(pointer(foos, Foo), foos_ptr)
bar(foos_ptr)
main()

with the following diff,

diff --git a/src/runtime/lpython/lpython.py b/src/runtime/lpython/lpython.py
index c6c53cf31..a82a190f8 100644
--- a/src/runtime/lpython/lpython.py+++ b/src/runtime/lpython/lpython.py@@ -448,6 +448,8 @@ def pointer(x, type_=None):
type_ = type(x)
from numpy import ndarray
if isinstance(x, ndarray):
+ if py_is_dataclass(type_):+ return x
return x.ctypes.data_as(ctypes.POINTER(convert_numpy_dtype_to_ctype(x.dtype)))
else:
if type_ == i8:

[czgdp1807]

Also see #1822. Luck by chance I also ended up with #1814 (comment).