interpreters/python: Add Python's port to NuttX

[tmedicci]

Summary

Say Hello to Python!

This is the NuttX's port of Python (CPython)!

Initial support of Python includes building the Python's static library and the python (Programs/python.c) application. Python's modules are stored in pyc (byte-code file) and loaded as needed from flash.

Considerations

Please note that this is the initial port of one of the most used programming languages for NuttX (and, maybe, for embedded RTOSes). There is a long list of To-Do to make it fully functional. NuttX is perhaps one of the best embedded RTOSes for working with Python due to its POSIX Compatibility and on-demand paging (https://nuttx.apache.org/docs/latest/components/paging.html#ondemandpaging). I hope this PR engages NuttX community to keep improving Python for use in embedded applications with NuttX.

Please consider that:

1. This PR aims to make it work with rv-virt (RISC-V QEMU). That being said, its interpreters/python/config.site and interpreters/python/Setup.local are somehow tied with the features enabled by the attached defconfig (Yes, I will submit a PR on https://github.com/apache/nuttx for it)
2. The "To-Do" list includes, for instance, generating these files dynamically depending on what features are enabled.
3. This initial implementation uses Make-based build system. Contributions for CMake are welcomed!
4. The pyc modules are loaded from a ROMFS image at startup. Therefore, we need an application to mount it (that's the mount_modules role!).
5. Also, the user needs to set PYTHONHOME and PYTHON_BASIC_REPL environment variables.
6. The "To-Do" list should include a wrapper application that 1) mounts the modules, 2) set the env vars and 3) runs main python application.

Impact

Provides initial support for Python on NuttX. Please note that it supports, initially, only the rv-virt (RISC-V QEMU) "board".

Testing

Internal CI testing + rv-virt:cpython.

Building

Create a defconfig file at nuttx/boards/risc-v/qemu-rv/rv-virt/configs/cpython/defconfig with the following content:

#
# This file is autogenerated: PLEASE DO NOT EDIT IT.
#
# You can use "make menuconfig" to make any modifications to the installed .config file.
# You can then do "make savedefconfig" to generate a new defconfig file that includes your
# modifications.
#
# CONFIG_DISABLE_OS_API is not set
# CONFIG_NSH_DISABLE_LOSMART is not set
CONFIG_16550_ADDRWIDTH=0
CONFIG_16550_UART0=y
CONFIG_16550_UART0_BASE=0x10000000
CONFIG_16550_UART0_CLOCK=3686400
CONFIG_16550_UART0_IRQ=37
CONFIG_16550_UART0_SERIAL_CONSOLE=y
CONFIG_16550_UART=y
CONFIG_ARCH="risc-v"
CONFIG_ARCH_BOARD="rv-virt"
CONFIG_ARCH_BOARD_QEMU_RV_VIRT=y
CONFIG_ARCH_CHIP="qemu-rv"
CONFIG_ARCH_CHIP_QEMU_RV32=y
CONFIG_ARCH_CHIP_QEMU_RV=y
CONFIG_ARCH_CHIP_QEMU_RV_ISA_A=y
CONFIG_ARCH_CHIP_QEMU_RV_ISA_C=y
CONFIG_ARCH_CHIP_QEMU_RV_ISA_M=y
CONFIG_ARCH_FLOAT_H=y
CONFIG_ARCH_INTERRUPTSTACK=2048
CONFIG_ARCH_RISCV=y
CONFIG_ARCH_STACKDUMP=y
CONFIG_BOARDCTL_POWEROFF=y
CONFIG_BOARDCTL_ROMDISK=y
CONFIG_BOARD_LOOPSPERMSEC=6366
CONFIG_BUILTIN=y
CONFIG_DEBUG_FEATURES=y
CONFIG_DEBUG_FULLOPT=y
CONFIG_DEBUG_SYMBOLS=y
CONFIG_DEV_URANDOM=y
CONFIG_DEV_ZERO=y
CONFIG_ELF=y
CONFIG_EVENT_FD=y
CONFIG_EXAMPLES_HELLO=y
CONFIG_EXPERIMENTAL=y
CONFIG_FRAME_POINTER=y
CONFIG_FS_HOSTFS=y
CONFIG_FS_PROCFS=y
CONFIG_FS_ROMFS=y
CONFIG_FS_TMPFS=y
CONFIG_IDLETHREAD_STACKSIZE=4096
CONFIG_INIT_ENTRYPOINT="nsh_main"
CONFIG_INIT_STACKSIZE=4096
CONFIG_INTERPRETER_CPYTHON=y
CONFIG_LIBC_ENVPATH=y
CONFIG_LIBC_EXECFUNCS=y
CONFIG_LIBC_LOCALE=y
CONFIG_LIBC_LOCALE_GETTEXT=y
CONFIG_LIBC_PERROR_STDOUT=y
CONFIG_LIBC_STRERROR=y
CONFIG_LIBM_NEWLIB=y
CONFIG_LIB_SYSCALL=y
CONFIG_LIB_ZLIB=y
CONFIG_NFILE_DESCRIPTORS_PER_BLOCK=6
CONFIG_NSH_ARCHINIT=y
CONFIG_NSH_BUILTIN_APPS=y
CONFIG_NSH_FILEIOSIZE=512
CONFIG_NSH_LINELEN=300
CONFIG_NSH_READLINE=y
CONFIG_NSH_VARS=y
CONFIG_PATH_INITIAL="/system/bin"
CONFIG_PIPES=y
CONFIG_PSEUDOFS_SOFTLINKS=y
CONFIG_RAM_SIZE=33554432
CONFIG_RAM_START=0x80000000
CONFIG_READLINE_CMD_HISTORY=y
CONFIG_RISCV_SEMIHOSTING_HOSTFS=y
CONFIG_RR_INTERVAL=200
CONFIG_SCHED_WAITPID=y
CONFIG_SERIAL_UART_ARCH_MMIO=y
CONFIG_STACK_COLORATION=y
CONFIG_START_MONTH=12
CONFIG_START_YEAR=2021
CONFIG_SYMTAB_ORDEREDBYNAME=y
CONFIG_SYSTEM_NSH=y
CONFIG_TESTING_GETPRIME=y
CONFIG_TESTING_OSTEST=y
CONFIG_TIMER_FD=y
CONFIG_TLS_NELEM=10
CONFIG_TLS_TASK_NELEM=10
CONFIG_USEC_PER_TICK=1000

Build it with:

make distclean && ./tools/configure.sh rv-virt:cpython && make -j$(nproc)

Testing

Run it with RISC-V QEMU:

qemu-system-riscv32 -semihosting -M virt,aclint=on -cpu rv32 -smp 1 -bios none -kernel nuttx -nographic

On NuttX:

nsh> mount_modules
Mounting ROMFS filesystem at target=/usr/local/lib/ with source=/dev/ram1
nsh> export PYTHONHOME /usr/local
nsh> export PYTHON_BASIC_REPL 1
nsh> python

Check the following record to run Python

(yes, you can copy the commands from there)

[nuttxpr]

[Experimental Bot, please feedback here]

This PR does not fully meet the NuttX requirements, although it presents a significant feature addition. Here's a breakdown:

Summary: While the summary explains the what and why, it lacks details on how the Python integration works internally. More specifics on memory management, interaction with the OS, and the build process would be beneficial. The "Considerations" section highlights important limitations and to-dos, which is good, but some of these should be addressed before merging.

Impact: The impact section is incomplete. While it mentions the target architecture, it needs to address all the points:

• Impact on user: YES. Users will need to set environment variables, mount filesystems, and potentially adapt their applications. The required steps should be documented.
• Impact on build: YES. A new configuration (cpython) and additional build steps are introduced. This needs detailed explanation, ideally with changes to the build system documentation.
• Impact on hardware: YES. Currently, it only supports rv-virt. This needs explicit mention. Future hardware support should be discussed.
• Impact on documentation: YES. Documentation is required for building, configuring, and using the Python integration. This is crucial for users.
• Impact on security: Potentially YES. Introducing a complex interpreter like Python can have security implications. This needs investigation and documentation.
• Impact on compatibility: Potentially YES. The use of ROMFS and specific environment variables may impact compatibility with other NuttX configurations.
• Anything else to consider: YES. The long "To-Do" list suggests this is an early-stage integration. Consider whether it's ready for merging in its current state or if it should be staged.

Testing: The testing section is insufficient.

• Build Host(s): Missing information about the host OS, compiler version, etc.
• Target(s): While rv-virt:cpython is mentioned, more details on the configuration are needed. The provided defconfig is helpful but doesn't replace a detailed description.
• Testing logs before change: N/A (as expected for a new feature), but should be stated explicitly.
• Testing logs after change: The asciinema recording is a good demonstration, but it doesn't replace textual logs showing the Python interpreter's output and any relevant NuttX messages. Simple Python scripts demonstrating core functionality should be included and their output logged.

Recommendations:

• Expand the Summary: Provide more technical details on the integration mechanism.
• Complete the Impact section: Address all points thoroughly.
• Improve the Testing section: Provide detailed host information, more comprehensive test cases with logged output, and clear instructions for reproducing the tests.
• Address the "To-Do" list: Prioritize essential items, like dynamic configuration generation and a wrapper application, and consider whether they should be completed before merging.
• Provide documentation: Document the build process, configuration options, and usage instructions.
• Consider staging the PR: If the "To-Do" list is extensive, consider breaking the integration into smaller, more manageable PRs.

This PR has great potential but needs more work to meet NuttX's standards for completeness and maintainability. Addressing these points will make the integration more robust and easier for the community to adopt.

[anjiahao1]

Great, nice work!

[anjiahao1]

need add cmake?

[tmedicci]

need add cmake?

Yes!

There is a long list of to-do, but I preferred to submit upstream because I need more hands working on it (and I think we can engage the NuttX community by doing that).

NuttX would be one of the first RTOS supporting Python ;)

[hartmannathan]

Very cool! I'm in favor of making an early version available for testing and further development by the community. Should it be behind a "EXPERIMENTAL" config, to set the right expectations?

[tmedicci]

Should it be behind a "EXPERIMENTAL" config, to set the right expectations?

Yes, it makes total sense!

[tmedicci]

Error for esp32-devkitc:nxdiag does not seem to be related to this PR (#2879 (review)). Checking it.

[tmedicci]

Error for esp32-devkitc:nxdiag does not seem to be related to this PR (#2879 (review)). Checking it.

I ran the CI locally (apache/nuttx#14601 (comment)) and it didn't fail. However, it may be related to an issue that I fixed at f064a19 (just sent along with this PR)

[fdcavalcanti]

This is awesome.
For the future, I think a good addition would be having the test directory available, so we can the interpreter unit tests in target. Since those are available already, we could use it!