Add support for WebAssembly System Interface (wasm32-wasi)

[tiran]

BPO	46315
Nosy	@tiran, @pmp-p, @kumaraditya303
PRs	bpo-46315: Add ifdef HAVE_ feature checks for WASI compatibility (GH-30507) #30507 bpo-46315: Use fopencookie() to avoid dup() in _PyTokenizer_FindEncodingFilename #32033 bpo-46315: Use fopencookie only on Emscripten 3.x and newer (GH-32266) #32266

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/tiran'
closed_at = None
created_at = <Date 2022-01-09.14:47:43.508>
labels = ['type-feature', 'build', '3.11']
title = 'Add support for WebAssembly System Interface (wasm32-wasi)'
updated_at = <Date 2022-04-02.21:11:46.918>
user = 'https://github.com/tiran'

bugs.python.org fields:

activity = <Date 2022-04-02.21:11:46.918>
actor = 'christian.heimes'
assignee = 'christian.heimes'
closed = False
closed_date = None
closer = None
components = ['Build']
creation = <Date 2022-01-09.14:47:43.508>
creator = 'christian.heimes'
dependencies = []
files = []
hgrepos = []
issue_num = 46315
keywords = ['patch']
message_count = 4.0
messages = ['410153', '410198', '410470', '416588']
nosy_count = 3.0
nosy_names = ['christian.heimes', 'pmpp', 'kumaraditya']
pr_nums = ['30507', '32033', '32266']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue46315'
versions = ['Python 3.11']

[tiran]

WASI is another WebAssembly platform similar to Emscripten (bpo-40280). Simply speaking Emscripten binaries (wasm32-emscripten) run inside a browser while WASI binaries target standalone runtimes like wasmtime [2][3] on the host. The lines are a bit blurry, as it is possible to run WASI binaries in the browser with help of JS-polyfills. WASI provides compile once, run anyway with JIT/AOT and sandboxing.

WASI is still under development and is lacking several core features:

• threading support and pthread API
• sockets
• signals are emulated (_WASI_EMULATED_SIGNAL and -lwasi-emulated-signal)
• DAC APIs like chmod, umask
• user-related APIs like pwd, grp
• dup(), dup2(), F_DUPFD

For 3.11 I plan to fix our use of #ifdef HAVE_FEATURE to make it easier to experiment with WASI. The pthread APIs need stubs, which I won't commit to 3.11 upstream yet.

[1] https://wasi.dev/
[2] https://github.com/bytecodealliance/wasmtime
[3] https://hacks.mozilla.org/2019/03/standardizing-wasi-a-webassembly-system-interface/

[tiran]

dup() is required by _PyTokenizer_FindEncodingFilename(). I came up with this hack:

// from wasi-libc libc-top-half/musl/src/internal/stdio_impl.h
struct _IO_FILE {
 unsigned flags;
 unsigned char *rpos, *rend;
 int (*close)(FILE *);
 unsigned char *wend, *wpos;
 // incomplete
};

static int
dummy_close(FILE *fp) {
 return 0;
};

static FILE *
_Py_fdopen_borrow(int fd, const char *mode) {
 FILE *fp = fdopen(fd, mode);
 ((struct _IO_FILE*)fp)->close = dummy_close;
 return fp;
}

keithw on #wasi pointed out that fopencookie() can archive the same outcome without resorting to ABI-specific hack. A trivial implementation is straight forward:

typedef union {
 void *cookie;
 int fd;
} borrowed;

static ssize_t
borrow_read(void *cookie, char *buf, size_t size)
{
borrowed b;
b.cookie = cookie;
return read(b.fd, (void *)buf, size);
}

static ssize_t
borrow_write(void *cookie, const char *buf, size_t size)
{
 errno = ENOTSUP;
 return -1;
}

static int
borrow_seek(void *cookie, off_t *off, int whence)
{
 borrowed b;
 b.cookie = cookie;
 off_t pos;
 pos = lseek(b.fd, *off, whence);
 if (pos == (off_t)-1) {
 return -1;
 } else {
 *off = pos;
 return 0; 
 }
}

static int
borrow_close(void *cookie)
{
 // does not close(fd)
 return 0;
}

FILE *
_Py_fdopen_borrow(int fd, const char *mode) {
 // only support read for now
 if (strcmp(mode, "r") != 0) {
 return NULL;
 }
 cookie_io_functions_t cookie_io = {
 borrow_read, borrow_write, borrow_seek, borrow_close
 };
 	// cookie is just the fd
 borrowed b;
 b.fd = fd;
 return fopencookie(b.cookie, "r", cookie_io);
}

[tiran]

New changeset a6ca8ee by Christian Heimes in branch 'main':
bpo-46315: Add ifdef HAVE_ feature checks for WASI compatibility (GH-30507)
a6ca8ee

[tiran]

New changeset 3df0e63 by Christian Heimes in branch 'main':
bpo-46315: Use fopencookie only on Emscripten 3.x and newer (GH-32266)
3df0e63