std: implement sendfile on linux by terinjokes · Pull Request #4131 · ziglang/zig

terinjokes · 2020-01-10T10:26:29Z

This changset adds a sendfile(2) syscall bindings to the linux bits
component. Where available, the sendfile64(2) syscall will be
transparently called.

A wrapping function has also been added to the std.os to transform
errno returns to Zig errors.

daurnimator

Should also add a prototype in std/c.zig so that libc's function is used when linking against libc.

daurnimator · 2020-01-10T11:40:44Z

+ while (true) {
+ var rc: usize = undefined;
+ var err: usize = undefined;
+ if (builtin.os == .linux) {


This should just be an unconditional system.sendfile. If someone attempts to call this function on a system without it then they will get a compile error indicating as much.

terinjokes · 2020-01-10T16:08:39Z

@daurnimator I had added it to c.zig locally, but building anything linking against C fails with an unrelated error, so I can't test if my changes are good.

terinjokes · 2020-01-10T18:14:08Z

/usr/lib/zig/std/os.zig:3301:60: error: cast discards const qualifier
switch (errno(system.setsockopt(fd, level, optname, opt.ptr, @intCast(socklen_t, opt.len)))) {

terinjokes · 2020-01-10T19:49:46Z

Per discussion on IRC, this PR can likely be extended with macOS and FreeBSD support:

Extend the os.zig wrapper to accept hdr, sbytes, and flags parameters.
Handle writing headers and trailers on systems where this isn't the default. Document what systems they're written atomically.
The BSD and Linux handle updates to the offset pointer different, so I may need to also document the behavior and what would be portable.

Windows has TransmitFile as part of mswsock, which looks similar. Will have to investigate farther.

andrewrk · 2020-01-10T21:28:19Z

+ /// There was insufficient memory for reading from infd.
+ NoMem,


I'm surprised (and disappointed) this error is even possible; the kernel should be able to make this "just work" even if it is forced to use a small stack-based buffer due to heap memory pressure.
By convention, ENOMEM in zig turns into one of these two things:
error.OutOfMemory if it is userspace heap memory that is exhausted
error.SystemResources if it is kernel space memory or other resources that are unavailable.
In this case I believe it would be error.SystemResources.

Reading through the kernel sources, this actually comes from files that have a mandatory lock, and the underlying splice() cannot return ENOMEM.

@shawnl Thanks, I can update the comment. I'll likely take @andrewrk's recommendation of using error.SystemResources in this case.

terinjokes · 2020-01-20T02:32:43Z

I've pushed updates to the Linux support to accept the wider *BSD parameters, and to send the header and trailer iovecs. I doubt I have implemented the behavior to match correctly.

Next up to to add support sendfile(2) from FreeBSD libc. Then I'll be in a better place to verify the behavior is the same.

This changset adds a `sendfile(2)` syscall bindings to the linux bits component. Where available, the `sendfile64(2)` syscall will be transparently called. A wrapping function has also been added to the std.os to transform errno returns to Zig errors. Change-Id: I86769fc4382c0771e3656e7b21137bafd99a4411

terinjokes · 2020-01-21T03:40:18Z

+ trailers: [*]iovec_const,
+ trl_cnt: c_int,
+};
+pub extern "c" fn sendfile(fd: c_int, s: c_int, offset: u64, nbytes: usize, sf_hdtr: ?*sf_hdtr, sbytes: ?*u64, flags: c_int) c_int;


I need to double check that the offset parameters here should be u64 instead of off_t.

I don't think zig should support 32-bit off_t, in which case they are the same, but yeah probably still use off_t even if it must be u64.

andrewrk · 2020-03-03T07:28:33Z

Thanks! This is merged into #4611.

terinjokes · 2020-03-03T07:30:19Z

Sorry for not getting this as polished as I liked. Thank you for taking the time to carry it through.

terinjokes · 2020-01-10T10:26:29Z

This changset adds a sendfile(2) syscall bindings to the linux bits
component. Where available, the sendfile64(2) syscall will be
transparently called.

A wrapping function has also been added to the std.os to transform
errno returns to Zig errors.

daurnimator

Should also add a prototype in std/c.zig so that libc's function is used when linking against libc.

daurnimator · 2020-01-10T11:40:44Z

+ while (true) {
+ var rc: usize = undefined;
+ var err: usize = undefined;
+ if (builtin.os == .linux) {


This should just be an unconditional system.sendfile. If someone attempts to call this function on a system without it then they will get a compile error indicating as much.

terinjokes · 2020-01-10T16:08:39Z

@daurnimator I had added it to c.zig locally, but building anything linking against C fails with an unrelated error, so I can't test if my changes are good.

terinjokes · 2020-01-10T18:14:08Z

/usr/lib/zig/std/os.zig:3301:60: error: cast discards const qualifier
switch (errno(system.setsockopt(fd, level, optname, opt.ptr, @intCast(socklen_t, opt.len)))) {

terinjokes · 2020-01-10T19:49:46Z

Per discussion on IRC, this PR can likely be extended with macOS and FreeBSD support:

Extend the os.zig wrapper to accept hdr, sbytes, and flags parameters.
Handle writing headers and trailers on systems where this isn't the default. Document what systems they're written atomically.
The BSD and Linux handle updates to the offset pointer different, so I may need to also document the behavior and what would be portable.

Windows has TransmitFile as part of mswsock, which looks similar. Will have to investigate farther.

andrewrk · 2020-01-10T21:28:19Z

+ /// There was insufficient memory for reading from infd.
+ NoMem,


I'm surprised (and disappointed) this error is even possible; the kernel should be able to make this "just work" even if it is forced to use a small stack-based buffer due to heap memory pressure.
By convention, ENOMEM in zig turns into one of these two things:
error.OutOfMemory if it is userspace heap memory that is exhausted
error.SystemResources if it is kernel space memory or other resources that are unavailable.
In this case I believe it would be error.SystemResources.

Reading through the kernel sources, this actually comes from files that have a mandatory lock, and the underlying splice() cannot return ENOMEM.

@shawnl Thanks, I can update the comment. I'll likely take @andrewrk's recommendation of using error.SystemResources in this case.

terinjokes · 2020-01-20T02:32:43Z

I've pushed updates to the Linux support to accept the wider *BSD parameters, and to send the header and trailer iovecs. I doubt I have implemented the behavior to match correctly.

Next up to to add support sendfile(2) from FreeBSD libc. Then I'll be in a better place to verify the behavior is the same.

This changset adds a `sendfile(2)` syscall bindings to the linux bits component. Where available, the `sendfile64(2)` syscall will be transparently called. A wrapping function has also been added to the std.os to transform errno returns to Zig errors. Change-Id: I86769fc4382c0771e3656e7b21137bafd99a4411

terinjokes · 2020-01-21T03:40:18Z

+ trailers: [*]iovec_const,
+ trl_cnt: c_int,
+};
+pub extern "c" fn sendfile(fd: c_int, s: c_int, offset: u64, nbytes: usize, sf_hdtr: ?*sf_hdtr, sbytes: ?*u64, flags: c_int) c_int;


I need to double check that the offset parameters here should be u64 instead of off_t.

I don't think zig should support 32-bit off_t, in which case they are the same, but yeah probably still use off_t even if it must be u64.

andrewrk · 2020-03-03T07:28:33Z

Thanks! This is merged into #4611.

terinjokes · 2020-03-03T07:30:19Z

Sorry for not getting this as polished as I liked. Thank you for taking the time to carry it through.

terinjokes · 2020-01-10T10:26:29Z

This changset adds a sendfile(2) syscall bindings to the linux bits
component. Where available, the sendfile64(2) syscall will be
transparently called.

A wrapping function has also been added to the std.os to transform
errno returns to Zig errors.

daurnimator

Should also add a prototype in std/c.zig so that libc's function is used when linking against libc.

daurnimator · 2020-01-10T11:40:44Z

+ while (true) {
+ var rc: usize = undefined;
+ var err: usize = undefined;
+ if (builtin.os == .linux) {


This should just be an unconditional system.sendfile. If someone attempts to call this function on a system without it then they will get a compile error indicating as much.

terinjokes · 2020-01-10T16:08:39Z

@daurnimator I had added it to c.zig locally, but building anything linking against C fails with an unrelated error, so I can't test if my changes are good.

terinjokes · 2020-01-10T18:14:08Z

/usr/lib/zig/std/os.zig:3301:60: error: cast discards const qualifier
switch (errno(system.setsockopt(fd, level, optname, opt.ptr, @intCast(socklen_t, opt.len)))) {

terinjokes · 2020-01-10T19:49:46Z

Per discussion on IRC, this PR can likely be extended with macOS and FreeBSD support:

Extend the os.zig wrapper to accept hdr, sbytes, and flags parameters.
Handle writing headers and trailers on systems where this isn't the default. Document what systems they're written atomically.
The BSD and Linux handle updates to the offset pointer different, so I may need to also document the behavior and what would be portable.

Windows has TransmitFile as part of mswsock, which looks similar. Will have to investigate farther.

andrewrk · 2020-01-10T21:28:19Z

+ /// There was insufficient memory for reading from infd.
+ NoMem,


I'm surprised (and disappointed) this error is even possible; the kernel should be able to make this "just work" even if it is forced to use a small stack-based buffer due to heap memory pressure.
By convention, ENOMEM in zig turns into one of these two things:
error.OutOfMemory if it is userspace heap memory that is exhausted
error.SystemResources if it is kernel space memory or other resources that are unavailable.
In this case I believe it would be error.SystemResources.

Reading through the kernel sources, this actually comes from files that have a mandatory lock, and the underlying splice() cannot return ENOMEM.

@shawnl Thanks, I can update the comment. I'll likely take @andrewrk's recommendation of using error.SystemResources in this case.

terinjokes · 2020-01-20T02:32:43Z

I've pushed updates to the Linux support to accept the wider *BSD parameters, and to send the header and trailer iovecs. I doubt I have implemented the behavior to match correctly.

Next up to to add support sendfile(2) from FreeBSD libc. Then I'll be in a better place to verify the behavior is the same.

This changset adds a `sendfile(2)` syscall bindings to the linux bits component. Where available, the `sendfile64(2)` syscall will be transparently called. A wrapping function has also been added to the std.os to transform errno returns to Zig errors. Change-Id: I86769fc4382c0771e3656e7b21137bafd99a4411

terinjokes · 2020-01-21T03:40:18Z

+ trailers: [*]iovec_const,
+ trl_cnt: c_int,
+};
+pub extern "c" fn sendfile(fd: c_int, s: c_int, offset: u64, nbytes: usize, sf_hdtr: ?*sf_hdtr, sbytes: ?*u64, flags: c_int) c_int;


I need to double check that the offset parameters here should be u64 instead of off_t.

I don't think zig should support 32-bit off_t, in which case they are the same, but yeah probably still use off_t even if it must be u64.

andrewrk · 2020-03-03T07:28:33Z

Thanks! This is merged into #4611.

terinjokes · 2020-03-03T07:30:19Z

Sorry for not getting this as polished as I liked. Thank you for taking the time to carry it through.

terinjokes · 2020-01-10T10:26:29Z

This changset adds a sendfile(2) syscall bindings to the linux bits
component. Where available, the sendfile64(2) syscall will be
transparently called.

A wrapping function has also been added to the std.os to transform
errno returns to Zig errors.

daurnimator

Should also add a prototype in std/c.zig so that libc's function is used when linking against libc.

daurnimator · 2020-01-10T11:40:44Z

+ while (true) {
+ var rc: usize = undefined;
+ var err: usize = undefined;
+ if (builtin.os == .linux) {


This should just be an unconditional system.sendfile. If someone attempts to call this function on a system without it then they will get a compile error indicating as much.

terinjokes · 2020-01-10T16:08:39Z

@daurnimator I had added it to c.zig locally, but building anything linking against C fails with an unrelated error, so I can't test if my changes are good.

terinjokes · 2020-01-10T18:14:08Z

/usr/lib/zig/std/os.zig:3301:60: error: cast discards const qualifier
switch (errno(system.setsockopt(fd, level, optname, opt.ptr, @intCast(socklen_t, opt.len)))) {

terinjokes · 2020-01-10T19:49:46Z

Per discussion on IRC, this PR can likely be extended with macOS and FreeBSD support:

Extend the os.zig wrapper to accept hdr, sbytes, and flags parameters.
Handle writing headers and trailers on systems where this isn't the default. Document what systems they're written atomically.
The BSD and Linux handle updates to the offset pointer different, so I may need to also document the behavior and what would be portable.

Windows has TransmitFile as part of mswsock, which looks similar. Will have to investigate farther.

andrewrk · 2020-01-10T21:28:19Z

+ /// There was insufficient memory for reading from infd.
+ NoMem,


I'm surprised (and disappointed) this error is even possible; the kernel should be able to make this "just work" even if it is forced to use a small stack-based buffer due to heap memory pressure.
By convention, ENOMEM in zig turns into one of these two things:
error.OutOfMemory if it is userspace heap memory that is exhausted
error.SystemResources if it is kernel space memory or other resources that are unavailable.
In this case I believe it would be error.SystemResources.

Reading through the kernel sources, this actually comes from files that have a mandatory lock, and the underlying splice() cannot return ENOMEM.

@shawnl Thanks, I can update the comment. I'll likely take @andrewrk's recommendation of using error.SystemResources in this case.

terinjokes · 2020-01-20T02:32:43Z

I've pushed updates to the Linux support to accept the wider *BSD parameters, and to send the header and trailer iovecs. I doubt I have implemented the behavior to match correctly.

Next up to to add support sendfile(2) from FreeBSD libc. Then I'll be in a better place to verify the behavior is the same.

This changset adds a `sendfile(2)` syscall bindings to the linux bits component. Where available, the `sendfile64(2)` syscall will be transparently called. A wrapping function has also been added to the std.os to transform errno returns to Zig errors. Change-Id: I86769fc4382c0771e3656e7b21137bafd99a4411

terinjokes · 2020-01-21T03:40:18Z

+ trailers: [*]iovec_const,
+ trl_cnt: c_int,
+};
+pub extern "c" fn sendfile(fd: c_int, s: c_int, offset: u64, nbytes: usize, sf_hdtr: ?*sf_hdtr, sbytes: ?*u64, flags: c_int) c_int;


I need to double check that the offset parameters here should be u64 instead of off_t.

I don't think zig should support 32-bit off_t, in which case they are the same, but yeah probably still use off_t even if it must be u64.

andrewrk · 2020-03-03T07:28:33Z

Thanks! This is merged into #4611.

terinjokes · 2020-03-03T07:30:19Z

Sorry for not getting this as polished as I liked. Thank you for taking the time to carry it through.

terinjokes · 2020-01-10T10:26:29Z

This changset adds a sendfile(2) syscall bindings to the linux bits
component. Where available, the sendfile64(2) syscall will be
transparently called.

A wrapping function has also been added to the std.os to transform
errno returns to Zig errors.

daurnimator

Should also add a prototype in std/c.zig so that libc's function is used when linking against libc.

daurnimator · 2020-01-10T11:40:44Z

+ while (true) {
+ var rc: usize = undefined;
+ var err: usize = undefined;
+ if (builtin.os == .linux) {


This should just be an unconditional system.sendfile. If someone attempts to call this function on a system without it then they will get a compile error indicating as much.

terinjokes · 2020-01-10T16:08:39Z

@daurnimator I had added it to c.zig locally, but building anything linking against C fails with an unrelated error, so I can't test if my changes are good.

terinjokes · 2020-01-10T18:14:08Z

/usr/lib/zig/std/os.zig:3301:60: error: cast discards const qualifier
switch (errno(system.setsockopt(fd, level, optname, opt.ptr, @intCast(socklen_t, opt.len)))) {

terinjokes · 2020-01-10T19:49:46Z

Per discussion on IRC, this PR can likely be extended with macOS and FreeBSD support:

Extend the os.zig wrapper to accept hdr, sbytes, and flags parameters.
Handle writing headers and trailers on systems where this isn't the default. Document what systems they're written atomically.
The BSD and Linux handle updates to the offset pointer different, so I may need to also document the behavior and what would be portable.

Windows has TransmitFile as part of mswsock, which looks similar. Will have to investigate farther.

andrewrk · 2020-01-10T21:28:19Z

+ /// There was insufficient memory for reading from infd.
+ NoMem,


I'm surprised (and disappointed) this error is even possible; the kernel should be able to make this "just work" even if it is forced to use a small stack-based buffer due to heap memory pressure.
By convention, ENOMEM in zig turns into one of these two things:
error.OutOfMemory if it is userspace heap memory that is exhausted
error.SystemResources if it is kernel space memory or other resources that are unavailable.
In this case I believe it would be error.SystemResources.

Reading through the kernel sources, this actually comes from files that have a mandatory lock, and the underlying splice() cannot return ENOMEM.

@shawnl Thanks, I can update the comment. I'll likely take @andrewrk's recommendation of using error.SystemResources in this case.

terinjokes · 2020-01-20T02:32:43Z

I've pushed updates to the Linux support to accept the wider *BSD parameters, and to send the header and trailer iovecs. I doubt I have implemented the behavior to match correctly.

Next up to to add support sendfile(2) from FreeBSD libc. Then I'll be in a better place to verify the behavior is the same.

This changset adds a `sendfile(2)` syscall bindings to the linux bits component. Where available, the `sendfile64(2)` syscall will be transparently called. A wrapping function has also been added to the std.os to transform errno returns to Zig errors. Change-Id: I86769fc4382c0771e3656e7b21137bafd99a4411

terinjokes · 2020-01-21T03:40:18Z

+ trailers: [*]iovec_const,
+ trl_cnt: c_int,
+};
+pub extern "c" fn sendfile(fd: c_int, s: c_int, offset: u64, nbytes: usize, sf_hdtr: ?*sf_hdtr, sbytes: ?*u64, flags: c_int) c_int;


I need to double check that the offset parameters here should be u64 instead of off_t.

I don't think zig should support 32-bit off_t, in which case they are the same, but yeah probably still use off_t even if it must be u64.

andrewrk · 2020-03-03T07:28:33Z

Thanks! This is merged into #4611.

terinjokes · 2020-03-03T07:30:19Z

Sorry for not getting this as polished as I liked. Thank you for taking the time to carry it through.

terinjokes · 2020-01-10T10:26:29Z

This changset adds a sendfile(2) syscall bindings to the linux bits
component. Where available, the sendfile64(2) syscall will be
transparently called.

A wrapping function has also been added to the std.os to transform
errno returns to Zig errors.

daurnimator

Should also add a prototype in std/c.zig so that libc's function is used when linking against libc.

daurnimator · 2020-01-10T11:40:44Z

+ while (true) {
+ var rc: usize = undefined;
+ var err: usize = undefined;
+ if (builtin.os == .linux) {


This should just be an unconditional system.sendfile. If someone attempts to call this function on a system without it then they will get a compile error indicating as much.

terinjokes · 2020-01-10T16:08:39Z

@daurnimator I had added it to c.zig locally, but building anything linking against C fails with an unrelated error, so I can't test if my changes are good.

terinjokes · 2020-01-10T18:14:08Z

/usr/lib/zig/std/os.zig:3301:60: error: cast discards const qualifier
switch (errno(system.setsockopt(fd, level, optname, opt.ptr, @intCast(socklen_t, opt.len)))) {

terinjokes · 2020-01-10T19:49:46Z

Per discussion on IRC, this PR can likely be extended with macOS and FreeBSD support:

Extend the os.zig wrapper to accept hdr, sbytes, and flags parameters.
Handle writing headers and trailers on systems where this isn't the default. Document what systems they're written atomically.
The BSD and Linux handle updates to the offset pointer different, so I may need to also document the behavior and what would be portable.

Windows has TransmitFile as part of mswsock, which looks similar. Will have to investigate farther.

andrewrk · 2020-01-10T21:28:19Z

+ /// There was insufficient memory for reading from infd.
+ NoMem,


I'm surprised (and disappointed) this error is even possible; the kernel should be able to make this "just work" even if it is forced to use a small stack-based buffer due to heap memory pressure.
By convention, ENOMEM in zig turns into one of these two things:
error.OutOfMemory if it is userspace heap memory that is exhausted
error.SystemResources if it is kernel space memory or other resources that are unavailable.
In this case I believe it would be error.SystemResources.

Reading through the kernel sources, this actually comes from files that have a mandatory lock, and the underlying splice() cannot return ENOMEM.

@shawnl Thanks, I can update the comment. I'll likely take @andrewrk's recommendation of using error.SystemResources in this case.

terinjokes · 2020-01-20T02:32:43Z

I've pushed updates to the Linux support to accept the wider *BSD parameters, and to send the header and trailer iovecs. I doubt I have implemented the behavior to match correctly.

Next up to to add support sendfile(2) from FreeBSD libc. Then I'll be in a better place to verify the behavior is the same.

This changset adds a `sendfile(2)` syscall bindings to the linux bits component. Where available, the `sendfile64(2)` syscall will be transparently called. A wrapping function has also been added to the std.os to transform errno returns to Zig errors. Change-Id: I86769fc4382c0771e3656e7b21137bafd99a4411

terinjokes · 2020-01-21T03:40:18Z

+ trailers: [*]iovec_const,
+ trl_cnt: c_int,
+};
+pub extern "c" fn sendfile(fd: c_int, s: c_int, offset: u64, nbytes: usize, sf_hdtr: ?*sf_hdtr, sbytes: ?*u64, flags: c_int) c_int;


I need to double check that the offset parameters here should be u64 instead of off_t.

I don't think zig should support 32-bit off_t, in which case they are the same, but yeah probably still use off_t even if it must be u64.

andrewrk · 2020-03-03T07:28:33Z

Thanks! This is merged into #4611.

terinjokes · 2020-03-03T07:30:19Z

Sorry for not getting this as polished as I liked. Thank you for taking the time to carry it through.

terinjokes · 2020-01-10T10:26:29Z

This changset adds a sendfile(2) syscall bindings to the linux bits
component. Where available, the sendfile64(2) syscall will be
transparently called.

A wrapping function has also been added to the std.os to transform
errno returns to Zig errors.

daurnimator

Should also add a prototype in std/c.zig so that libc's function is used when linking against libc.

daurnimator · 2020-01-10T11:40:44Z

+ while (true) {
+ var rc: usize = undefined;
+ var err: usize = undefined;
+ if (builtin.os == .linux) {


This should just be an unconditional system.sendfile. If someone attempts to call this function on a system without it then they will get a compile error indicating as much.

terinjokes · 2020-01-10T16:08:39Z

@daurnimator I had added it to c.zig locally, but building anything linking against C fails with an unrelated error, so I can't test if my changes are good.

terinjokes · 2020-01-10T18:14:08Z

/usr/lib/zig/std/os.zig:3301:60: error: cast discards const qualifier
switch (errno(system.setsockopt(fd, level, optname, opt.ptr, @intCast(socklen_t, opt.len)))) {

terinjokes · 2020-01-10T19:49:46Z

Per discussion on IRC, this PR can likely be extended with macOS and FreeBSD support:

Extend the os.zig wrapper to accept hdr, sbytes, and flags parameters.
Handle writing headers and trailers on systems where this isn't the default. Document what systems they're written atomically.
The BSD and Linux handle updates to the offset pointer different, so I may need to also document the behavior and what would be portable.

Windows has TransmitFile as part of mswsock, which looks similar. Will have to investigate farther.

andrewrk · 2020-01-10T21:28:19Z

+ /// There was insufficient memory for reading from infd.
+ NoMem,


I'm surprised (and disappointed) this error is even possible; the kernel should be able to make this "just work" even if it is forced to use a small stack-based buffer due to heap memory pressure.
By convention, ENOMEM in zig turns into one of these two things:
error.OutOfMemory if it is userspace heap memory that is exhausted
error.SystemResources if it is kernel space memory or other resources that are unavailable.
In this case I believe it would be error.SystemResources.

Reading through the kernel sources, this actually comes from files that have a mandatory lock, and the underlying splice() cannot return ENOMEM.

@shawnl Thanks, I can update the comment. I'll likely take @andrewrk's recommendation of using error.SystemResources in this case.

terinjokes · 2020-01-20T02:32:43Z

I've pushed updates to the Linux support to accept the wider *BSD parameters, and to send the header and trailer iovecs. I doubt I have implemented the behavior to match correctly.

Next up to to add support sendfile(2) from FreeBSD libc. Then I'll be in a better place to verify the behavior is the same.

This changset adds a `sendfile(2)` syscall bindings to the linux bits component. Where available, the `sendfile64(2)` syscall will be transparently called. A wrapping function has also been added to the std.os to transform errno returns to Zig errors. Change-Id: I86769fc4382c0771e3656e7b21137bafd99a4411

terinjokes · 2020-01-21T03:40:18Z

+ trailers: [*]iovec_const,
+ trl_cnt: c_int,
+};
+pub extern "c" fn sendfile(fd: c_int, s: c_int, offset: u64, nbytes: usize, sf_hdtr: ?*sf_hdtr, sbytes: ?*u64, flags: c_int) c_int;


I need to double check that the offset parameters here should be u64 instead of off_t.

I don't think zig should support 32-bit off_t, in which case they are the same, but yeah probably still use off_t even if it must be u64.

andrewrk · 2020-03-03T07:28:33Z

Thanks! This is merged into #4611.

terinjokes · 2020-03-03T07:30:19Z

Sorry for not getting this as polished as I liked. Thank you for taking the time to carry it through.

terinjokes · 2020-01-10T10:26:29Z

This changset adds a sendfile(2) syscall bindings to the linux bits
component. Where available, the sendfile64(2) syscall will be
transparently called.

A wrapping function has also been added to the std.os to transform
errno returns to Zig errors.

daurnimator

Should also add a prototype in std/c.zig so that libc's function is used when linking against libc.

daurnimator · 2020-01-10T11:40:44Z

+ while (true) {
+ var rc: usize = undefined;
+ var err: usize = undefined;
+ if (builtin.os == .linux) {


This should just be an unconditional system.sendfile. If someone attempts to call this function on a system without it then they will get a compile error indicating as much.

terinjokes · 2020-01-10T16:08:39Z

@daurnimator I had added it to c.zig locally, but building anything linking against C fails with an unrelated error, so I can't test if my changes are good.

terinjokes · 2020-01-10T18:14:08Z

/usr/lib/zig/std/os.zig:3301:60: error: cast discards const qualifier
switch (errno(system.setsockopt(fd, level, optname, opt.ptr, @intCast(socklen_t, opt.len)))) {

terinjokes · 2020-01-10T19:49:46Z

Per discussion on IRC, this PR can likely be extended with macOS and FreeBSD support:

Extend the os.zig wrapper to accept hdr, sbytes, and flags parameters.
Handle writing headers and trailers on systems where this isn't the default. Document what systems they're written atomically.
The BSD and Linux handle updates to the offset pointer different, so I may need to also document the behavior and what would be portable.

Windows has TransmitFile as part of mswsock, which looks similar. Will have to investigate farther.

andrewrk · 2020-01-10T21:28:19Z

+ /// There was insufficient memory for reading from infd.
+ NoMem,


I'm surprised (and disappointed) this error is even possible; the kernel should be able to make this "just work" even if it is forced to use a small stack-based buffer due to heap memory pressure.
By convention, ENOMEM in zig turns into one of these two things:
error.OutOfMemory if it is userspace heap memory that is exhausted
error.SystemResources if it is kernel space memory or other resources that are unavailable.
In this case I believe it would be error.SystemResources.

Reading through the kernel sources, this actually comes from files that have a mandatory lock, and the underlying splice() cannot return ENOMEM.

@shawnl Thanks, I can update the comment. I'll likely take @andrewrk's recommendation of using error.SystemResources in this case.

terinjokes · 2020-01-20T02:32:43Z

I've pushed updates to the Linux support to accept the wider *BSD parameters, and to send the header and trailer iovecs. I doubt I have implemented the behavior to match correctly.

Next up to to add support sendfile(2) from FreeBSD libc. Then I'll be in a better place to verify the behavior is the same.

This changset adds a `sendfile(2)` syscall bindings to the linux bits component. Where available, the `sendfile64(2)` syscall will be transparently called. A wrapping function has also been added to the std.os to transform errno returns to Zig errors. Change-Id: I86769fc4382c0771e3656e7b21137bafd99a4411

terinjokes · 2020-01-21T03:40:18Z

+ trailers: [*]iovec_const,
+ trl_cnt: c_int,
+};
+pub extern "c" fn sendfile(fd: c_int, s: c_int, offset: u64, nbytes: usize, sf_hdtr: ?*sf_hdtr, sbytes: ?*u64, flags: c_int) c_int;


I need to double check that the offset parameters here should be u64 instead of off_t.

I don't think zig should support 32-bit off_t, in which case they are the same, but yeah probably still use off_t even if it must be u64.

andrewrk · 2020-03-03T07:28:33Z

Thanks! This is merged into #4611.

terinjokes · 2020-03-03T07:30:19Z

Sorry for not getting this as polished as I liked. Thank you for taking the time to carry it through.

daurnimator suggested changes Jan 10, 2020
View reviewed changes

daurnimator added os-linux Linux standard library This issue involves writing Zig code for the standard library. labels Jan 10, 2020

andrewrk reviewed Jan 10, 2020
View reviewed changes

terinjokes force-pushed the patches/sendfile branch from 95ca1df to 68186bbCompare January 20, 2020 02:24

terinjokes force-pushed the patches/sendfile branch from 68186bb to fe10fdeCompare January 21, 2020 03:19

terinjokes commented Jan 21, 2020
View reviewed changes

andrewrk mentioned this pull request Feb 25, 2020
introduce operating system version ranges as part of the target; self-host native dynamic linker detection and native glibc version detection #4550
Merged
14 tasks

andrewrk mentioned this pull request Mar 3, 2020
std.os read/write functions + sendfile #4611
Closed

andrewrk closed this Mar 3, 2020

andrewrk mentioned this pull request Mar 3, 2020
std.os read/write functions + sendfile #4612
Merged

daurnimator suggested changes Jan 10, 2020
View reviewed changes

daurnimator added os-linux Linux standard library This issue involves writing Zig code for the standard library. labels Jan 10, 2020

andrewrk reviewed Jan 10, 2020
View reviewed changes

terinjokes force-pushed the patches/sendfile branch from 95ca1df to 68186bbCompare January 20, 2020 02:24

terinjokes force-pushed the patches/sendfile branch from 68186bb to fe10fdeCompare January 21, 2020 03:19

terinjokes commented Jan 21, 2020
View reviewed changes

andrewrk mentioned this pull request Feb 25, 2020
introduce operating system version ranges as part of the target; self-host native dynamic linker detection and native glibc version detection #4550
Merged
14 tasks

andrewrk mentioned this pull request Mar 3, 2020
std.os read/write functions + sendfile #4611
Closed

andrewrk closed this Mar 3, 2020

andrewrk mentioned this pull request Mar 3, 2020
std.os read/write functions + sendfile #4612
Merged

daurnimator suggested changes Jan 10, 2020
View reviewed changes

daurnimator added os-linux Linux standard library This issue involves writing Zig code for the standard library. labels Jan 10, 2020

andrewrk reviewed Jan 10, 2020
View reviewed changes

terinjokes force-pushed the patches/sendfile branch from 95ca1df to 68186bbCompare January 20, 2020 02:24

terinjokes force-pushed the patches/sendfile branch from 68186bb to fe10fdeCompare January 21, 2020 03:19

terinjokes commented Jan 21, 2020
View reviewed changes

andrewrk mentioned this pull request Feb 25, 2020
introduce operating system version ranges as part of the target; self-host native dynamic linker detection and native glibc version detection #4550
Merged
14 tasks

andrewrk mentioned this pull request Mar 3, 2020
std.os read/write functions + sendfile #4611
Closed

andrewrk closed this Mar 3, 2020

andrewrk mentioned this pull request Mar 3, 2020
std.os read/write functions + sendfile #4612
Merged

daurnimator suggested changes Jan 10, 2020
View reviewed changes

daurnimator added os-linux Linux standard library This issue involves writing Zig code for the standard library. labels Jan 10, 2020

andrewrk reviewed Jan 10, 2020
View reviewed changes

terinjokes force-pushed the patches/sendfile branch from 95ca1df to 68186bbCompare January 20, 2020 02:24

terinjokes force-pushed the patches/sendfile branch from 68186bb to fe10fdeCompare January 21, 2020 03:19

terinjokes commented Jan 21, 2020
View reviewed changes

andrewrk mentioned this pull request Feb 25, 2020
introduce operating system version ranges as part of the target; self-host native dynamic linker detection and native glibc version detection #4550
Merged
14 tasks

andrewrk mentioned this pull request Mar 3, 2020
std.os read/write functions + sendfile #4611
Closed

andrewrk closed this Mar 3, 2020

andrewrk mentioned this pull request Mar 3, 2020
std.os read/write functions + sendfile #4612
Merged

daurnimator suggested changes Jan 10, 2020
View reviewed changes

daurnimator added os-linux Linux standard library This issue involves writing Zig code for the standard library. labels Jan 10, 2020

andrewrk reviewed Jan 10, 2020
View reviewed changes

terinjokes force-pushed the patches/sendfile branch from 95ca1df to 68186bbCompare January 20, 2020 02:24

terinjokes force-pushed the patches/sendfile branch from 68186bb to fe10fdeCompare January 21, 2020 03:19

terinjokes commented Jan 21, 2020
View reviewed changes

andrewrk mentioned this pull request Feb 25, 2020
introduce operating system version ranges as part of the target; self-host native dynamic linker detection and native glibc version detection #4550
Merged
14 tasks

andrewrk mentioned this pull request Mar 3, 2020
std.os read/write functions + sendfile #4611
Closed

andrewrk closed this Mar 3, 2020

andrewrk mentioned this pull request Mar 3, 2020
std.os read/write functions + sendfile #4612
Merged

daurnimator suggested changes Jan 10, 2020
View reviewed changes

daurnimator added os-linux Linux standard library This issue involves writing Zig code for the standard library. labels Jan 10, 2020

andrewrk reviewed Jan 10, 2020
View reviewed changes

terinjokes force-pushed the patches/sendfile branch from 95ca1df to 68186bbCompare January 20, 2020 02:24

terinjokes force-pushed the patches/sendfile branch from 68186bb to fe10fdeCompare January 21, 2020 03:19

terinjokes commented Jan 21, 2020
View reviewed changes

andrewrk mentioned this pull request Feb 25, 2020
introduce operating system version ranges as part of the target; self-host native dynamic linker detection and native glibc version detection #4550
Merged
14 tasks

andrewrk mentioned this pull request Mar 3, 2020
std.os read/write functions + sendfile #4611
Closed

andrewrk closed this Mar 3, 2020

andrewrk mentioned this pull request Mar 3, 2020
std.os read/write functions + sendfile #4612
Merged

daurnimator suggested changes Jan 10, 2020
View reviewed changes

daurnimator added os-linux Linux standard library This issue involves writing Zig code for the standard library. labels Jan 10, 2020

andrewrk reviewed Jan 10, 2020
View reviewed changes

terinjokes force-pushed the patches/sendfile branch from 95ca1df to 68186bbCompare January 20, 2020 02:24

terinjokes force-pushed the patches/sendfile branch from 68186bb to fe10fdeCompare January 21, 2020 03:19

terinjokes commented Jan 21, 2020
View reviewed changes

andrewrk mentioned this pull request Feb 25, 2020
introduce operating system version ranges as part of the target; self-host native dynamic linker detection and native glibc version detection #4550
Merged
14 tasks

andrewrk mentioned this pull request Mar 3, 2020
std.os read/write functions + sendfile #4611
Closed

andrewrk closed this Mar 3, 2020

andrewrk mentioned this pull request Mar 3, 2020
std.os read/write functions + sendfile #4612
Merged

daurnimator suggested changes Jan 10, 2020
View reviewed changes

daurnimator added os-linux Linux standard library This issue involves writing Zig code for the standard library. labels Jan 10, 2020

andrewrk reviewed Jan 10, 2020
View reviewed changes

terinjokes force-pushed the patches/sendfile branch from 95ca1df to 68186bbCompare January 20, 2020 02:24

terinjokes force-pushed the patches/sendfile branch from 68186bb to fe10fdeCompare January 21, 2020 03:19

terinjokes commented Jan 21, 2020
View reviewed changes

andrewrk mentioned this pull request Feb 25, 2020
introduce operating system version ranges as part of the target; self-host native dynamic linker detection and native glibc version detection #4550
Merged
14 tasks

andrewrk mentioned this pull request Mar 3, 2020
std.os read/write functions + sendfile #4611
Closed

andrewrk closed this Mar 3, 2020

andrewrk mentioned this pull request Mar 3, 2020
std.os read/write functions + sendfile #4612
Merged

		/// There was insufficient memory for reading from infd.
		NoMem,

Uh oh!

Uh oh!

Conversation

terinjokes commented Jan 10, 2020

Uh oh!

daurnimator left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

terinjokes commented Jan 10, 2020

Uh oh!

terinjokes commented Jan 10, 2020

Uh oh!

terinjokes commented Jan 10, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

terinjokes commented Jan 20, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewrk commented Mar 3, 2020

Uh oh!

terinjokes commented Mar 3, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Uh oh!

Conversation

terinjokes commented Jan 10, 2020

Uh oh!

daurnimator left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

terinjokes commented Jan 10, 2020

Uh oh!

terinjokes commented Jan 10, 2020

Uh oh!

terinjokes commented Jan 10, 2020• edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

terinjokes commented Jan 20, 2020

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

andrewrk commented Mar 3, 2020

Uh oh!

terinjokes commented Mar 3, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

terinjokes commented Jan 10, 2020•
edited
Loading

terinjokes commented Jan 10, 2020•
edited
Loading

terinjokes commented Jan 10, 2020•
edited
Loading