Skip to content

Add support for fs.ReadStream/fs.WriteStream #60

Description

@nobugs-hare

Portable implementation:

  • libuv-style, with thread pool for handling blocking file I/O requests

Efficient Implementation:

  • Unlike libuv, we want to do all file I/O REALLY asynchronously; to do this, the following ("opportunistic") model is supposed (for those platforms which DO support async file IO, BUT do NOT allow intermixing them with poll()):
    • all file read/write calls are initiated asynchronous
      • at any point, we know if there is outstanding async file I/O for current thread
    • before any call to poll(), if there is outstanding async file I/O, our poll()-ing thread checks (with 0 timeout) if there is any finished async I/O , and if yes - processes it, and repeats this item
    • if there is NO outstanding async file I/O - we're waiting in poll() according to current timers (as if there is no file I/O at all)
    • if there IS outstanding async file I/O - we're merely checking poll() (with 0 timeout), and are processing whatever we can process; then go to "before any call to poll()" item
    • if poll() returned that there is no socket I/O, BUT there is still outstanding async file I/O - we're waiting for file I/O, and are processing it, then going back to "before any call to poll()" item
      • this is based on an assumption that file I/O is fast
        • we also MAY want to limit wait on async file I/O to approx 10-20ms just in case (it should happen REALLY rarely) - and to go on another round of poll() if timeout is exceeded
        • alternatively, we may also want to split "big" writes (larger than 1MB) into smaller chunks (so that each chunk is fast) - with 2-3 chunks outstanding at each moment to run big writes smoothly
        • on the client-side, we MIGHT also want to use separate threads for read/writes from/to "slow" (USB) drives.

Suggested implementations per platform:

  • Linux: "opportunistic" as described above, with poll() or epoll() + AIO
  • Windows: "opportunistic" as described above, with WSAPoll() + WaitForMultipleObjects for files (if number of outstanding requests exceeds 64 - it should be ok to wait for first 64).
  • FreeBSD: it seems that kqueue() will allow direct non-blocking implementation

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions