Avoid re-using non-keep-alive "main" HTTP client

[developit]

I noticed a comment on actions/upload-artifact#62 citing 15% slower uploads and thought I'd poke around. It looks like the chunked uploads are using HTTP clients starting from index 0, but the client at index 0 is also used for things other than uploading.

I'll be honest, it seems like allocating a separate HTTP client outside the pool for the non-upload work would make more sense here, or just using the default Agent.

[konradpabjan]

Thanks for the PR @developit

The basic flow for uploading an artifact is:

• 1 HTTP POST call is made to create a container for the artifact (uses non keep-alive index 0 http-client)
• Each individual file gets uploaded using a HTTP PUT call (keep-alive is used here and the number of http clients corresponds to the upload concurrency)
• 1 HTTP PATCH call is made to update the size of the artifact after everything is complete (uses index 0 http-client)

Only 2 HTTP calls use this index 0 http-client. If we're uploading 1000 small files as part of the artifact, then there will be 1002 HTTP calls made so this will have a pretty negligible impact (if any) on the total performance. If a file is greater than 4MB, it is uploaded in chunks so a 15MB file would require 4 HTTP put calls. The majority of the time spent is making these PUT calls.

At one point I even had these 2 separate HTTP calls use a separate client but that didn't have any significant impact on the performance. To not have the upload-http-client instantiating its own httpclients i made the two arbitrary calls use the first client from the pool just so that all clients get managed from a central place.

Thanks for the submitting the PR, but I don't think this is. 😥

[developit]

I noticed a comment on actions/upload-artifact#62 citing 15% slower uploads and thought I'd poke around. It looks like the chunked uploads are using HTTP clients starting from index 0, but the client at index 0 is also used for things other than uploading.

I'll be honest, it seems like allocating a separate HTTP client outside the pool for the non-upload work would make more sense here, or just using the default Agent.

[konradpabjan]

Thanks for the PR @developit

The basic flow for uploading an artifact is:

• 1 HTTP POST call is made to create a container for the artifact (uses non keep-alive index 0 http-client)
• Each individual file gets uploaded using a HTTP PUT call (keep-alive is used here and the number of http clients corresponds to the upload concurrency)
• 1 HTTP PATCH call is made to update the size of the artifact after everything is complete (uses index 0 http-client)

Only 2 HTTP calls use this index 0 http-client. If we're uploading 1000 small files as part of the artifact, then there will be 1002 HTTP calls made so this will have a pretty negligible impact (if any) on the total performance. If a file is greater than 4MB, it is uploaded in chunks so a 15MB file would require 4 HTTP put calls. The majority of the time spent is making these PUT calls.

At one point I even had these 2 separate HTTP calls use a separate client but that didn't have any significant impact on the performance. To not have the upload-http-client instantiating its own httpclients i made the two arbitrary calls use the first client from the pool just so that all clients get managed from a central place.

Thanks for the submitting the PR, but I don't think this is. 😥

[developit]

I noticed a comment on actions/upload-artifact#62 citing 15% slower uploads and thought I'd poke around. It looks like the chunked uploads are using HTTP clients starting from index 0, but the client at index 0 is also used for things other than uploading.

I'll be honest, it seems like allocating a separate HTTP client outside the pool for the non-upload work would make more sense here, or just using the default Agent.

[konradpabjan]

Thanks for the PR @developit

The basic flow for uploading an artifact is:

• 1 HTTP POST call is made to create a container for the artifact (uses non keep-alive index 0 http-client)
• Each individual file gets uploaded using a HTTP PUT call (keep-alive is used here and the number of http clients corresponds to the upload concurrency)
• 1 HTTP PATCH call is made to update the size of the artifact after everything is complete (uses index 0 http-client)

Only 2 HTTP calls use this index 0 http-client. If we're uploading 1000 small files as part of the artifact, then there will be 1002 HTTP calls made so this will have a pretty negligible impact (if any) on the total performance. If a file is greater than 4MB, it is uploaded in chunks so a 15MB file would require 4 HTTP put calls. The majority of the time spent is making these PUT calls.

At one point I even had these 2 separate HTTP calls use a separate client but that didn't have any significant impact on the performance. To not have the upload-http-client instantiating its own httpclients i made the two arbitrary calls use the first client from the pool just so that all clients get managed from a central place.

Thanks for the submitting the PR, but I don't think this is. 😥

[developit]

I noticed a comment on actions/upload-artifact#62 citing 15% slower uploads and thought I'd poke around. It looks like the chunked uploads are using HTTP clients starting from index 0, but the client at index 0 is also used for things other than uploading.

I'll be honest, it seems like allocating a separate HTTP client outside the pool for the non-upload work would make more sense here, or just using the default Agent.

[konradpabjan]

Thanks for the PR @developit

The basic flow for uploading an artifact is:

• 1 HTTP POST call is made to create a container for the artifact (uses non keep-alive index 0 http-client)
• Each individual file gets uploaded using a HTTP PUT call (keep-alive is used here and the number of http clients corresponds to the upload concurrency)
• 1 HTTP PATCH call is made to update the size of the artifact after everything is complete (uses index 0 http-client)

Only 2 HTTP calls use this index 0 http-client. If we're uploading 1000 small files as part of the artifact, then there will be 1002 HTTP calls made so this will have a pretty negligible impact (if any) on the total performance. If a file is greater than 4MB, it is uploaded in chunks so a 15MB file would require 4 HTTP put calls. The majority of the time spent is making these PUT calls.

At one point I even had these 2 separate HTTP calls use a separate client but that didn't have any significant impact on the performance. To not have the upload-http-client instantiating its own httpclients i made the two arbitrary calls use the first client from the pool just so that all clients get managed from a central place.

Thanks for the submitting the PR, but I don't think this is. 😥

[developit]

I noticed a comment on actions/upload-artifact#62 citing 15% slower uploads and thought I'd poke around. It looks like the chunked uploads are using HTTP clients starting from index 0, but the client at index 0 is also used for things other than uploading.

I'll be honest, it seems like allocating a separate HTTP client outside the pool for the non-upload work would make more sense here, or just using the default Agent.

[konradpabjan]

Thanks for the PR @developit

The basic flow for uploading an artifact is:

• 1 HTTP POST call is made to create a container for the artifact (uses non keep-alive index 0 http-client)
• Each individual file gets uploaded using a HTTP PUT call (keep-alive is used here and the number of http clients corresponds to the upload concurrency)
• 1 HTTP PATCH call is made to update the size of the artifact after everything is complete (uses index 0 http-client)

Only 2 HTTP calls use this index 0 http-client. If we're uploading 1000 small files as part of the artifact, then there will be 1002 HTTP calls made so this will have a pretty negligible impact (if any) on the total performance. If a file is greater than 4MB, it is uploaded in chunks so a 15MB file would require 4 HTTP put calls. The majority of the time spent is making these PUT calls.

At one point I even had these 2 separate HTTP calls use a separate client but that didn't have any significant impact on the performance. To not have the upload-http-client instantiating its own httpclients i made the two arbitrary calls use the first client from the pool just so that all clients get managed from a central place.

Thanks for the submitting the PR, but I don't think this is. 😥

[developit]

I noticed a comment on actions/upload-artifact#62 citing 15% slower uploads and thought I'd poke around. It looks like the chunked uploads are using HTTP clients starting from index 0, but the client at index 0 is also used for things other than uploading.

I'll be honest, it seems like allocating a separate HTTP client outside the pool for the non-upload work would make more sense here, or just using the default Agent.

[konradpabjan]

Thanks for the PR @developit

The basic flow for uploading an artifact is:

• 1 HTTP POST call is made to create a container for the artifact (uses non keep-alive index 0 http-client)
• Each individual file gets uploaded using a HTTP PUT call (keep-alive is used here and the number of http clients corresponds to the upload concurrency)
• 1 HTTP PATCH call is made to update the size of the artifact after everything is complete (uses index 0 http-client)

Only 2 HTTP calls use this index 0 http-client. If we're uploading 1000 small files as part of the artifact, then there will be 1002 HTTP calls made so this will have a pretty negligible impact (if any) on the total performance. If a file is greater than 4MB, it is uploaded in chunks so a 15MB file would require 4 HTTP put calls. The majority of the time spent is making these PUT calls.

At one point I even had these 2 separate HTTP calls use a separate client but that didn't have any significant impact on the performance. To not have the upload-http-client instantiating its own httpclients i made the two arbitrary calls use the first client from the pool just so that all clients get managed from a central place.

Thanks for the submitting the PR, but I don't think this is. 😥

[developit]

I noticed a comment on actions/upload-artifact#62 citing 15% slower uploads and thought I'd poke around. It looks like the chunked uploads are using HTTP clients starting from index 0, but the client at index 0 is also used for things other than uploading.

I'll be honest, it seems like allocating a separate HTTP client outside the pool for the non-upload work would make more sense here, or just using the default Agent.

[konradpabjan]

Thanks for the PR @developit

The basic flow for uploading an artifact is:

• 1 HTTP POST call is made to create a container for the artifact (uses non keep-alive index 0 http-client)
• Each individual file gets uploaded using a HTTP PUT call (keep-alive is used here and the number of http clients corresponds to the upload concurrency)
• 1 HTTP PATCH call is made to update the size of the artifact after everything is complete (uses index 0 http-client)

Only 2 HTTP calls use this index 0 http-client. If we're uploading 1000 small files as part of the artifact, then there will be 1002 HTTP calls made so this will have a pretty negligible impact (if any) on the total performance. If a file is greater than 4MB, it is uploaded in chunks so a 15MB file would require 4 HTTP put calls. The majority of the time spent is making these PUT calls.

At one point I even had these 2 separate HTTP calls use a separate client but that didn't have any significant impact on the performance. To not have the upload-http-client instantiating its own httpclients i made the two arbitrary calls use the first client from the pool just so that all clients get managed from a central place.

Thanks for the submitting the PR, but I don't think this is. 😥

[developit]

I noticed a comment on actions/upload-artifact#62 citing 15% slower uploads and thought I'd poke around. It looks like the chunked uploads are using HTTP clients starting from index 0, but the client at index 0 is also used for things other than uploading.

I'll be honest, it seems like allocating a separate HTTP client outside the pool for the non-upload work would make more sense here, or just using the default Agent.

[konradpabjan]

Thanks for the PR @developit

The basic flow for uploading an artifact is:

• 1 HTTP POST call is made to create a container for the artifact (uses non keep-alive index 0 http-client)
• Each individual file gets uploaded using a HTTP PUT call (keep-alive is used here and the number of http clients corresponds to the upload concurrency)
• 1 HTTP PATCH call is made to update the size of the artifact after everything is complete (uses index 0 http-client)

Only 2 HTTP calls use this index 0 http-client. If we're uploading 1000 small files as part of the artifact, then there will be 1002 HTTP calls made so this will have a pretty negligible impact (if any) on the total performance. If a file is greater than 4MB, it is uploaded in chunks so a 15MB file would require 4 HTTP put calls. The majority of the time spent is making these PUT calls.

At one point I even had these 2 separate HTTP calls use a separate client but that didn't have any significant impact on the performance. To not have the upload-http-client instantiating its own httpclients i made the two arbitrary calls use the first client from the pool just so that all clients get managed from a central place.

Thanks for the submitting the PR, but I don't think this is. 😥