Fix encoding/decoding of base-256 numbers

[justfalter]

This PR fixes issues I've identified with the handling of base-256 encoded numbers within node-tar (see #188). The issues would generally present themselves when attempting to extract a gnu-formatted tar entry for a file greater than 8gb in size.

• The last byte of the buffer was incorrectly being ignored when encoding/decoding.
• Javascript can only have safe integer precision for numbers between -9007199254740991 and 9007199254740991. Any numbers outside these bounds will see the lowest-order bits rounded off. For example, 9007199254749999 will be rounded to 9007199254750000.
  • This is likely what accounted for the belief that the last byte of the buffer was always set to 00 or 20.
  • See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number/MAX_SAFE_INTEGER for more details.
• I've modified large-integer.jsparse and encode functions to throw TypeError exceptions if they encounter a number that will not be precisely represented as a javascript integer, or if the buffer being decoded does not appear to be a base-256 encoded number (must start with 80 or ff).

[isaacs]

I see what happened here.

To encode files over 8gb, bsdtar drops the trailing 0x20, and uses that as a part of the octal-in-ascii number. This is an ambiguous part of the spec (such as it is), which gnutar interprets differently. Thankfully, bsdtar also prepends a PAX extended file attributes entry, which gnutar interprets properly. So, for example, a 10gb file would fill those 12 bytes with '120000000000', no terminator. Gnutar misinterprets this as 0o12000000000 (a mere 1.25 Gb), but that's overridden by the PAX header.

I'm not sure what bsdtar would write in that block if the file size couldn't fit in 12 octal digits, but I'm also not eager to create a 64 gb file on my laptop to find out.

[isaacs]

Actually I did get curious and checked. Bsdtar does the same thing as gnutar, but only when the file is 64gb or greater.

In short, this patch is good, it's landed, and published to latest. Thank you for digging in and fixing this. The reason it escaped my notice for so long is frankly that pax headers are so much more straightforward and make this type of bug irrelevant in so many cases.

The checks for Number.MAX_SAFE_INTEGER made me chuckle. I thought for a second that throwing a new error should be semver major bump, but if people are using this library to pack and unpack tarballs with 8 petabyte files in them, then the world is definitely in trouble.

[justfalter]

Thanks for taking the time to go over this, @isaacs. I’ve got something upstream from my project that is explicitly producing gnu-formatted tarballs, for some reason.

Honestly, the only reason I even thought to do the min/max int checks were because you had tests that would have violated those checks. 8 petabyte files are pretty unlikely, but I wondered if there weren’t other numbers that might be encoded (uid, his, etc) that might someday exceed max int. I dug a bit into libarchive, and they have similar checks, there, as well.

[justfalter]

This PR fixes issues I've identified with the handling of base-256 encoded numbers within node-tar (see #188). The issues would generally present themselves when attempting to extract a gnu-formatted tar entry for a file greater than 8gb in size.

• The last byte of the buffer was incorrectly being ignored when encoding/decoding.
• Javascript can only have safe integer precision for numbers between -9007199254740991 and 9007199254740991. Any numbers outside these bounds will see the lowest-order bits rounded off. For example, 9007199254749999 will be rounded to 9007199254750000.
  • This is likely what accounted for the belief that the last byte of the buffer was always set to 00 or 20.
  • See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number/MAX_SAFE_INTEGER for more details.
• I've modified large-integer.jsparse and encode functions to throw TypeError exceptions if they encounter a number that will not be precisely represented as a javascript integer, or if the buffer being decoded does not appear to be a base-256 encoded number (must start with 80 or ff).

[isaacs]

I see what happened here.

To encode files over 8gb, bsdtar drops the trailing 0x20, and uses that as a part of the octal-in-ascii number. This is an ambiguous part of the spec (such as it is), which gnutar interprets differently. Thankfully, bsdtar also prepends a PAX extended file attributes entry, which gnutar interprets properly. So, for example, a 10gb file would fill those 12 bytes with '120000000000', no terminator. Gnutar misinterprets this as 0o12000000000 (a mere 1.25 Gb), but that's overridden by the PAX header.

I'm not sure what bsdtar would write in that block if the file size couldn't fit in 12 octal digits, but I'm also not eager to create a 64 gb file on my laptop to find out.

[isaacs]

Actually I did get curious and checked. Bsdtar does the same thing as gnutar, but only when the file is 64gb or greater.

In short, this patch is good, it's landed, and published to latest. Thank you for digging in and fixing this. The reason it escaped my notice for so long is frankly that pax headers are so much more straightforward and make this type of bug irrelevant in so many cases.

The checks for Number.MAX_SAFE_INTEGER made me chuckle. I thought for a second that throwing a new error should be semver major bump, but if people are using this library to pack and unpack tarballs with 8 petabyte files in them, then the world is definitely in trouble.

[justfalter]

Thanks for taking the time to go over this, @isaacs. I’ve got something upstream from my project that is explicitly producing gnu-formatted tarballs, for some reason.

Honestly, the only reason I even thought to do the min/max int checks were because you had tests that would have violated those checks. 8 petabyte files are pretty unlikely, but I wondered if there weren’t other numbers that might be encoded (uid, his, etc) that might someday exceed max int. I dug a bit into libarchive, and they have similar checks, there, as well.

[justfalter]

This PR fixes issues I've identified with the handling of base-256 encoded numbers within node-tar (see #188). The issues would generally present themselves when attempting to extract a gnu-formatted tar entry for a file greater than 8gb in size.

• The last byte of the buffer was incorrectly being ignored when encoding/decoding.
• Javascript can only have safe integer precision for numbers between -9007199254740991 and 9007199254740991. Any numbers outside these bounds will see the lowest-order bits rounded off. For example, 9007199254749999 will be rounded to 9007199254750000.
  • This is likely what accounted for the belief that the last byte of the buffer was always set to 00 or 20.
  • See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number/MAX_SAFE_INTEGER for more details.
• I've modified large-integer.jsparse and encode functions to throw TypeError exceptions if they encounter a number that will not be precisely represented as a javascript integer, or if the buffer being decoded does not appear to be a base-256 encoded number (must start with 80 or ff).

[isaacs]

I see what happened here.

To encode files over 8gb, bsdtar drops the trailing 0x20, and uses that as a part of the octal-in-ascii number. This is an ambiguous part of the spec (such as it is), which gnutar interprets differently. Thankfully, bsdtar also prepends a PAX extended file attributes entry, which gnutar interprets properly. So, for example, a 10gb file would fill those 12 bytes with '120000000000', no terminator. Gnutar misinterprets this as 0o12000000000 (a mere 1.25 Gb), but that's overridden by the PAX header.

I'm not sure what bsdtar would write in that block if the file size couldn't fit in 12 octal digits, but I'm also not eager to create a 64 gb file on my laptop to find out.

[isaacs]

Actually I did get curious and checked. Bsdtar does the same thing as gnutar, but only when the file is 64gb or greater.

In short, this patch is good, it's landed, and published to latest. Thank you for digging in and fixing this. The reason it escaped my notice for so long is frankly that pax headers are so much more straightforward and make this type of bug irrelevant in so many cases.

The checks for Number.MAX_SAFE_INTEGER made me chuckle. I thought for a second that throwing a new error should be semver major bump, but if people are using this library to pack and unpack tarballs with 8 petabyte files in them, then the world is definitely in trouble.

[justfalter]

Thanks for taking the time to go over this, @isaacs. I’ve got something upstream from my project that is explicitly producing gnu-formatted tarballs, for some reason.

Honestly, the only reason I even thought to do the min/max int checks were because you had tests that would have violated those checks. 8 petabyte files are pretty unlikely, but I wondered if there weren’t other numbers that might be encoded (uid, his, etc) that might someday exceed max int. I dug a bit into libarchive, and they have similar checks, there, as well.

[justfalter]

This PR fixes issues I've identified with the handling of base-256 encoded numbers within node-tar (see #188). The issues would generally present themselves when attempting to extract a gnu-formatted tar entry for a file greater than 8gb in size.

• The last byte of the buffer was incorrectly being ignored when encoding/decoding.
• Javascript can only have safe integer precision for numbers between -9007199254740991 and 9007199254740991. Any numbers outside these bounds will see the lowest-order bits rounded off. For example, 9007199254749999 will be rounded to 9007199254750000.
  • This is likely what accounted for the belief that the last byte of the buffer was always set to 00 or 20.
  • See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number/MAX_SAFE_INTEGER for more details.
• I've modified large-integer.jsparse and encode functions to throw TypeError exceptions if they encounter a number that will not be precisely represented as a javascript integer, or if the buffer being decoded does not appear to be a base-256 encoded number (must start with 80 or ff).

[isaacs]

I see what happened here.

To encode files over 8gb, bsdtar drops the trailing 0x20, and uses that as a part of the octal-in-ascii number. This is an ambiguous part of the spec (such as it is), which gnutar interprets differently. Thankfully, bsdtar also prepends a PAX extended file attributes entry, which gnutar interprets properly. So, for example, a 10gb file would fill those 12 bytes with '120000000000', no terminator. Gnutar misinterprets this as 0o12000000000 (a mere 1.25 Gb), but that's overridden by the PAX header.

I'm not sure what bsdtar would write in that block if the file size couldn't fit in 12 octal digits, but I'm also not eager to create a 64 gb file on my laptop to find out.

[isaacs]

Actually I did get curious and checked. Bsdtar does the same thing as gnutar, but only when the file is 64gb or greater.

In short, this patch is good, it's landed, and published to latest. Thank you for digging in and fixing this. The reason it escaped my notice for so long is frankly that pax headers are so much more straightforward and make this type of bug irrelevant in so many cases.

The checks for Number.MAX_SAFE_INTEGER made me chuckle. I thought for a second that throwing a new error should be semver major bump, but if people are using this library to pack and unpack tarballs with 8 petabyte files in them, then the world is definitely in trouble.

[justfalter]

Thanks for taking the time to go over this, @isaacs. I’ve got something upstream from my project that is explicitly producing gnu-formatted tarballs, for some reason.

Honestly, the only reason I even thought to do the min/max int checks were because you had tests that would have violated those checks. 8 petabyte files are pretty unlikely, but I wondered if there weren’t other numbers that might be encoded (uid, his, etc) that might someday exceed max int. I dug a bit into libarchive, and they have similar checks, there, as well.

[justfalter]

This PR fixes issues I've identified with the handling of base-256 encoded numbers within node-tar (see #188). The issues would generally present themselves when attempting to extract a gnu-formatted tar entry for a file greater than 8gb in size.

• The last byte of the buffer was incorrectly being ignored when encoding/decoding.
• Javascript can only have safe integer precision for numbers between -9007199254740991 and 9007199254740991. Any numbers outside these bounds will see the lowest-order bits rounded off. For example, 9007199254749999 will be rounded to 9007199254750000.
  • This is likely what accounted for the belief that the last byte of the buffer was always set to 00 or 20.
  • See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number/MAX_SAFE_INTEGER for more details.
• I've modified large-integer.jsparse and encode functions to throw TypeError exceptions if they encounter a number that will not be precisely represented as a javascript integer, or if the buffer being decoded does not appear to be a base-256 encoded number (must start with 80 or ff).

[isaacs]

I see what happened here.

To encode files over 8gb, bsdtar drops the trailing 0x20, and uses that as a part of the octal-in-ascii number. This is an ambiguous part of the spec (such as it is), which gnutar interprets differently. Thankfully, bsdtar also prepends a PAX extended file attributes entry, which gnutar interprets properly. So, for example, a 10gb file would fill those 12 bytes with '120000000000', no terminator. Gnutar misinterprets this as 0o12000000000 (a mere 1.25 Gb), but that's overridden by the PAX header.

I'm not sure what bsdtar would write in that block if the file size couldn't fit in 12 octal digits, but I'm also not eager to create a 64 gb file on my laptop to find out.

[isaacs]

Actually I did get curious and checked. Bsdtar does the same thing as gnutar, but only when the file is 64gb or greater.

In short, this patch is good, it's landed, and published to latest. Thank you for digging in and fixing this. The reason it escaped my notice for so long is frankly that pax headers are so much more straightforward and make this type of bug irrelevant in so many cases.

The checks for Number.MAX_SAFE_INTEGER made me chuckle. I thought for a second that throwing a new error should be semver major bump, but if people are using this library to pack and unpack tarballs with 8 petabyte files in them, then the world is definitely in trouble.

[justfalter]

Thanks for taking the time to go over this, @isaacs. I’ve got something upstream from my project that is explicitly producing gnu-formatted tarballs, for some reason.

Honestly, the only reason I even thought to do the min/max int checks were because you had tests that would have violated those checks. 8 petabyte files are pretty unlikely, but I wondered if there weren’t other numbers that might be encoded (uid, his, etc) that might someday exceed max int. I dug a bit into libarchive, and they have similar checks, there, as well.

[justfalter]

This PR fixes issues I've identified with the handling of base-256 encoded numbers within node-tar (see #188). The issues would generally present themselves when attempting to extract a gnu-formatted tar entry for a file greater than 8gb in size.

• The last byte of the buffer was incorrectly being ignored when encoding/decoding.
• Javascript can only have safe integer precision for numbers between -9007199254740991 and 9007199254740991. Any numbers outside these bounds will see the lowest-order bits rounded off. For example, 9007199254749999 will be rounded to 9007199254750000.
  • This is likely what accounted for the belief that the last byte of the buffer was always set to 00 or 20.
  • See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number/MAX_SAFE_INTEGER for more details.
• I've modified large-integer.jsparse and encode functions to throw TypeError exceptions if they encounter a number that will not be precisely represented as a javascript integer, or if the buffer being decoded does not appear to be a base-256 encoded number (must start with 80 or ff).

[isaacs]

I see what happened here.

To encode files over 8gb, bsdtar drops the trailing 0x20, and uses that as a part of the octal-in-ascii number. This is an ambiguous part of the spec (such as it is), which gnutar interprets differently. Thankfully, bsdtar also prepends a PAX extended file attributes entry, which gnutar interprets properly. So, for example, a 10gb file would fill those 12 bytes with '120000000000', no terminator. Gnutar misinterprets this as 0o12000000000 (a mere 1.25 Gb), but that's overridden by the PAX header.

I'm not sure what bsdtar would write in that block if the file size couldn't fit in 12 octal digits, but I'm also not eager to create a 64 gb file on my laptop to find out.

[isaacs]

Actually I did get curious and checked. Bsdtar does the same thing as gnutar, but only when the file is 64gb or greater.

In short, this patch is good, it's landed, and published to latest. Thank you for digging in and fixing this. The reason it escaped my notice for so long is frankly that pax headers are so much more straightforward and make this type of bug irrelevant in so many cases.

The checks for Number.MAX_SAFE_INTEGER made me chuckle. I thought for a second that throwing a new error should be semver major bump, but if people are using this library to pack and unpack tarballs with 8 petabyte files in them, then the world is definitely in trouble.

[justfalter]

Thanks for taking the time to go over this, @isaacs. I’ve got something upstream from my project that is explicitly producing gnu-formatted tarballs, for some reason.

Honestly, the only reason I even thought to do the min/max int checks were because you had tests that would have violated those checks. 8 petabyte files are pretty unlikely, but I wondered if there weren’t other numbers that might be encoded (uid, his, etc) that might someday exceed max int. I dug a bit into libarchive, and they have similar checks, there, as well.

[justfalter]

This PR fixes issues I've identified with the handling of base-256 encoded numbers within node-tar (see #188). The issues would generally present themselves when attempting to extract a gnu-formatted tar entry for a file greater than 8gb in size.

• The last byte of the buffer was incorrectly being ignored when encoding/decoding.
• Javascript can only have safe integer precision for numbers between -9007199254740991 and 9007199254740991. Any numbers outside these bounds will see the lowest-order bits rounded off. For example, 9007199254749999 will be rounded to 9007199254750000.
  • This is likely what accounted for the belief that the last byte of the buffer was always set to 00 or 20.
  • See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number/MAX_SAFE_INTEGER for more details.
• I've modified large-integer.jsparse and encode functions to throw TypeError exceptions if they encounter a number that will not be precisely represented as a javascript integer, or if the buffer being decoded does not appear to be a base-256 encoded number (must start with 80 or ff).

[isaacs]

I see what happened here.

To encode files over 8gb, bsdtar drops the trailing 0x20, and uses that as a part of the octal-in-ascii number. This is an ambiguous part of the spec (such as it is), which gnutar interprets differently. Thankfully, bsdtar also prepends a PAX extended file attributes entry, which gnutar interprets properly. So, for example, a 10gb file would fill those 12 bytes with '120000000000', no terminator. Gnutar misinterprets this as 0o12000000000 (a mere 1.25 Gb), but that's overridden by the PAX header.

I'm not sure what bsdtar would write in that block if the file size couldn't fit in 12 octal digits, but I'm also not eager to create a 64 gb file on my laptop to find out.

[isaacs]

Actually I did get curious and checked. Bsdtar does the same thing as gnutar, but only when the file is 64gb or greater.

In short, this patch is good, it's landed, and published to latest. Thank you for digging in and fixing this. The reason it escaped my notice for so long is frankly that pax headers are so much more straightforward and make this type of bug irrelevant in so many cases.

The checks for Number.MAX_SAFE_INTEGER made me chuckle. I thought for a second that throwing a new error should be semver major bump, but if people are using this library to pack and unpack tarballs with 8 petabyte files in them, then the world is definitely in trouble.

[justfalter]

Thanks for taking the time to go over this, @isaacs. I’ve got something upstream from my project that is explicitly producing gnu-formatted tarballs, for some reason.

Honestly, the only reason I even thought to do the min/max int checks were because you had tests that would have violated those checks. 8 petabyte files are pretty unlikely, but I wondered if there weren’t other numbers that might be encoded (uid, his, etc) that might someday exceed max int. I dug a bit into libarchive, and they have similar checks, there, as well.

[justfalter]

This PR fixes issues I've identified with the handling of base-256 encoded numbers within node-tar (see #188). The issues would generally present themselves when attempting to extract a gnu-formatted tar entry for a file greater than 8gb in size.

• The last byte of the buffer was incorrectly being ignored when encoding/decoding.
• Javascript can only have safe integer precision for numbers between -9007199254740991 and 9007199254740991. Any numbers outside these bounds will see the lowest-order bits rounded off. For example, 9007199254749999 will be rounded to 9007199254750000.
  • This is likely what accounted for the belief that the last byte of the buffer was always set to 00 or 20.
  • See https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Number/MAX_SAFE_INTEGER for more details.
• I've modified large-integer.jsparse and encode functions to throw TypeError exceptions if they encounter a number that will not be precisely represented as a javascript integer, or if the buffer being decoded does not appear to be a base-256 encoded number (must start with 80 or ff).

[isaacs]

I see what happened here.

To encode files over 8gb, bsdtar drops the trailing 0x20, and uses that as a part of the octal-in-ascii number. This is an ambiguous part of the spec (such as it is), which gnutar interprets differently. Thankfully, bsdtar also prepends a PAX extended file attributes entry, which gnutar interprets properly. So, for example, a 10gb file would fill those 12 bytes with '120000000000', no terminator. Gnutar misinterprets this as 0o12000000000 (a mere 1.25 Gb), but that's overridden by the PAX header.

I'm not sure what bsdtar would write in that block if the file size couldn't fit in 12 octal digits, but I'm also not eager to create a 64 gb file on my laptop to find out.

[isaacs]

Actually I did get curious and checked. Bsdtar does the same thing as gnutar, but only when the file is 64gb or greater.

In short, this patch is good, it's landed, and published to latest. Thank you for digging in and fixing this. The reason it escaped my notice for so long is frankly that pax headers are so much more straightforward and make this type of bug irrelevant in so many cases.

The checks for Number.MAX_SAFE_INTEGER made me chuckle. I thought for a second that throwing a new error should be semver major bump, but if people are using this library to pack and unpack tarballs with 8 petabyte files in them, then the world is definitely in trouble.

[justfalter]

Thanks for taking the time to go over this, @isaacs. I’ve got something upstream from my project that is explicitly producing gnu-formatted tarballs, for some reason.

Honestly, the only reason I even thought to do the min/max int checks were because you had tests that would have violated those checks. 8 petabyte files are pretty unlikely, but I wondered if there weren’t other numbers that might be encoded (uid, his, etc) that might someday exceed max int. I dug a bit into libarchive, and they have similar checks, there, as well.