Blob's output stream's write function sometimes returns invalid values.

[juergw]

Environment details

• OS type and version: Linux, Ubuntu 20.04
• Python version: 3.9.21
• pip version: I used bazel.
• google-cloud-storage` version: 3.1.0

Steps to reproduce

1. create a bucket.
2. upload a large file (in 1k chunks) to the bucket with the Python client. It must be larger than "chunk_size".
3. check that the value returned by the write() method returns the correct value.
4. this works, until the client uploads the first chunk to the bucket. Then the write method returns a value that is too large.

Code example

import random
from absl import app
from google.cloud import storage

SMALL_CHUNK_SIZE = 256 * 1024 # smallest chunk size allowed.
BLOCK_SIZE = 1024 # 1kb
BLOCK_NUM = 40_000
HEADER = b'start'
FOOTER = b'end'
CREDS = ...
BUCKET_NAME = ...
BLOB_NAME = ...

def main(argv):
 del argv # Unused.
 storage_client = storage.Client.from_service_account_json(CREDS)
 bucket = storage_client.bucket(BUCKET_NAME)
 blob = bucket.blob(BLOB_NAME, chunk_size=SMALL_CHUNK_SIZE)
 output_stream = blob.open('wb', ignore_flush=True)

 output_stream.write(HEADER)
 for _ in range(BLOCK_NUM):
 rand_block = random.randbytes(BLOCK_SIZE)
 written = output_stream.write(rand_block)
 if written > len(rand_block):
 raise ValueError(
 'written is larger than expected: written = %d, len(rand_block)'
 ' = %d' % (written, len(rand_block))
 )
 output_stream.write(FOOTER)
 output_stream.close()

if __name__ == '__main__':
 app.run(main)

Stack trace

 raise ValueError(
ValueError: written is larger than expected: written = 263168, len(rand_block) = 1024

[chandra-siri]

@Pulkit0110 as discussed , assigning it to you.

[Pulkit0110]

github.com//pull/1490 should fix this issue. After applying the changes in the PR, I'm not getting the errors.