Skip to content

Fix XComObjectStoreBackend config var in docs and bug when xcom_objectstore_compression is empty - #38142

Merged
pankajastro merged 3 commits into
apache:mainfrom
astronomer:fix_object_storage_docs
Mar 14, 2024
Merged

Fix XComObjectStoreBackend config var in docs and bug when xcom_objectstore_compression is empty#38142
pankajastro merged 3 commits into
apache:mainfrom
astronomer:fix_object_storage_docs

Conversation

@pankajastro

@pankajastropankajastro commented Mar 14, 2024

Copy link
Copy Markdown
Member

Look like the XComObjectStoreBackend config
variables are incorrect in docs or updated but
the docs still have old names. The implementation
code *store* but docs has *storage*

defserialize_value(
value: T,
*,
key: str|None=None,
task_id: str|None=None,
dag_id: str|None=None,
run_id: str|None=None,
map_index: int|None=None,
) ->bytes|str:
# we will always serialize ourselves and not by BaseXCom as the deserialize method
# from BaseXCom accepts only XCom objects and not the value directly
s_val=json.dumps(value, cls=XComEncoder).encode("utf-8")
path=conf.get(SECTION, "xcom_objectstore_path", fallback="")
compression=conf.get(SECTION, "xcom_objectstore_compression", fallback=None)
ifcompression:
suffix="."+_get_compression_suffix(compression)
else:
suffix=""
threshold=conf.getint(SECTION, "xcom_objectstore_threshold", fallback=-1)
ifpathand-1<threshold<len(s_val):
# safeguard against collisions
whileTrue:
p=ObjectStoragePath(path) /f"{dag_id}/{run_id}/{task_id}/{str(uuid.uuid4())}{suffix}"
ifnotp.exists():
break
ifnotp.parent.exists():
p.parent.mkdir(parents=True, exist_ok=True)
withp.open(mode="wb", compression=compression) asf:
f.write(s_val)
returnBaseXCom.serialize_value(str(p))
else:
returns_val

Also, tests are when xcom_objectstore_compression an empty string so I have set it to None if it empty string


^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

@pankajastro
pankajastro merged commit 0df0e09 into apache:mainMar 14, 2024
@pankajastro
pankajastro deleted the fix_object_storage_docs branch March 14, 2024 15:50
@ephraimbuddyephraimbuddy added the changelog:skip Changes that should be skipped from the changelog (CI, tests, etc..) label Mar 14, 2024
@pankajastropankajastro changed the title Fix XComObjectStoreBackend config var in docsFix XComObjectStoreBackend config var in docs and bug when xcom_objectstore_compression is emptyMar 15, 2024
@bolkedebruin

bolkedebruin commented Mar 19, 2024

Copy link
Copy Markdown
Contributor

Maybe it would have been better to move the implementation to xcom_objectstorage_path instead of adopting a new name? So making it consistent instead of deviating. Storage is used elsewhere and this very subtle naming difference might not be caught?

@pankajastro

Copy link
Copy Markdown
MemberAuthor

Indeed, I noticed that throughout the codebase, we've used the name objectstorage in various places. During testing, I discovered that the expected config variable name should be objectstore instead of objectstorage, so I made the necessary changes. However, for consistency, I propose reverting back to objectstore since we've not released it yet.

@pankajastro

Copy link
Copy Markdown
MemberAuthor

@bolkedebruin create a PR #38415 PTAL

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providerschangelog:skipChanges that should be skipped from the changelog (CI, tests, etc..)kind:documentationprovider:common-io

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants

@pankajastro@bolkedebruin@potiuk@ephraimbuddy