Skip to content

Add redact feature for file uploads - #10

Open
tonyskapunk wants to merge 1 commit into
mainfrom
redact_file
Open

Add redact feature for file uploads#10
tonyskapunk wants to merge 1 commit into
mainfrom
redact_file

Conversation

@tonyskapunk

@tonyskapunktonyskapunk commented Feb 18, 2026•
edited
Loading

Copy link
Copy Markdown
Contributor

Provide functions to strip sensitive data (tokens, credentials, pull secrets) from content before uploading to the DCI control server.

A helper determines whether redaction is active, checking an environment variable first and falling back to a caller-provided default. This lets callers enable or disable redaction globally

The API offers a file path and a string variant, both built on top of a line-by-line streaming. This keeps memory usage constant regardless of file size, which matters for large log and configuration files.

Custom regex patterns can be supplied via environment variable, replacing the built-in defaults, to support project-specific secrets.

Assisted-by: Claude

Gitleaks-Sign: OC4zMC4wfDIwMjYtMDYtMjRUMTQ6MDQ6MDl8ZjIwYjBhNTFiYTYxMjk5NjhmY2JhYWM5ZmI0ZDZiMjhhZmQwNjA4NA==
Gitleaks-Hash: 58e7a79cda4babef8a0a89ee58ac56d654994bde75c5f7dbc6475047626f62ba

This feature was initially included in dci-ansible as part of the dci callback plugin via distributedci/dci-ansible#4 but it will be required as well in dci-pipeline, thus making dciclient a better candidate for this functionality so its code can be reused in different places.

Later it mimic the file api, providing a single function to do multiple things, while convenient on the code it was not clear for the consumer, forcing to understand really well how to make use of it.

Now, it offers functions to redact based on content or file path. Making the api less complex.

@tonyskapunktonyskapunk self-assigned this Feb 18, 2026
@tonyskapunk
tonyskapunk marked this pull request as ready for review February 18, 2026 17:20
@tonyskapunktonyskapunk mentioned this pull request Feb 18, 2026
Open
@dcibot

dcibot commented Mar 11, 2026

Copy link
Copy Markdown

from change #10:

  • no check (not a code change)

@dcibot

dcibot commented Mar 11, 2026

Copy link
Copy Markdown

from change distributedci/dci-ansible#5:

@dcibot

dcibot commented Mar 11, 2026

Copy link
Copy Markdown

from change #10:

  • no check (not a code change)

rh-gvincent
rh-gvincent reviewed Mar 12, 2026
View reviewed changes
Comment threaddciclient/v1/api/redact.py Outdated
rh-gvincent
rh-gvincent requested changes Mar 12, 2026
View reviewed changes
Comment threaddciclient/v1/api/file.py
@dcibot

dcibot commented Mar 17, 2026

Copy link
Copy Markdown

from change #10:

  • no check (not a code change)

@tonyskapunk

tonyskapunk commented Mar 19, 2026

Copy link
Copy Markdown
ContributorAuthor

recheck

@dcibot

dcibot commented Mar 19, 2026

Copy link
Copy Markdown

from change #10:

  • no check (not a code change)

@tonyskapunk

tonyskapunk commented Mar 26, 2026

Copy link
Copy Markdown
ContributorAuthor

recheck

@dcibot

dcibot commented Mar 26, 2026

Copy link
Copy Markdown

from change #10:

  • no check (not a code change)

@dcibot

dcibot commented Apr 4, 2026

Copy link
Copy Markdown

from change distributedci/dci-ansible#5:

@dcibot

dcibot commented Apr 4, 2026

Copy link
Copy Markdown

from change distributedci/dci-ansible#5:

@dcibot

dcibot commented Apr 4, 2026

Copy link
Copy Markdown

from change distributedci/dci-ansible#5:

@dcibot

dcibot commented Apr 9, 2026

Copy link
Copy Markdown

from change #10:

  • no check (not a code change)

@dcibot

dcibot commented Apr 13, 2026

Copy link
Copy Markdown

from change distributedci/dci-ansible#5:

@dcibot

dcibot commented Apr 13, 2026

Copy link
Copy Markdown

from change distributedci/dci-ansible#5:

@dcibot

dcibot commented May 8, 2026

Copy link
Copy Markdown

from change distributedci/dci-ansible#5:

@dcibot

dcibot commented May 10, 2026

Copy link
Copy Markdown

from change distributedci/dci-ansible#5:

@dcibot

dcibot commented May 11, 2026

Copy link
Copy Markdown

from change distributedci/dci-ansible#5:

@dcibot

dcibot commented May 11, 2026

Copy link
Copy Markdown

from change distributedci/dci-ansible#5:

@dcibot

dcibot commented May 12, 2026

Copy link
Copy Markdown

from change distributedci/dci-ansible#5:

@dcibot

dcibot commented May 12, 2026

Copy link
Copy Markdown

from change distributedci/dci-ansible#5:

@tonyskapunk

tonyskapunk commented Jun 17, 2026

Copy link
Copy Markdown
ContributorAuthor
* I'm also wondering why we need both `redact_content()` and `redact_stream()`. My expectation was to have a single streaming redaction API to avoid loading large files entirely into memory. Having two functions makes the intended usage less clear.

redact_content() operates on a complete string/bytes object, while redact_stream() processes a file-like object line by line. However, redact_stream() still returns a new BytesIO containing the entire redacted content, meaning the whole file eventually ends up in memory anyway.

Indeed 😞 my bad, I thought I could avoid the load into memory with file-like objects.

I'll get rid of it and only use a single call. I'm including another way to load small files into memory but larger ones will be written to disk, consumers will be able to pass either to file when uploading them to the control server transparently, I can modify it if needed or if it turns not to be a good approach.

@dcibot

dcibot commented Jun 17, 2026

Copy link
Copy Markdown

from change #10:

  • no check (not a code change)

@dcibot

dcibot commented Jun 17, 2026

Copy link
Copy Markdown

from change #10:

  • no check (not a code change)

@dcibot

dcibot commented Jun 17, 2026

Copy link
Copy Markdown

from change #10:

  • no check (not a code change)

@dcibot

dcibot commented Jun 17, 2026

Copy link
Copy Markdown

from change distributedci/dci-ansible#5:

@dcibot

dcibot commented Jun 17, 2026

Copy link
Copy Markdown

from change #10:

  • no check (not a code change)

@dcibot

dcibot commented Jun 17, 2026

Copy link
Copy Markdown

from change distributedci/dci-ansible#5:

@dcibot

dcibot commented Jun 24, 2026

Copy link
Copy Markdown

from change #10:

  • no check (not a code change)

@tonyskapunktonyskapunk changed the title Add redact feature to create fileAdd redact feature for file uploadsJun 24, 2026
@dcibot

dcibot commented Jun 24, 2026

Copy link
Copy Markdown

from change distributedci/dci-ansible#5:

@dcibot

dcibot commented Jun 25, 2026

Copy link
Copy Markdown

from change distributedci/dci-ansible#5:

@dcibot

dcibot commented Jun 25, 2026

Copy link
Copy Markdown

from change distributedci/dci-ansible#5:

@dcibot

dcibot commented Jun 25, 2026

Copy link
Copy Markdown

from change distributedci/dci-ansible#5:

@tonyskapunktonyskapunk mentioned this pull request Aug 10, 2026
Merged
2 tasks
@tonyskapunk
Add redact feature for file uploads
63433b0
Provide functions to strip sensitive data (tokens, credentials, pull
secrets) from content before uploading to the DCI control server.
A helper determines whether redaction is active, checking an
environment variable first and falling back to a caller-provided
default. This lets callers enable or disable redaction globally
The API offers a file path and a string variant, both built on top of
a line-by-line streaming. This keeps memory usage constant regardless
of file size, which matters for large log and configuration files.
Custom regex patterns can be supplied via environment variable,
replacing the built-in defaults, to support project-specific secrets.
Assisted-by: Claude
Gitleaks-Sign: OC4zMC4wfDIwMjYtMDgtMTBUMjI6MTM6MTF8YzBlYjhlYjQ5YjVmMTk3N2VlOTJiZjk0NmM2MmZmZDUwMTY5ZDU4Yw==
Gitleaks-Hash: 6f1eb83afc605505c0daf1dae6905745408162f062db4334044236205d3a36b8
@dcibot

dcibot commented Aug 21, 2026

Copy link
Copy Markdown

from change distributedci/dci-ansible#5:

@tonyskapunk

tonyskapunk commented Aug 24, 2026

Copy link
Copy Markdown
ContributorAuthor

recheck

@dcibot

dcibot commented Aug 24, 2026

Copy link
Copy Markdown

from change distributedci/dci-ansible#5:

@dcibot

dcibot commented Aug 25, 2026

Copy link
Copy Markdown

from change distributedci/dci-ansible#5:

@dcibot

dcibot commented Aug 25, 2026

Copy link
Copy Markdown

from change distributedci/dci-ansible#5:

@dcibot

dcibot commented Aug 25, 2026

Copy link
Copy Markdown

from change distributedci/dci-ansible#5:

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@rh-gvincentrh-gvincentAwaiting requested review from rh-gvincent

Requested changes must be addressed to merge this pull request.

Assignees

@tonyskapunktonyskapunk

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@tonyskapunk@dcibot@rh-gvincent