Make content_ids migration incremental and restartable - #8021

Merged
gerrod3 merged 1 commit into
pulp:mainfrom
gerrod3:fix/0152-content-ids-migration
Aug 31, 2026
Merged

Make content_ids migration incremental and restartable#8021
gerrod3 merged 1 commit into
pulp:mainfrom
gerrod3:fix/0152-content-ids-migration

Conversation

@gerrod3

Copy link
Copy Markdown
Contributor

Summary

  • Rewrite the 0152 content_ids data fill to compute each repository version in Postgres from the previous version (plus added, minus removed) instead of an N+1 ORM loop that ships UUID arrays through Python.
  • Commit per repository (atomic = False) so a DBaaS statement/session timeout does not roll back already-filled repos. Re-running migrate continues from remaining NULLs, then AlterField makes the column required.
  • Alternative to fix(0152_alter_repositoryversion_content_ids): add a batch size to avoid timeout on large update #8017: shrinking bulk_update batches does not fix the per-version recompute or the single wrapping transaction.

Test plan

  • Unit test covering add/remove across versions, already-filled post-3.83 versions as the incremental base, and a second repo left untouched
  • CI unit tests on this PR
  • Confirm an upgrade that previously died on 0152 can re-run migrate without losing progress

Made with Cursor

@gerrod3
gerrod3 marked this pull request as draft August 27, 2026 14:46

class Migration(migrations.Migration):
# Per-repository commits inside RunPython; AlterField runs afterwards.
atomic = False

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This opens up a window, right?
But as long as nothing runs at the same time, or (in the case of ZDU) the currently running code is sufficiantly new there is nothing to introduce new repository versions with NULL. I think this is actually fine.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe, if the user is doing a big jump and trying to do ZDU they might run into a problem if an old worker gets a task to create a new repo-vesion. Theoretically if that were to happen and then the ALTER TABLE call fails, they could just rerun the migration again and we would only need to update that one new repo-version that slipped through.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"big jump and ZDU" is not supported. Let's please not use vague language around that.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gotcha, then there should be no window if they upgrade properly.


import django.contrib.postgres.fields
from django.db import migrations, models
from django.db import migrations, models, transaction

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking about something along the lines of
UPDATE core.repositoryversion AS rv SET content_ids=(SELECT ARRAY_AGG(content_id) FROM core.repositoryversion_content WHERE version_added <= rv.version and version_removed > rv.version) WHERE rv.content_ids is NULL;

Maybe needs a few more joins and some care for version_removed = NULL.

Am oversimplifying the challenge here?

Maybe one can even batch that by adding a LIMIT 1000 clause.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the query to be the exact SQL version of _content_relationships. Calculating the memberships can take a little while so not sure if batching them would be good if we want to avoid timeouts.

Perform the cache populating entirely in SQL and add per repository commits.
Assisted By: Cursor Grok 4.6
Co-authored-by: Cursor <cursoragent@cursor.com>
@gerrod3
gerrod3force-pushed the fix/0152-content-ids-migration branch from 87e316f to ceba57eCompareAugust 31, 2026 15:54
@gerrod3
gerrod3 marked this pull request as ready for review August 31, 2026 17:14
@gerrod3
gerrod3 merged commit edef435 into pulp:mainAug 31, 2026
13 of 14 checks passed
@patchback

patchbackBot commented Aug 31, 2026

Copy link
Copy Markdown

Backport to 3.116: 💚 backport PR created

✅ Backport PR branch: patchback/backports/3.116/edef435849726ca02cda937da16bdd7a0abc1ea4/pr-8021

Backported as #8031

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

@gerrod3
gerrod3 deleted the fix/0152-content-ids-migration branch August 31, 2026 17:17
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@gerrod3@mdellweg
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Add copy buttons to all
 blocks\n(function() {\n function addCopyButtons() {\n document.querySelectorAll('pre code').forEach(function(codeBlock) {\n if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;\n codeBlock.parentElement.setAttribute('data-copy-added', 'true');\n \n var btn = document.createElement('button');\n btn.textContent = 'Copy';\n btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';\n btn.onmouseover = function() { this.style.opacity = '1'; };\n btn.onmouseout = function() { this.style.opacity = '0.7'; };\n btn.onclick = function() {\n navigator.clipboard.writeText(codeBlock.textContent).then(function() {\n btn.textContent = 'Copied!';\n setTimeout(function() { btn.textContent = 'Copy'; }, 1500);\n });\n };\n codeBlock.parentElement.style.position = 'relative';\n codeBlock.parentElement.appendChild(btn);\n });\n }\n \n addCopyButtons();\n \n // Re-run on dynamic content\n var observer = new MutationObserver(addCopyButtons);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Add Copy Buttons to Code Blocks");
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Skip to content

Make content_ids migration incremental and restartable - #8021

Merged
gerrod3 merged 1 commit into
pulp:mainfrom
gerrod3:fix/0152-content-ids-migration
Aug 31, 2026
Merged

Make content_ids migration incremental and restartable#8021
gerrod3 merged 1 commit into
pulp:mainfrom
gerrod3:fix/0152-content-ids-migration

Conversation

@gerrod3

Copy link
Copy Markdown
Contributor

Summary

  • Rewrite the 0152 content_ids data fill to compute each repository version in Postgres from the previous version (plus added, minus removed) instead of an N+1 ORM loop that ships UUID arrays through Python.
  • Commit per repository (atomic = False) so a DBaaS statement/session timeout does not roll back already-filled repos. Re-running migrate continues from remaining NULLs, then AlterField makes the column required.
  • Alternative to fix(0152_alter_repositoryversion_content_ids): add a batch size to avoid timeout on large update #8017: shrinking bulk_update batches does not fix the per-version recompute or the single wrapping transaction.

Test plan

  • Unit test covering add/remove across versions, already-filled post-3.83 versions as the incremental base, and a second repo left untouched
  • CI unit tests on this PR
  • Confirm an upgrade that previously died on 0152 can re-run migrate without losing progress

Made with Cursor

@gerrod3
gerrod3 marked this pull request as draft August 27, 2026 14:46

class Migration(migrations.Migration):
# Per-repository commits inside RunPython; AlterField runs afterwards.
atomic = False

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This opens up a window, right?
But as long as nothing runs at the same time, or (in the case of ZDU) the currently running code is sufficiantly new there is nothing to introduce new repository versions with NULL. I think this is actually fine.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe, if the user is doing a big jump and trying to do ZDU they might run into a problem if an old worker gets a task to create a new repo-vesion. Theoretically if that were to happen and then the ALTER TABLE call fails, they could just rerun the migration again and we would only need to update that one new repo-version that slipped through.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"big jump and ZDU" is not supported. Let's please not use vague language around that.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gotcha, then there should be no window if they upgrade properly.


import django.contrib.postgres.fields
from django.db import migrations, models
from django.db import migrations, models, transaction

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking about something along the lines of
UPDATE core.repositoryversion AS rv SET content_ids=(SELECT ARRAY_AGG(content_id) FROM core.repositoryversion_content WHERE version_added <= rv.version and version_removed > rv.version) WHERE rv.content_ids is NULL;

Maybe needs a few more joins and some care for version_removed = NULL.

Am oversimplifying the challenge here?

Maybe one can even batch that by adding a LIMIT 1000 clause.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the query to be the exact SQL version of _content_relationships. Calculating the memberships can take a little while so not sure if batching them would be good if we want to avoid timeouts.

Perform the cache populating entirely in SQL and add per repository commits.
Assisted By: Cursor Grok 4.6
Co-authored-by: Cursor <cursoragent@cursor.com>
@gerrod3
gerrod3force-pushed the fix/0152-content-ids-migration branch from 87e316f to ceba57eCompareAugust 31, 2026 15:54
@gerrod3
gerrod3 marked this pull request as ready for review August 31, 2026 17:14
@gerrod3
gerrod3 merged commit edef435 into pulp:mainAug 31, 2026
13 of 14 checks passed
@patchback

patchbackBot commented Aug 31, 2026

Copy link
Copy Markdown

Backport to 3.116: 💚 backport PR created

✅ Backport PR branch: patchback/backports/3.116/edef435849726ca02cda937da16bdd7a0abc1ea4/pr-8021

Backported as #8031

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

@gerrod3
gerrod3 deleted the fix/0152-content-ids-migration branch August 31, 2026 17:17
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@gerrod3@mdellweg
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Force GitHub README to respect dark mode\n(function() {\n var style = document.createElement('style');\n style.textContent = '\n .markdown-body {\n color-scheme: dark light;\n }\n .markdown-body pre { background: #161b22 !important; }\n .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }\n .markdown-body table th, .markdown-body table td { border-color: #30363d !important; }\n .markdown-body img { background: #0d1117; }\n .markdown-body blockquote { border-left-color: #8b949e; }\n .markdown-body hr { border-color: #30363d; }\n ';\n document.head.appendChild(style);\n})();", "GitHub Dark Mode README Fix"); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Make content_ids migration incremental and restartable - #8021

Merged
gerrod3 merged 1 commit into
pulp:mainfrom
gerrod3:fix/0152-content-ids-migration
Aug 31, 2026
Merged

Make content_ids migration incremental and restartable#8021
gerrod3 merged 1 commit into
pulp:mainfrom
gerrod3:fix/0152-content-ids-migration

Conversation

@gerrod3

Copy link
Copy Markdown
Contributor

Summary

  • Rewrite the 0152 content_ids data fill to compute each repository version in Postgres from the previous version (plus added, minus removed) instead of an N+1 ORM loop that ships UUID arrays through Python.
  • Commit per repository (atomic = False) so a DBaaS statement/session timeout does not roll back already-filled repos. Re-running migrate continues from remaining NULLs, then AlterField makes the column required.
  • Alternative to fix(0152_alter_repositoryversion_content_ids): add a batch size to avoid timeout on large update #8017: shrinking bulk_update batches does not fix the per-version recompute or the single wrapping transaction.

Test plan

  • Unit test covering add/remove across versions, already-filled post-3.83 versions as the incremental base, and a second repo left untouched
  • CI unit tests on this PR
  • Confirm an upgrade that previously died on 0152 can re-run migrate without losing progress

Made with Cursor

@gerrod3
gerrod3 marked this pull request as draft August 27, 2026 14:46

class Migration(migrations.Migration):
# Per-repository commits inside RunPython; AlterField runs afterwards.
atomic = False

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This opens up a window, right?
But as long as nothing runs at the same time, or (in the case of ZDU) the currently running code is sufficiantly new there is nothing to introduce new repository versions with NULL. I think this is actually fine.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe, if the user is doing a big jump and trying to do ZDU they might run into a problem if an old worker gets a task to create a new repo-vesion. Theoretically if that were to happen and then the ALTER TABLE call fails, they could just rerun the migration again and we would only need to update that one new repo-version that slipped through.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"big jump and ZDU" is not supported. Let's please not use vague language around that.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gotcha, then there should be no window if they upgrade properly.


import django.contrib.postgres.fields
from django.db import migrations, models
from django.db import migrations, models, transaction

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking about something along the lines of
UPDATE core.repositoryversion AS rv SET content_ids=(SELECT ARRAY_AGG(content_id) FROM core.repositoryversion_content WHERE version_added <= rv.version and version_removed > rv.version) WHERE rv.content_ids is NULL;

Maybe needs a few more joins and some care for version_removed = NULL.

Am oversimplifying the challenge here?

Maybe one can even batch that by adding a LIMIT 1000 clause.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the query to be the exact SQL version of _content_relationships. Calculating the memberships can take a little while so not sure if batching them would be good if we want to avoid timeouts.

Perform the cache populating entirely in SQL and add per repository commits.
Assisted By: Cursor Grok 4.6
Co-authored-by: Cursor <cursoragent@cursor.com>
@gerrod3
gerrod3force-pushed the fix/0152-content-ids-migration branch from 87e316f to ceba57eCompareAugust 31, 2026 15:54
@gerrod3
gerrod3 marked this pull request as ready for review August 31, 2026 17:14
@gerrod3
gerrod3 merged commit edef435 into pulp:mainAug 31, 2026
13 of 14 checks passed
@patchback

patchbackBot commented Aug 31, 2026

Copy link
Copy Markdown

Backport to 3.116: 💚 backport PR created

✅ Backport PR branch: patchback/backports/3.116/edef435849726ca02cda937da16bdd7a0abc1ea4/pr-8021

Backported as #8031

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

@gerrod3
gerrod3 deleted the fix/0152-content-ids-migration branch August 31, 2026 17:17
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@gerrod3@mdellweg
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Highlight search terms from Google/DuckDuckGo/Bing referrer\n(function() {\n var ref = document.referrer;\n var terms = [];\n \n if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {\n var url = new URL(ref);\n var q = url.searchParams.get('q') || url.searchParams.get('p');\n if (q) {\n terms = q.split(/\\s+/).filter(function(t) { return t.length > 2; });\n }\n }\n \n if (terms.length === 0) return;\n \n var style = document.createElement('style');\n style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';\n document.head.appendChild(style);\n \n function highlight(node) {\n if (node.nodeType === 3) { // text node\n var text = node.textContent;\n var found = false;\n terms.forEach(function(term) {\n var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\\]\\\\]/g, '\\\\') + ')', 'gi');\n if (regex.test(text)) {\n found = true;\n var frag = document.createDocumentFragment();\n var parts = text.split(regex);\n parts.forEach(function(part, i) {\n if (i % 2 === 0) {\n frag.appendChild(document.createTextNode(part));\n } else {\n var span = document.createElement('span');\n span.className = 'userscript-highlight';\n span.textContent = part;\n frag.appendChild(span);\n }\n });\n node.parentNode.replaceChild(frag, node);\n }\n });\n } else if (node.nodeType === 1 && node.childNodes) { // element\n var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];\n if (!skipTags.includes(node.tagName)) {\n Array.from(node.childNodes).forEach(highlight);\n }\n }\n }\n \n highlight(document.body);\n \n // Re-highlight on dynamic content\n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1 || node.nodeType === 3) highlight(node);\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Highlight Search Terms"); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Make content_ids migration incremental and restartable - #8021

Merged
gerrod3 merged 1 commit into
pulp:mainfrom
gerrod3:fix/0152-content-ids-migration
Aug 31, 2026
Merged

Make content_ids migration incremental and restartable#8021
gerrod3 merged 1 commit into
pulp:mainfrom
gerrod3:fix/0152-content-ids-migration

Conversation

@gerrod3

Copy link
Copy Markdown
Contributor

Summary

  • Rewrite the 0152 content_ids data fill to compute each repository version in Postgres from the previous version (plus added, minus removed) instead of an N+1 ORM loop that ships UUID arrays through Python.
  • Commit per repository (atomic = False) so a DBaaS statement/session timeout does not roll back already-filled repos. Re-running migrate continues from remaining NULLs, then AlterField makes the column required.
  • Alternative to fix(0152_alter_repositoryversion_content_ids): add a batch size to avoid timeout on large update #8017: shrinking bulk_update batches does not fix the per-version recompute or the single wrapping transaction.

Test plan

  • Unit test covering add/remove across versions, already-filled post-3.83 versions as the incremental base, and a second repo left untouched
  • CI unit tests on this PR
  • Confirm an upgrade that previously died on 0152 can re-run migrate without losing progress

Made with Cursor

@gerrod3
gerrod3 marked this pull request as draft August 27, 2026 14:46

class Migration(migrations.Migration):
# Per-repository commits inside RunPython; AlterField runs afterwards.
atomic = False

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This opens up a window, right?
But as long as nothing runs at the same time, or (in the case of ZDU) the currently running code is sufficiantly new there is nothing to introduce new repository versions with NULL. I think this is actually fine.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe, if the user is doing a big jump and trying to do ZDU they might run into a problem if an old worker gets a task to create a new repo-vesion. Theoretically if that were to happen and then the ALTER TABLE call fails, they could just rerun the migration again and we would only need to update that one new repo-version that slipped through.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"big jump and ZDU" is not supported. Let's please not use vague language around that.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gotcha, then there should be no window if they upgrade properly.


import django.contrib.postgres.fields
from django.db import migrations, models
from django.db import migrations, models, transaction

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking about something along the lines of
UPDATE core.repositoryversion AS rv SET content_ids=(SELECT ARRAY_AGG(content_id) FROM core.repositoryversion_content WHERE version_added <= rv.version and version_removed > rv.version) WHERE rv.content_ids is NULL;

Maybe needs a few more joins and some care for version_removed = NULL.

Am oversimplifying the challenge here?

Maybe one can even batch that by adding a LIMIT 1000 clause.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the query to be the exact SQL version of _content_relationships. Calculating the memberships can take a little while so not sure if batching them would be good if we want to avoid timeouts.

Perform the cache populating entirely in SQL and add per repository commits.
Assisted By: Cursor Grok 4.6
Co-authored-by: Cursor <cursoragent@cursor.com>
@gerrod3
gerrod3force-pushed the fix/0152-content-ids-migration branch from 87e316f to ceba57eCompareAugust 31, 2026 15:54
@gerrod3
gerrod3 marked this pull request as ready for review August 31, 2026 17:14
@gerrod3
gerrod3 merged commit edef435 into pulp:mainAug 31, 2026
13 of 14 checks passed
@patchback

patchbackBot commented Aug 31, 2026

Copy link
Copy Markdown

Backport to 3.116: 💚 backport PR created

✅ Backport PR branch: patchback/backports/3.116/edef435849726ca02cda937da16bdd7a0abc1ea4/pr-8021

Backported as #8031

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

@gerrod3
gerrod3 deleted the fix/0152-content-ids-migration branch August 31, 2026 17:17
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@gerrod3@mdellweg
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Strip utm_, fbclid, gclid, etc. from all links on page\n(function() {\n var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',\n 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',\n 'ref', 'ref_src', 'source', 'medium', 'campaign'];\n \n function cleanUrl(url) {\n try {\n var u = new URL(url, window.location.origin);\n var changed = false;\n trackingParams.forEach(function(p) {\n if (u.searchParams.has(p)) {\n u.searchParams.delete(p);\n changed = true;\n }\n });\n return changed ? u.toString() : url;\n } catch (e) {\n return url;\n }\n }\n \n function cleanLinks() {\n document.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n \n cleanLinks();\n \n var observer = new MutationObserver(function(mutations) {\n mutations.forEach(function(m) {\n m.addedNodes.forEach(function(node) {\n if (node.nodeType === 1) {\n if (node.tagName === 'A') cleanLinks();\n node.querySelectorAll('a[href]').forEach(function(a) {\n var clean = cleanUrl(a.href);\n if (clean !== a.href) a.href = clean;\n });\n }\n });\n });\n });\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "Remove Tracking Parameters from Links"); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + '
Skip to content

Make content_ids migration incremental and restartable - #8021

Merged
gerrod3 merged 1 commit into
pulp:mainfrom
gerrod3:fix/0152-content-ids-migration
Aug 31, 2026
Merged

Make content_ids migration incremental and restartable#8021
gerrod3 merged 1 commit into
pulp:mainfrom
gerrod3:fix/0152-content-ids-migration

Conversation

@gerrod3

Copy link
Copy Markdown
Contributor

Summary

  • Rewrite the 0152 content_ids data fill to compute each repository version in Postgres from the previous version (plus added, minus removed) instead of an N+1 ORM loop that ships UUID arrays through Python.
  • Commit per repository (atomic = False) so a DBaaS statement/session timeout does not roll back already-filled repos. Re-running migrate continues from remaining NULLs, then AlterField makes the column required.
  • Alternative to fix(0152_alter_repositoryversion_content_ids): add a batch size to avoid timeout on large update #8017: shrinking bulk_update batches does not fix the per-version recompute or the single wrapping transaction.

Test plan

  • Unit test covering add/remove across versions, already-filled post-3.83 versions as the incremental base, and a second repo left untouched
  • CI unit tests on this PR
  • Confirm an upgrade that previously died on 0152 can re-run migrate without losing progress

Made with Cursor

@gerrod3
gerrod3 marked this pull request as draft August 27, 2026 14:46

class Migration(migrations.Migration):
# Per-repository commits inside RunPython; AlterField runs afterwards.
atomic = False

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This opens up a window, right?
But as long as nothing runs at the same time, or (in the case of ZDU) the currently running code is sufficiantly new there is nothing to introduce new repository versions with NULL. I think this is actually fine.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe, if the user is doing a big jump and trying to do ZDU they might run into a problem if an old worker gets a task to create a new repo-vesion. Theoretically if that were to happen and then the ALTER TABLE call fails, they could just rerun the migration again and we would only need to update that one new repo-version that slipped through.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"big jump and ZDU" is not supported. Let's please not use vague language around that.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gotcha, then there should be no window if they upgrade properly.


import django.contrib.postgres.fields
from django.db import migrations, models
from django.db import migrations, models, transaction

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking about something along the lines of
UPDATE core.repositoryversion AS rv SET content_ids=(SELECT ARRAY_AGG(content_id) FROM core.repositoryversion_content WHERE version_added <= rv.version and version_removed > rv.version) WHERE rv.content_ids is NULL;

Maybe needs a few more joins and some care for version_removed = NULL.

Am oversimplifying the challenge here?

Maybe one can even batch that by adding a LIMIT 1000 clause.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the query to be the exact SQL version of _content_relationships. Calculating the memberships can take a little while so not sure if batching them would be good if we want to avoid timeouts.

Perform the cache populating entirely in SQL and add per repository commits.
Assisted By: Cursor Grok 4.6
Co-authored-by: Cursor <cursoragent@cursor.com>
@gerrod3
gerrod3force-pushed the fix/0152-content-ids-migration branch from 87e316f to ceba57eCompareAugust 31, 2026 15:54
@gerrod3
gerrod3 marked this pull request as ready for review August 31, 2026 17:14
@gerrod3
gerrod3 merged commit edef435 into pulp:mainAug 31, 2026
13 of 14 checks passed
@patchback

patchbackBot commented Aug 31, 2026

Copy link
Copy Markdown

Backport to 3.116: 💚 backport PR created

✅ Backport PR branch: patchback/backports/3.116/edef435849726ca02cda937da16bdd7a0abc1ea4/pr-8021

Backported as #8031

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

@gerrod3
gerrod3 deleted the fix/0152-content-ids-migration branch August 31, 2026 17:17
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@gerrod3@mdellweg
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Auto-enable theater mode on YouTube\n(function() {\n function tryTheater() {\n var btn = document.querySelector('button[aria-label=\"Theater mode\"], ytd-player #player button[title=\"Theater mode\"]');\n if (btn && !btn.classList.contains('activated')) {\n btn.click();\n }\n }\n \n // Try immediately\n tryTheater();\n \n // Try after navigation (SPA)\n var lastUrl = location.href;\n setInterval(function() {\n if (location.href !== lastUrl) {\n lastUrl = location.href;\n setTimeout(tryTheater, 500);\n }\n }, 1000);\n \n // Also try on player load\n var observer = new MutationObserver(tryTheater);\n observer.observe(document.body, { childList: true, subtree: true });\n})();", "YouTube Theater Mode Default"); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Make content_ids migration incremental and restartable - #8021

Merged
gerrod3 merged 1 commit into
pulp:mainfrom
gerrod3:fix/0152-content-ids-migration
Aug 31, 2026
Merged

Make content_ids migration incremental and restartable#8021
gerrod3 merged 1 commit into
pulp:mainfrom
gerrod3:fix/0152-content-ids-migration

Conversation

@gerrod3

Copy link
Copy Markdown
Contributor

Summary

  • Rewrite the 0152 content_ids data fill to compute each repository version in Postgres from the previous version (plus added, minus removed) instead of an N+1 ORM loop that ships UUID arrays through Python.
  • Commit per repository (atomic = False) so a DBaaS statement/session timeout does not roll back already-filled repos. Re-running migrate continues from remaining NULLs, then AlterField makes the column required.
  • Alternative to fix(0152_alter_repositoryversion_content_ids): add a batch size to avoid timeout on large update #8017: shrinking bulk_update batches does not fix the per-version recompute or the single wrapping transaction.

Test plan

  • Unit test covering add/remove across versions, already-filled post-3.83 versions as the incremental base, and a second repo left untouched
  • CI unit tests on this PR
  • Confirm an upgrade that previously died on 0152 can re-run migrate without losing progress

Made with Cursor

@gerrod3
gerrod3 marked this pull request as draft August 27, 2026 14:46

class Migration(migrations.Migration):
# Per-repository commits inside RunPython; AlterField runs afterwards.
atomic = False

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This opens up a window, right?
But as long as nothing runs at the same time, or (in the case of ZDU) the currently running code is sufficiantly new there is nothing to introduce new repository versions with NULL. I think this is actually fine.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe, if the user is doing a big jump and trying to do ZDU they might run into a problem if an old worker gets a task to create a new repo-vesion. Theoretically if that were to happen and then the ALTER TABLE call fails, they could just rerun the migration again and we would only need to update that one new repo-version that slipped through.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"big jump and ZDU" is not supported. Let's please not use vague language around that.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gotcha, then there should be no window if they upgrade properly.


import django.contrib.postgres.fields
from django.db import migrations, models
from django.db import migrations, models, transaction

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking about something along the lines of
UPDATE core.repositoryversion AS rv SET content_ids=(SELECT ARRAY_AGG(content_id) FROM core.repositoryversion_content WHERE version_added <= rv.version and version_removed > rv.version) WHERE rv.content_ids is NULL;

Maybe needs a few more joins and some care for version_removed = NULL.

Am oversimplifying the challenge here?

Maybe one can even batch that by adding a LIMIT 1000 clause.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the query to be the exact SQL version of _content_relationships. Calculating the memberships can take a little while so not sure if batching them would be good if we want to avoid timeouts.

Perform the cache populating entirely in SQL and add per repository commits.
Assisted By: Cursor Grok 4.6
Co-authored-by: Cursor <cursoragent@cursor.com>
@gerrod3
gerrod3force-pushed the fix/0152-content-ids-migration branch from 87e316f to ceba57eCompareAugust 31, 2026 15:54
@gerrod3
gerrod3 marked this pull request as ready for review August 31, 2026 17:14
@gerrod3
gerrod3 merged commit edef435 into pulp:mainAug 31, 2026
13 of 14 checks passed
@patchback

patchbackBot commented Aug 31, 2026

Copy link
Copy Markdown

Backport to 3.116: 💚 backport PR created

✅ Backport PR branch: patchback/backports/3.116/edef435849726ca02cda937da16bdd7a0abc1ea4/pr-8021

Backported as #8031

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

@gerrod3
gerrod3 deleted the fix/0152-content-ids-migration branch August 31, 2026 17:17
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@gerrod3@mdellweg
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Remove or un-stick sticky/fixed headers that block content\n(function() {\n function unstick() {\n document.querySelectorAll('header, nav, [role=\"banner\"], .header, .navbar, .sticky, .fixed-top, [style*=\"position: fixed\"], [style*=\"position:sticky\"]').forEach(function(el) {\n if (el.style.position === 'fixed' || el.style.position === 'sticky' || \n getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {\n el.style.position = 'static';\n el.style.top = 'auto';\n el.style.zIndex = 'auto';\n }\n });\n }\n \n unstick();\n \n var observer = new MutationObserver(unstick);\n observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });\n})();", "Kill Sticky Headers"); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + '
Skip to content

Make content_ids migration incremental and restartable - #8021

Merged
gerrod3 merged 1 commit into
pulp:mainfrom
gerrod3:fix/0152-content-ids-migration
Aug 31, 2026
Merged

Make content_ids migration incremental and restartable#8021
gerrod3 merged 1 commit into
pulp:mainfrom
gerrod3:fix/0152-content-ids-migration

Conversation

@gerrod3

Copy link
Copy Markdown
Contributor

Summary

  • Rewrite the 0152 content_ids data fill to compute each repository version in Postgres from the previous version (plus added, minus removed) instead of an N+1 ORM loop that ships UUID arrays through Python.
  • Commit per repository (atomic = False) so a DBaaS statement/session timeout does not roll back already-filled repos. Re-running migrate continues from remaining NULLs, then AlterField makes the column required.
  • Alternative to fix(0152_alter_repositoryversion_content_ids): add a batch size to avoid timeout on large update #8017: shrinking bulk_update batches does not fix the per-version recompute or the single wrapping transaction.

Test plan

  • Unit test covering add/remove across versions, already-filled post-3.83 versions as the incremental base, and a second repo left untouched
  • CI unit tests on this PR
  • Confirm an upgrade that previously died on 0152 can re-run migrate without losing progress

Made with Cursor

@gerrod3
gerrod3 marked this pull request as draft August 27, 2026 14:46

class Migration(migrations.Migration):
# Per-repository commits inside RunPython; AlterField runs afterwards.
atomic = False

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This opens up a window, right?
But as long as nothing runs at the same time, or (in the case of ZDU) the currently running code is sufficiantly new there is nothing to introduce new repository versions with NULL. I think this is actually fine.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe, if the user is doing a big jump and trying to do ZDU they might run into a problem if an old worker gets a task to create a new repo-vesion. Theoretically if that were to happen and then the ALTER TABLE call fails, they could just rerun the migration again and we would only need to update that one new repo-version that slipped through.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"big jump and ZDU" is not supported. Let's please not use vague language around that.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gotcha, then there should be no window if they upgrade properly.


import django.contrib.postgres.fields
from django.db import migrations, models
from django.db import migrations, models, transaction

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking about something along the lines of
UPDATE core.repositoryversion AS rv SET content_ids=(SELECT ARRAY_AGG(content_id) FROM core.repositoryversion_content WHERE version_added <= rv.version and version_removed > rv.version) WHERE rv.content_ids is NULL;

Maybe needs a few more joins and some care for version_removed = NULL.

Am oversimplifying the challenge here?

Maybe one can even batch that by adding a LIMIT 1000 clause.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the query to be the exact SQL version of _content_relationships. Calculating the memberships can take a little while so not sure if batching them would be good if we want to avoid timeouts.

Perform the cache populating entirely in SQL and add per repository commits.
Assisted By: Cursor Grok 4.6
Co-authored-by: Cursor <cursoragent@cursor.com>
@gerrod3
gerrod3force-pushed the fix/0152-content-ids-migration branch from 87e316f to ceba57eCompareAugust 31, 2026 15:54
@gerrod3
gerrod3 marked this pull request as ready for review August 31, 2026 17:14
@gerrod3
gerrod3 merged commit edef435 into pulp:mainAug 31, 2026
13 of 14 checks passed
@patchback

patchbackBot commented Aug 31, 2026

Copy link
Copy Markdown

Backport to 3.116: 💚 backport PR created

✅ Backport PR branch: patchback/backports/3.116/edef435849726ca02cda937da16bdd7a0abc1ea4/pr-8021

Backported as #8031

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

@gerrod3
gerrod3 deleted the fix/0152-content-ids-migration branch August 31, 2026 17:17
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@gerrod3@mdellweg
, 'i'); if (__m === '*' || __re.test(location.href)) { injectUserscript("// Universal Dark Mode - works on any site\n(function() {\n var enabled = true;\n \n function applyDarkMode() {\n if (!enabled) return;\n \n // Create style element if it doesn't exist\n var style = document.getElementById('universal-dark-mode-style');\n if (!style) {\n style = document.createElement('style');\n style.id = 'universal-dark-mode-style';\n document.head.appendChild(style);\n }\n \n // Dark mode CSS - inverts colors but preserves images/video\n style.textContent = '\n /* Invert everything except media */\n html {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #1a1a2e !important;\n }\n \n /* Restore images, videos, iframes, canvas */\n img, video, iframe, canvas, svg, picture, [style*=\"background-image\"] {\n filter: invert(1) hue-rotate(180deg) !important;\n }\n \n /* Preserve specific elements that should not be inverted */\n .no-dark-mode, .no-dark-mode *,\n [data-theme=\"light\"], [data-theme=\"light\"],\n .ace_editor, .ace_editor *,\n .CodeMirror, .CodeMirror *,\n .monaco-editor, .monaco-editor *,\n .markdown-body pre, .markdown-body pre *,\n .highlight, .highlight *,\n pre code, pre code * {\n filter: none !important;\n }\n \n /* Fix common UI elements */\n .modal, .popup, .dropdown-menu, .tooltip, .popover {\n filter: invert(1) hue-rotate(180deg) !important;\n background: #2d2d44 !important;\n border-color: #444 !important;\n }\n \n /* Scrollbars */\n ::-webkit-scrollbar { background: #1a1a2e !important; }\n ::-webkit-scrollbar-thumb { background: #444 !important; }\n ::-webkit-scrollbar-thumb:hover { background: #555 !important; }\n \n /* Selection */\n ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }\n ';\n }\n \n function removeDarkMode() {\n var style = document.getElementById('universal-dark-mode-style');\n if (style) style.remove();\n }\n \n // Toggle with Alt+Shift+D\n document.addEventListener('keydown', function(e) {\n if (e.altKey && e.shiftKey && e.key === 'D') {\n e.preventDefault();\n enabled = !enabled;\n if (enabled) {\n applyDarkMode();\n console.log('[Universal Dark Mode] Enabled');\n } else {\n removeDarkMode();\n console.log('[Universal Dark Mode] Disabled');\n }\n }\n });\n \n // Apply on load\n applyDarkMode();\n \n // Re-apply on dynamic content\n var observer = new MutationObserver(function(mutations) {\n if (enabled && !document.getElementById('universal-dark-mode-style')) {\n applyDarkMode();\n }\n });\n observer.observe(document.head, { childList: true });\n \n console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');\n})();", "Universal Dark Mode"); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })();
Skip to content

Make content_ids migration incremental and restartable - #8021

Merged
gerrod3 merged 1 commit into
pulp:mainfrom
gerrod3:fix/0152-content-ids-migration
Aug 31, 2026
Merged

Make content_ids migration incremental and restartable#8021
gerrod3 merged 1 commit into
pulp:mainfrom
gerrod3:fix/0152-content-ids-migration

Conversation

@gerrod3

Copy link
Copy Markdown
Contributor

Summary

  • Rewrite the 0152 content_ids data fill to compute each repository version in Postgres from the previous version (plus added, minus removed) instead of an N+1 ORM loop that ships UUID arrays through Python.
  • Commit per repository (atomic = False) so a DBaaS statement/session timeout does not roll back already-filled repos. Re-running migrate continues from remaining NULLs, then AlterField makes the column required.
  • Alternative to fix(0152_alter_repositoryversion_content_ids): add a batch size to avoid timeout on large update #8017: shrinking bulk_update batches does not fix the per-version recompute or the single wrapping transaction.

Test plan

  • Unit test covering add/remove across versions, already-filled post-3.83 versions as the incremental base, and a second repo left untouched
  • CI unit tests on this PR
  • Confirm an upgrade that previously died on 0152 can re-run migrate without losing progress

Made with Cursor

@gerrod3
gerrod3 marked this pull request as draft August 27, 2026 14:46

class Migration(migrations.Migration):
# Per-repository commits inside RunPython; AlterField runs afterwards.
atomic = False

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This opens up a window, right?
But as long as nothing runs at the same time, or (in the case of ZDU) the currently running code is sufficiantly new there is nothing to introduce new repository versions with NULL. I think this is actually fine.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe, if the user is doing a big jump and trying to do ZDU they might run into a problem if an old worker gets a task to create a new repo-vesion. Theoretically if that were to happen and then the ALTER TABLE call fails, they could just rerun the migration again and we would only need to update that one new repo-version that slipped through.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"big jump and ZDU" is not supported. Let's please not use vague language around that.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Gotcha, then there should be no window if they upgrade properly.


import django.contrib.postgres.fields
from django.db import migrations, models
from django.db import migrations, models, transaction

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was thinking about something along the lines of
UPDATE core.repositoryversion AS rv SET content_ids=(SELECT ARRAY_AGG(content_id) FROM core.repositoryversion_content WHERE version_added <= rv.version and version_removed > rv.version) WHERE rv.content_ids is NULL;

Maybe needs a few more joins and some care for version_removed = NULL.

Am oversimplifying the challenge here?

Maybe one can even batch that by adding a LIMIT 1000 clause.

Copy link
Copy Markdown
ContributorAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated the query to be the exact SQL version of _content_relationships. Calculating the memberships can take a little while so not sure if batching them would be good if we want to avoid timeouts.

Perform the cache populating entirely in SQL and add per repository commits.
Assisted By: Cursor Grok 4.6
Co-authored-by: Cursor <cursoragent@cursor.com>
@gerrod3
gerrod3force-pushed the fix/0152-content-ids-migration branch from 87e316f to ceba57eCompareAugust 31, 2026 15:54
@gerrod3
gerrod3 marked this pull request as ready for review August 31, 2026 17:14
@gerrod3
gerrod3 merged commit edef435 into pulp:mainAug 31, 2026
13 of 14 checks passed
@patchback

patchbackBot commented Aug 31, 2026

Copy link
Copy Markdown

Backport to 3.116: 💚 backport PR created

✅ Backport PR branch: patchback/backports/3.116/edef435849726ca02cda937da16bdd7a0abc1ea4/pr-8021

Backported as #8031

🤖 @patchback
I'm built with octomachinery and
my source is open — https://github.com/sanitizers/patchback-github-app.

@gerrod3
gerrod3 deleted the fix/0152-content-ids-migration branch August 31, 2026 17:17
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@gerrod3@mdellweg