Skip to content

Add Openverse Fetch Script (Initial Implementation) - #185

Merged
TimidRobot merged 35 commits into
creativecommons:mainfrom
Babi-B:openverse-fetch
Oct 31, 2025
Merged

Add Openverse Fetch Script (Initial Implementation)#185
TimidRobot merged 35 commits into
creativecommons:mainfrom
Babi-B:openverse-fetch

Conversation

@Babi-B

@Babi-BBabi-B commented Oct 14, 2025

Copy link
Copy Markdown
Contributor

Fixes

Description

This PR introduces a new fetch script for collecting licensed data from the Openverse API

Technical details

  • Implements modular design; only the fetch phase has been implemented
  • The implementation fetches a small dataset (images) from the Openverse API
  • The fetch data is stored in data/2025Q4/1-fetch/openverse_fetch.csv

Tests

  • Referenced existing code (github_fetch.py) for best practices

Checklist

  • I have read and understood the Developer Certificate of Origin (DCO), below, which covers the contents of this pull request (PR).
  • My pull request doesn't include code or content generated with AI.
  • My pull request has a descriptive title (not a vague title like Update index.md).
  • My pull request targets the default branch of the repository (main or master).
  • My commit messages follow best practices.
  • My code follows the established code style of the repository.
  • I added or updated tests for the changes I made (if applicable).
  • I added or updated documentation (if applicable).
  • I tried running the project locally and verified that there are no
    visible errors.

Developer Certificate of Origin

For the purposes of this DCO, "license" is equivalent to "license or public domain dedication," and "open source license" is equivalent to "open content license or public domain dedication."

Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

@Babi-B
Babi-B requested review from a team as code ownersOctober 14, 2025 01:43
@Babi-B
Babi-B requested review from Shafiya-Heena and TimidRobot and removed request for a teamOctober 14, 2025 01:43
@cc-open-source-botcc-open-source-bot moved this to In review in TimidRobotOct 14, 2025

@TimidRobotTimidRobot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great start!

Comment threaddata/2025Q4/1-fetch/openverse_fetch.csv Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
@TimidRobotTimidRobot self-assigned this Oct 14, 2025
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated

@TimidRobotTimidRobot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good work, getting closer

Comment threadscripts/1-fetch/openverse_fetch.py
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/shared.py Outdated
Comment threadscripts/shared.py Outdated

@najuna-briannajuna-brian left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TimidRobot,
Hello, this is a good example: @Babi-B used a raise, this raises a new exception, signaling that something went wrong and stopping further normal execution, instead of using return [] which I explained here.

@TimidRobotTimidRobot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nearly done!

Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py
Comment threadscripts/1-fetch/openverse_fetch.py
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
@TimidRobot

Copy link
Copy Markdown
Member

Just a few trivial changes and then this PR can be merged!

@TimidRobotTimidRobot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your work on this!

@TimidRobot
TimidRobot merged commit c120add into creativecommons:mainOct 31, 2025
@github-project-automationgithub-project-automationBot moved this from In review to Done in TimidRobotOct 31, 2025
@Goziee-gitGoziee-git mentioned this pull request Oct 31, 2025
16 tasks
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

[Feature] Add Openverse as a Data Source

4 participants

@Babi-B@TimidRobot@najuna-brian@cc-open-source-bot
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '
Add Openverse Fetch Script (Initial Implementation) by Babi-B · Pull Request #185 · creativecommons/quantifying · GitHub
Skip to content

Add Openverse Fetch Script (Initial Implementation) - #185

Merged
TimidRobot merged 35 commits into
creativecommons:mainfrom
Babi-B:openverse-fetch
Oct 31, 2025
Merged

Add Openverse Fetch Script (Initial Implementation)#185
TimidRobot merged 35 commits into
creativecommons:mainfrom
Babi-B:openverse-fetch

Conversation

@Babi-B

@Babi-BBabi-B commented Oct 14, 2025

Copy link
Copy Markdown
Contributor

Fixes

Description

This PR introduces a new fetch script for collecting licensed data from the Openverse API

Technical details

  • Implements modular design; only the fetch phase has been implemented
  • The implementation fetches a small dataset (images) from the Openverse API
  • The fetch data is stored in data/2025Q4/1-fetch/openverse_fetch.csv

Tests

  • Referenced existing code (github_fetch.py) for best practices

Checklist

  • I have read and understood the Developer Certificate of Origin (DCO), below, which covers the contents of this pull request (PR).
  • My pull request doesn't include code or content generated with AI.
  • My pull request has a descriptive title (not a vague title like Update index.md).
  • My pull request targets the default branch of the repository (main or master).
  • My commit messages follow best practices.
  • My code follows the established code style of the repository.
  • I added or updated tests for the changes I made (if applicable).
  • I added or updated documentation (if applicable).
  • I tried running the project locally and verified that there are no
    visible errors.

Developer Certificate of Origin

For the purposes of this DCO, "license" is equivalent to "license or public domain dedication," and "open source license" is equivalent to "open content license or public domain dedication."

Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

@Babi-B
Babi-B requested review from a team as code ownersOctober 14, 2025 01:43
@Babi-B
Babi-B requested review from Shafiya-Heena and TimidRobot and removed request for a teamOctober 14, 2025 01:43
@cc-open-source-botcc-open-source-bot moved this to In review in TimidRobotOct 14, 2025

@TimidRobotTimidRobot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great start!

Comment threaddata/2025Q4/1-fetch/openverse_fetch.csv Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
@TimidRobotTimidRobot self-assigned this Oct 14, 2025
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated

@TimidRobotTimidRobot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good work, getting closer

Comment threadscripts/1-fetch/openverse_fetch.py
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/shared.py Outdated
Comment threadscripts/shared.py Outdated

@najuna-briannajuna-brian left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TimidRobot,
Hello, this is a good example: @Babi-B used a raise, this raises a new exception, signaling that something went wrong and stopping further normal execution, instead of using return [] which I explained here.

@TimidRobotTimidRobot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nearly done!

Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py
Comment threadscripts/1-fetch/openverse_fetch.py
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
@TimidRobot

Copy link
Copy Markdown
Member

Just a few trivial changes and then this PR can be merged!

@TimidRobotTimidRobot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your work on this!

@TimidRobot
TimidRobot merged commit c120add into creativecommons:mainOct 31, 2025
@github-project-automationgithub-project-automationBot moved this from In review to Done in TimidRobotOct 31, 2025
@Goziee-gitGoziee-git mentioned this pull request Oct 31, 2025
16 tasks
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

[Feature] Add Openverse as a Data Source

4 participants

@Babi-B@TimidRobot@najuna-brian@cc-open-source-bot
, 'i'); if (__m === '*' || __re.test(location.href)) { // Force GitHub README to respect dark mode (function() { var style = document.createElement('style'); style.textContent = ' .markdown-body { color-scheme: dark light; } .markdown-body pre { background: #161b22 !important; } .markdown-body code { background: rgba(110, 118, 129, 0.4) !important; } .markdown-body table th, .markdown-body table td { border-color: #30363d !important; } .markdown-body img { background: #0d1117; } .markdown-body blockquote { border-left-color: #8b949e; } .markdown-body hr { border-color: #30363d; } '; document.head.appendChild(style); })(); } } catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Add Openverse Fetch Script (Initial Implementation) by Babi-B · Pull Request #185 · creativecommons/quantifying · GitHub
Skip to content

Add Openverse Fetch Script (Initial Implementation) - #185

Merged
TimidRobot merged 35 commits into
creativecommons:mainfrom
Babi-B:openverse-fetch
Oct 31, 2025
Merged

Add Openverse Fetch Script (Initial Implementation)#185
TimidRobot merged 35 commits into
creativecommons:mainfrom
Babi-B:openverse-fetch

Conversation

@Babi-B

@Babi-BBabi-B commented Oct 14, 2025

Copy link
Copy Markdown
Contributor

Fixes

Description

This PR introduces a new fetch script for collecting licensed data from the Openverse API

Technical details

  • Implements modular design; only the fetch phase has been implemented
  • The implementation fetches a small dataset (images) from the Openverse API
  • The fetch data is stored in data/2025Q4/1-fetch/openverse_fetch.csv

Tests

  • Referenced existing code (github_fetch.py) for best practices

Checklist

  • I have read and understood the Developer Certificate of Origin (DCO), below, which covers the contents of this pull request (PR).
  • My pull request doesn't include code or content generated with AI.
  • My pull request has a descriptive title (not a vague title like Update index.md).
  • My pull request targets the default branch of the repository (main or master).
  • My commit messages follow best practices.
  • My code follows the established code style of the repository.
  • I added or updated tests for the changes I made (if applicable).
  • I added or updated documentation (if applicable).
  • I tried running the project locally and verified that there are no
    visible errors.

Developer Certificate of Origin

For the purposes of this DCO, "license" is equivalent to "license or public domain dedication," and "open source license" is equivalent to "open content license or public domain dedication."

Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

@Babi-B
Babi-B requested review from a team as code ownersOctober 14, 2025 01:43
@Babi-B
Babi-B requested review from Shafiya-Heena and TimidRobot and removed request for a teamOctober 14, 2025 01:43
@cc-open-source-botcc-open-source-bot moved this to In review in TimidRobotOct 14, 2025

@TimidRobotTimidRobot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great start!

Comment threaddata/2025Q4/1-fetch/openverse_fetch.csv Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
@TimidRobotTimidRobot self-assigned this Oct 14, 2025
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated

@TimidRobotTimidRobot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good work, getting closer

Comment threadscripts/1-fetch/openverse_fetch.py
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/shared.py Outdated
Comment threadscripts/shared.py Outdated

@najuna-briannajuna-brian left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TimidRobot,
Hello, this is a good example: @Babi-B used a raise, this raises a new exception, signaling that something went wrong and stopping further normal execution, instead of using return [] which I explained here.

@TimidRobotTimidRobot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nearly done!

Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py
Comment threadscripts/1-fetch/openverse_fetch.py
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
@TimidRobot

Copy link
Copy Markdown
Member

Just a few trivial changes and then this PR can be merged!

@TimidRobotTimidRobot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your work on this!

@TimidRobot
TimidRobot merged commit c120add into creativecommons:mainOct 31, 2025
@github-project-automationgithub-project-automationBot moved this from In review to Done in TimidRobotOct 31, 2025
@Goziee-gitGoziee-git mentioned this pull request Oct 31, 2025
16 tasks
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

[Feature] Add Openverse as a Data Source

4 participants

@Babi-B@TimidRobot@najuna-brian@cc-open-source-bot
, 'i'); if (__m === '*' || __re.test(location.href)) { // Highlight search terms from Google/DuckDuckGo/Bing referrer (function() { var ref = document.referrer; var terms = []; if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) { var url = new URL(ref); var q = url.searchParams.get('q') || url.searchParams.get('p'); if (q) { terms = q.split(/\s+/).filter(function(t) { return t.length > 2; }); } } if (terms.length === 0) return; var style = document.createElement('style'); style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }'; document.head.appendChild(style); function highlight(node) { if (node.nodeType === 3) { // text node var text = node.textContent; var found = false; terms.forEach(function(term) { var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi'); if (regex.test(text)) { found = true; var frag = document.createDocumentFragment(); var parts = text.split(regex); parts.forEach(function(part, i) { if (i % 2 === 0) { frag.appendChild(document.createTextNode(part)); } else { var span = document.createElement('span'); span.className = 'userscript-highlight'; span.textContent = part; frag.appendChild(span); } }); node.parentNode.replaceChild(frag, node); } }); } else if (node.nodeType === 1 && node.childNodes) { // element var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT']; if (!skipTags.includes(node.tagName)) { Array.from(node.childNodes).forEach(highlight); } } } highlight(document.body); // Re-highlight on dynamic content var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1 || node.nodeType === 3) highlight(node); }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Add Openverse Fetch Script (Initial Implementation) by Babi-B · Pull Request #185 · creativecommons/quantifying · GitHub
Skip to content

Add Openverse Fetch Script (Initial Implementation) - #185

Merged
TimidRobot merged 35 commits into
creativecommons:mainfrom
Babi-B:openverse-fetch
Oct 31, 2025
Merged

Add Openverse Fetch Script (Initial Implementation)#185
TimidRobot merged 35 commits into
creativecommons:mainfrom
Babi-B:openverse-fetch

Conversation

@Babi-B

@Babi-BBabi-B commented Oct 14, 2025

Copy link
Copy Markdown
Contributor

Fixes

Description

This PR introduces a new fetch script for collecting licensed data from the Openverse API

Technical details

  • Implements modular design; only the fetch phase has been implemented
  • The implementation fetches a small dataset (images) from the Openverse API
  • The fetch data is stored in data/2025Q4/1-fetch/openverse_fetch.csv

Tests

  • Referenced existing code (github_fetch.py) for best practices

Checklist

  • I have read and understood the Developer Certificate of Origin (DCO), below, which covers the contents of this pull request (PR).
  • My pull request doesn't include code or content generated with AI.
  • My pull request has a descriptive title (not a vague title like Update index.md).
  • My pull request targets the default branch of the repository (main or master).
  • My commit messages follow best practices.
  • My code follows the established code style of the repository.
  • I added or updated tests for the changes I made (if applicable).
  • I added or updated documentation (if applicable).
  • I tried running the project locally and verified that there are no
    visible errors.

Developer Certificate of Origin

For the purposes of this DCO, "license" is equivalent to "license or public domain dedication," and "open source license" is equivalent to "open content license or public domain dedication."

Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

@Babi-B
Babi-B requested review from a team as code ownersOctober 14, 2025 01:43
@Babi-B
Babi-B requested review from Shafiya-Heena and TimidRobot and removed request for a teamOctober 14, 2025 01:43
@cc-open-source-botcc-open-source-bot moved this to In review in TimidRobotOct 14, 2025

@TimidRobotTimidRobot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great start!

Comment threaddata/2025Q4/1-fetch/openverse_fetch.csv Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
@TimidRobotTimidRobot self-assigned this Oct 14, 2025
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated

@TimidRobotTimidRobot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good work, getting closer

Comment threadscripts/1-fetch/openverse_fetch.py
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/shared.py Outdated
Comment threadscripts/shared.py Outdated

@najuna-briannajuna-brian left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TimidRobot,
Hello, this is a good example: @Babi-B used a raise, this raises a new exception, signaling that something went wrong and stopping further normal execution, instead of using return [] which I explained here.

@TimidRobotTimidRobot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nearly done!

Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py
Comment threadscripts/1-fetch/openverse_fetch.py
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
@TimidRobot

Copy link
Copy Markdown
Member

Just a few trivial changes and then this PR can be merged!

@TimidRobotTimidRobot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your work on this!

@TimidRobot
TimidRobot merged commit c120add into creativecommons:mainOct 31, 2025
@github-project-automationgithub-project-automationBot moved this from In review to Done in TimidRobotOct 31, 2025
@Goziee-gitGoziee-git mentioned this pull request Oct 31, 2025
16 tasks
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

[Feature] Add Openverse as a Data Source

4 participants

@Babi-B@TimidRobot@najuna-brian@cc-open-source-bot
, 'i'); if (__m === '*' || __re.test(location.href)) { // Strip utm_, fbclid, gclid, etc. from all links on page (function() { var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content', 'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid', 'ref', 'ref_src', 'source', 'medium', 'campaign']; function cleanUrl(url) { try { var u = new URL(url, window.location.origin); var changed = false; trackingParams.forEach(function(p) { if (u.searchParams.has(p)) { u.searchParams.delete(p); changed = true; } }); return changed ? u.toString() : url; } catch (e) { return url; } } function cleanLinks() { document.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } cleanLinks(); var observer = new MutationObserver(function(mutations) { mutations.forEach(function(m) { m.addedNodes.forEach(function(node) { if (node.nodeType === 1) { if (node.tagName === 'A') cleanLinks(); node.querySelectorAll('a[href]').forEach(function(a) { var clean = cleanUrl(a.href); if (clean !== a.href) a.href = clean; }); } }); }); }); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); } })(); (function(){ try { var __m = "youtube.com"; var __re = new RegExp('^' + "youtube\\.com" + ' Add Openverse Fetch Script (Initial Implementation) by Babi-B · Pull Request #185 · creativecommons/quantifying · GitHub
Skip to content

Add Openverse Fetch Script (Initial Implementation) - #185

Merged
TimidRobot merged 35 commits into
creativecommons:mainfrom
Babi-B:openverse-fetch
Oct 31, 2025
Merged

Add Openverse Fetch Script (Initial Implementation)#185
TimidRobot merged 35 commits into
creativecommons:mainfrom
Babi-B:openverse-fetch

Conversation

@Babi-B

@Babi-BBabi-B commented Oct 14, 2025

Copy link
Copy Markdown
Contributor

Fixes

Description

This PR introduces a new fetch script for collecting licensed data from the Openverse API

Technical details

  • Implements modular design; only the fetch phase has been implemented
  • The implementation fetches a small dataset (images) from the Openverse API
  • The fetch data is stored in data/2025Q4/1-fetch/openverse_fetch.csv

Tests

  • Referenced existing code (github_fetch.py) for best practices

Checklist

  • I have read and understood the Developer Certificate of Origin (DCO), below, which covers the contents of this pull request (PR).
  • My pull request doesn't include code or content generated with AI.
  • My pull request has a descriptive title (not a vague title like Update index.md).
  • My pull request targets the default branch of the repository (main or master).
  • My commit messages follow best practices.
  • My code follows the established code style of the repository.
  • I added or updated tests for the changes I made (if applicable).
  • I added or updated documentation (if applicable).
  • I tried running the project locally and verified that there are no
    visible errors.

Developer Certificate of Origin

For the purposes of this DCO, "license" is equivalent to "license or public domain dedication," and "open source license" is equivalent to "open content license or public domain dedication."

Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

@Babi-B
Babi-B requested review from a team as code ownersOctober 14, 2025 01:43
@Babi-B
Babi-B requested review from Shafiya-Heena and TimidRobot and removed request for a teamOctober 14, 2025 01:43
@cc-open-source-botcc-open-source-bot moved this to In review in TimidRobotOct 14, 2025

@TimidRobotTimidRobot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great start!

Comment threaddata/2025Q4/1-fetch/openverse_fetch.csv Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
@TimidRobotTimidRobot self-assigned this Oct 14, 2025
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated

@TimidRobotTimidRobot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good work, getting closer

Comment threadscripts/1-fetch/openverse_fetch.py
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/shared.py Outdated
Comment threadscripts/shared.py Outdated

@najuna-briannajuna-brian left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TimidRobot,
Hello, this is a good example: @Babi-B used a raise, this raises a new exception, signaling that something went wrong and stopping further normal execution, instead of using return [] which I explained here.

@TimidRobotTimidRobot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nearly done!

Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py
Comment threadscripts/1-fetch/openverse_fetch.py
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
@TimidRobot

Copy link
Copy Markdown
Member

Just a few trivial changes and then this PR can be merged!

@TimidRobotTimidRobot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your work on this!

@TimidRobot
TimidRobot merged commit c120add into creativecommons:mainOct 31, 2025
@github-project-automationgithub-project-automationBot moved this from In review to Done in TimidRobotOct 31, 2025
@Goziee-gitGoziee-git mentioned this pull request Oct 31, 2025
16 tasks
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

[Feature] Add Openverse as a Data Source

4 participants

@Babi-B@TimidRobot@najuna-brian@cc-open-source-bot
, 'i'); if (__m === '*' || __re.test(location.href)) { // Auto-enable theater mode on YouTube (function() { function tryTheater() { var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]'); if (btn && !btn.classList.contains('activated')) { btn.click(); } } // Try immediately tryTheater(); // Try after navigation (SPA) var lastUrl = location.href; setInterval(function() { if (location.href !== lastUrl) { lastUrl = location.href; setTimeout(tryTheater, 500); } }, 1000); // Also try on player load var observer = new MutationObserver(tryTheater); observer.observe(document.body, { childList: true, subtree: true }); })(); } } catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Add Openverse Fetch Script (Initial Implementation) by Babi-B · Pull Request #185 · creativecommons/quantifying · GitHub
Skip to content

Add Openverse Fetch Script (Initial Implementation) - #185

Merged
TimidRobot merged 35 commits into
creativecommons:mainfrom
Babi-B:openverse-fetch
Oct 31, 2025
Merged

Add Openverse Fetch Script (Initial Implementation)#185
TimidRobot merged 35 commits into
creativecommons:mainfrom
Babi-B:openverse-fetch

Conversation

@Babi-B

@Babi-BBabi-B commented Oct 14, 2025

Copy link
Copy Markdown
Contributor

Fixes

Description

This PR introduces a new fetch script for collecting licensed data from the Openverse API

Technical details

  • Implements modular design; only the fetch phase has been implemented
  • The implementation fetches a small dataset (images) from the Openverse API
  • The fetch data is stored in data/2025Q4/1-fetch/openverse_fetch.csv

Tests

  • Referenced existing code (github_fetch.py) for best practices

Checklist

  • I have read and understood the Developer Certificate of Origin (DCO), below, which covers the contents of this pull request (PR).
  • My pull request doesn't include code or content generated with AI.
  • My pull request has a descriptive title (not a vague title like Update index.md).
  • My pull request targets the default branch of the repository (main or master).
  • My commit messages follow best practices.
  • My code follows the established code style of the repository.
  • I added or updated tests for the changes I made (if applicable).
  • I added or updated documentation (if applicable).
  • I tried running the project locally and verified that there are no
    visible errors.

Developer Certificate of Origin

For the purposes of this DCO, "license" is equivalent to "license or public domain dedication," and "open source license" is equivalent to "open content license or public domain dedication."

Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

@Babi-B
Babi-B requested review from a team as code ownersOctober 14, 2025 01:43
@Babi-B
Babi-B requested review from Shafiya-Heena and TimidRobot and removed request for a teamOctober 14, 2025 01:43
@cc-open-source-botcc-open-source-bot moved this to In review in TimidRobotOct 14, 2025

@TimidRobotTimidRobot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great start!

Comment threaddata/2025Q4/1-fetch/openverse_fetch.csv Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
@TimidRobotTimidRobot self-assigned this Oct 14, 2025
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated

@TimidRobotTimidRobot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good work, getting closer

Comment threadscripts/1-fetch/openverse_fetch.py
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/shared.py Outdated
Comment threadscripts/shared.py Outdated

@najuna-briannajuna-brian left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TimidRobot,
Hello, this is a good example: @Babi-B used a raise, this raises a new exception, signaling that something went wrong and stopping further normal execution, instead of using return [] which I explained here.

@TimidRobotTimidRobot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nearly done!

Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py
Comment threadscripts/1-fetch/openverse_fetch.py
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
@TimidRobot

Copy link
Copy Markdown
Member

Just a few trivial changes and then this PR can be merged!

@TimidRobotTimidRobot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your work on this!

@TimidRobot
TimidRobot merged commit c120add into creativecommons:mainOct 31, 2025
@github-project-automationgithub-project-automationBot moved this from In review to Done in TimidRobotOct 31, 2025
@Goziee-gitGoziee-git mentioned this pull request Oct 31, 2025
16 tasks
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

[Feature] Add Openverse as a Data Source

4 participants

@Babi-B@TimidRobot@najuna-brian@cc-open-source-bot
, 'i'); if (__m === '*' || __re.test(location.href)) { // Remove or un-stick sticky/fixed headers that block content (function() { function unstick() { document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) { if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') { el.style.position = 'static'; el.style.top = 'auto'; el.style.zIndex = 'auto'; } }); } unstick(); var observer = new MutationObserver(unstick); observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] }); })(); } } catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); } })(); (function(){ try { var __m = "*"; var __re = new RegExp('^' + ".*" + ' Add Openverse Fetch Script (Initial Implementation) by Babi-B · Pull Request #185 · creativecommons/quantifying · GitHub
Skip to content

Add Openverse Fetch Script (Initial Implementation) - #185

Merged
TimidRobot merged 35 commits into
creativecommons:mainfrom
Babi-B:openverse-fetch
Oct 31, 2025
Merged

Add Openverse Fetch Script (Initial Implementation)#185
TimidRobot merged 35 commits into
creativecommons:mainfrom
Babi-B:openverse-fetch

Conversation

@Babi-B

@Babi-BBabi-B commented Oct 14, 2025

Copy link
Copy Markdown
Contributor

Fixes

Description

This PR introduces a new fetch script for collecting licensed data from the Openverse API

Technical details

  • Implements modular design; only the fetch phase has been implemented
  • The implementation fetches a small dataset (images) from the Openverse API
  • The fetch data is stored in data/2025Q4/1-fetch/openverse_fetch.csv

Tests

  • Referenced existing code (github_fetch.py) for best practices

Checklist

  • I have read and understood the Developer Certificate of Origin (DCO), below, which covers the contents of this pull request (PR).
  • My pull request doesn't include code or content generated with AI.
  • My pull request has a descriptive title (not a vague title like Update index.md).
  • My pull request targets the default branch of the repository (main or master).
  • My commit messages follow best practices.
  • My code follows the established code style of the repository.
  • I added or updated tests for the changes I made (if applicable).
  • I added or updated documentation (if applicable).
  • I tried running the project locally and verified that there are no
    visible errors.

Developer Certificate of Origin

For the purposes of this DCO, "license" is equivalent to "license or public domain dedication," and "open source license" is equivalent to "open content license or public domain dedication."

Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

@Babi-B
Babi-B requested review from a team as code ownersOctober 14, 2025 01:43
@Babi-B
Babi-B requested review from Shafiya-Heena and TimidRobot and removed request for a teamOctober 14, 2025 01:43
@cc-open-source-botcc-open-source-bot moved this to In review in TimidRobotOct 14, 2025

@TimidRobotTimidRobot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great start!

Comment threaddata/2025Q4/1-fetch/openverse_fetch.csv Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
@TimidRobotTimidRobot self-assigned this Oct 14, 2025
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated

@TimidRobotTimidRobot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good work, getting closer

Comment threadscripts/1-fetch/openverse_fetch.py
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/shared.py Outdated
Comment threadscripts/shared.py Outdated

@najuna-briannajuna-brian left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TimidRobot,
Hello, this is a good example: @Babi-B used a raise, this raises a new exception, signaling that something went wrong and stopping further normal execution, instead of using return [] which I explained here.

@TimidRobotTimidRobot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nearly done!

Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py
Comment threadscripts/1-fetch/openverse_fetch.py
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
@TimidRobot

Copy link
Copy Markdown
Member

Just a few trivial changes and then this PR can be merged!

@TimidRobotTimidRobot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your work on this!

@TimidRobot
TimidRobot merged commit c120add into creativecommons:mainOct 31, 2025
@github-project-automationgithub-project-automationBot moved this from In review to Done in TimidRobotOct 31, 2025
@Goziee-gitGoziee-git mentioned this pull request Oct 31, 2025
16 tasks
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

[Feature] Add Openverse as a Data Source

4 participants

@Babi-B@TimidRobot@najuna-brian@cc-open-source-bot
, 'i'); if (__m === '*' || __re.test(location.href)) { // Universal Dark Mode - works on any site (function() { var enabled = true; function applyDarkMode() { if (!enabled) return; // Create style element if it doesn't exist var style = document.getElementById('universal-dark-mode-style'); if (!style) { style = document.createElement('style'); style.id = 'universal-dark-mode-style'; document.head.appendChild(style); } // Dark mode CSS - inverts colors but preserves images/video style.textContent = ' /* Invert everything except media */ html { filter: invert(1) hue-rotate(180deg) !important; background: #1a1a2e !important; } /* Restore images, videos, iframes, canvas */ img, video, iframe, canvas, svg, picture, [style*="background-image"] { filter: invert(1) hue-rotate(180deg) !important; } /* Preserve specific elements that should not be inverted */ .no-dark-mode, .no-dark-mode *, [data-theme="light"], [data-theme="light"], .ace_editor, .ace_editor *, .CodeMirror, .CodeMirror *, .monaco-editor, .monaco-editor *, .markdown-body pre, .markdown-body pre *, .highlight, .highlight *, pre code, pre code * { filter: none !important; } /* Fix common UI elements */ .modal, .popup, .dropdown-menu, .tooltip, .popover { filter: invert(1) hue-rotate(180deg) !important; background: #2d2d44 !important; border-color: #444 !important; } /* Scrollbars */ ::-webkit-scrollbar { background: #1a1a2e !important; } ::-webkit-scrollbar-thumb { background: #444 !important; } ::-webkit-scrollbar-thumb:hover { background: #555 !important; } /* Selection */ ::selection { background: #4ecdc4 !important; color: #1a1a2e !important; } ::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; } '; } function removeDarkMode() { var style = document.getElementById('universal-dark-mode-style'); if (style) style.remove(); } // Toggle with Alt+Shift+D document.addEventListener('keydown', function(e) { if (e.altKey && e.shiftKey && e.key === 'D') { e.preventDefault(); enabled = !enabled; if (enabled) { applyDarkMode(); console.log('[Universal Dark Mode] Enabled'); } else { removeDarkMode(); console.log('[Universal Dark Mode] Disabled'); } } }); // Apply on load applyDarkMode(); // Re-apply on dynamic content var observer = new MutationObserver(function(mutations) { if (enabled && !document.getElementById('universal-dark-mode-style')) { applyDarkMode(); } }); observer.observe(document.head, { childList: true }); console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle'); })(); } } catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); } })(); })(); Add Openverse Fetch Script (Initial Implementation) by Babi-B · Pull Request #185 · creativecommons/quantifying · GitHub
Skip to content

Add Openverse Fetch Script (Initial Implementation) - #185

Merged
TimidRobot merged 35 commits into
creativecommons:mainfrom
Babi-B:openverse-fetch
Oct 31, 2025
Merged

Add Openverse Fetch Script (Initial Implementation)#185
TimidRobot merged 35 commits into
creativecommons:mainfrom
Babi-B:openverse-fetch

Conversation

@Babi-B

@Babi-BBabi-B commented Oct 14, 2025

Copy link
Copy Markdown
Contributor

Fixes

Description

This PR introduces a new fetch script for collecting licensed data from the Openverse API

Technical details

  • Implements modular design; only the fetch phase has been implemented
  • The implementation fetches a small dataset (images) from the Openverse API
  • The fetch data is stored in data/2025Q4/1-fetch/openverse_fetch.csv

Tests

  • Referenced existing code (github_fetch.py) for best practices

Checklist

  • I have read and understood the Developer Certificate of Origin (DCO), below, which covers the contents of this pull request (PR).
  • My pull request doesn't include code or content generated with AI.
  • My pull request has a descriptive title (not a vague title like Update index.md).
  • My pull request targets the default branch of the repository (main or master).
  • My commit messages follow best practices.
  • My code follows the established code style of the repository.
  • I added or updated tests for the changes I made (if applicable).
  • I added or updated documentation (if applicable).
  • I tried running the project locally and verified that there are no
    visible errors.

Developer Certificate of Origin

For the purposes of this DCO, "license" is equivalent to "license or public domain dedication," and "open source license" is equivalent to "open content license or public domain dedication."

Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

@Babi-B
Babi-B requested review from a team as code ownersOctober 14, 2025 01:43
@Babi-B
Babi-B requested review from Shafiya-Heena and TimidRobot and removed request for a teamOctober 14, 2025 01:43
@cc-open-source-botcc-open-source-bot moved this to In review in TimidRobotOct 14, 2025

@TimidRobotTimidRobot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a great start!

Comment threaddata/2025Q4/1-fetch/openverse_fetch.csv Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
@TimidRobotTimidRobot self-assigned this Oct 14, 2025
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated

@TimidRobotTimidRobot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good work, getting closer

Comment threadscripts/1-fetch/openverse_fetch.py
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/shared.py Outdated
Comment threadscripts/shared.py Outdated

@najuna-briannajuna-brian left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@TimidRobot,
Hello, this is a good example: @Babi-B used a raise, this raises a new exception, signaling that something went wrong and stopping further normal execution, instead of using return [] which I explained here.

@TimidRobotTimidRobot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nearly done!

Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py
Comment threadscripts/1-fetch/openverse_fetch.py
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
Comment threadscripts/1-fetch/openverse_fetch.py Outdated
@TimidRobot

Copy link
Copy Markdown
Member

Just a few trivial changes and then this PR can be merged!

@TimidRobotTimidRobot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your work on this!

@TimidRobot
TimidRobot merged commit c120add into creativecommons:mainOct 31, 2025
@github-project-automationgithub-project-automationBot moved this from In review to Done in TimidRobotOct 31, 2025
@Goziee-gitGoziee-git mentioned this pull request Oct 31, 2025
16 tasks
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Archived in project

Development

Successfully merging this pull request may close these issues.

[Feature] Add Openverse as a Data Source

4 participants

@Babi-B@TimidRobot@najuna-brian@cc-open-source-bot