Add Europeana integration - #200

Merged
TimidRobot merged 33 commits into
creativecommons:mainfrom
Joyakis:europeana-feature
Nov 1, 2025
Merged

Add Europeana integration#200
TimidRobot merged 33 commits into
creativecommons:mainfrom
Joyakis:europeana-feature

Conversation

@Joyakis

@JoyakisJoyakis commented Oct 15, 2025•
edited by TimidRobot
Loading

Copy link
Copy Markdown
Contributor

Fixes

Description

Add Europeana API integration for metrics collection

This PR adds a new script europeana_fetch.py that fetches and aggregates data from the Europeana Search API.
The script collects high-level statistics about cultural heritage content available through Europeana, focusing on data provider distribution and content types rather than fragile license parsing.

Technical details

Script Location: scripts/1-fetch/europeana_fetch.py

  • Data Output: data/2025Q4/1-fetch/europeana_1_count.csv
  • Key Features:
    • Fetches data from Europeana Search API using multiple content queries
    • Aggregates by DATA_PROVIDER and content metadata
    • Includes proper error handling and API rate limiting
    • Integrates with existing project structure and git workflows
  • Environment: Updated env.example with EUROPEANA_API_KEY placeholder

Tests

  1. Set up Europeana API key in environment variables
  2. Run the script: python scripts/1-fetch/europeana_fetch.py --enable-save
  3. Verify CSV output is generated in data/2025Q4/1-fetch/europeana_1_count.csv
  4. Check that data contains aggregated counts by DATA_PROVIDER

Checklist

  • I have read and understood the Developer Certificate of Origin (DCO), below, which covers the contents of this pull request (PR).
  • My pull request doesn't include code or content generated with AI.
  • My pull request has a descriptive title (not a vague title like Update index.md).
  • My pull request targets the default branch of the repository (main or master).
  • My commit messages follow best practices.
  • My code follows the established code style of the repository.
  • I added or updated tests for the changes I made (if applicable).
  • I added or updated documentation (if applicable).
  • I tried running the project locally and verified that there are no
    visible errors.

Developer Certificate of Origin

For the purposes of this DCO, "license" is equivalent to "license or public domain dedication," and "open source license" is equivalent to "open content license or public domain dedication."

Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

@Joyakis
Joyakis requested review from a team as code ownersOctober 15, 2025 19:48
@Joyakis
Joyakis requested review from TimidRobot and possumbilities and removed request for a teamOctober 15, 2025 19:48
@cc-open-source-botcc-open-source-bot moved this to In review in TimidRobotOct 15, 2025
Babi-B
Babi-B reviewed Oct 15, 2025
View reviewed changes
Comment threadenv.example
TimidRobot
TimidRobot requested changes Oct 17, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py Outdated
Comment threadscripts/1-fetch/europeana_fetch.py Outdated
Comment threadscripts/1-fetch/europeana_fetch.py Outdated
Comment threadscripts/1-fetch/europeana_fetch.py Outdated
Comment threadenv.example
Comment threadPipfile Outdated
Comment threadPipfile.lock
@TimidRobotTimidRobot self-assigned this Oct 18, 2025
@TimidRobotTimidRobot changed the title Added Europeana integrationAdd Europeana integrationOct 20, 2025
TimidRobot
TimidRobot reviewed Oct 20, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py Outdated
@TimidRobot

TimidRobot commented Oct 21, 2025

Copy link
Copy Markdown
Member

@Joyakis I resolved the conflicts with the main branch (due to changes from another pull request being merged). Please remember to fetch the changes to your computer.

@Joyakis

Joyakis commented Oct 21, 2025

Copy link
Copy Markdown
ContributorAuthor

@Joyakis I resolved the conflicts with the main branch (due to changes from another pull request being merged). Please remember to fetch the changes to your computer.

@TimidRobot Done!

TimidRobot
TimidRobot requested changes Oct 21, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Comment threadscripts/1-fetch/europeana_fetch.py Outdated
TimidRobot
TimidRobot requested changes Oct 21, 2025
View reviewed changes

@TimidRobotTimidRobot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please resolve all comments before requesting a new review

Comment threadscripts/1-fetch/europeana_fetch.py
Comment threadscripts/1-fetch/europeana_fetch.py Outdated
Comment threadscripts/1-fetch/europeana_fetch.py Outdated
Comment threadscripts/1-fetch/europeana_fetch.py Outdated
Comment threadscripts/1-fetch/europeana_fetch.py Outdated
Comment threadscripts/1-fetch/europeana_fetch.py Outdated
Comment threadscripts/1-fetch/europeana_fetch.py
Comment threadscripts/1-fetch/europeana_fetch.py Outdated
@TimidRobot

TimidRobot commented Oct 24, 2025

Copy link
Copy Markdown
Member

Please focus on resolving conversations before you add new features.

Please remove custom logging (print statements) and follow logging conventions in other scripts.

@Joyakis

Joyakis commented Oct 24, 2025•
edited
Loading

Copy link
Copy Markdown
ContributorAuthor

Please focus on resolving conversations before you add new features.

Please remove custom logging (print statements) and follow logging conventions in other scripts.

Hello @TimidRobot The script now gets all providers and rights.

Thank you for pointing out where the problem was.

I have also removed the unnecessary printing statements

@Joyakis

Joyakis commented Oct 28, 2025

Copy link
Copy Markdown
ContributorAuthor

2. you review the pull request Files changed tab yourself to see if there are any unintended files or other obvious issues

Hello....I reviewed the Files changed tab in my pull request as you advised.
Only the intended updates to europeana_fetch.py are listed.
The other files in the folder (env.example and sources.md) came from merging the latest main branch into my feature branch and they do not show up in the diff .

@Joyakis

Joyakis commented Oct 28, 2025

Copy link
Copy Markdown
ContributorAuthor

Greetings @TimidRobot
This is a screenshot of the data without themes

image

This is the one with themes

image

TimidRobot
TimidRobot reviewed Oct 29, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
TimidRobot
TimidRobot reviewed Oct 30, 2025
View reviewed changes
Comment threadsources.md
@Joyakis

Joyakis commented Oct 30, 2025

Copy link
Copy Markdown
ContributorAuthor

Hello @TimidRobot

I have updated both the env.example and sources file

TimidRobot
TimidRobot reviewed Oct 30, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
TimidRobot
TimidRobot reviewed Oct 30, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py Outdated
TimidRobot
TimidRobot reviewed Oct 30, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
TimidRobot
TimidRobot reviewed Oct 30, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Joyakisand others added 5 commits October 31, 2025 13:06
TimidRobot
TimidRobot approved these changes Nov 1, 2025
View reviewed changes

@TimidRobotTimidRobot left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work on this! Thank you!!

@TimidRobot
TimidRobot merged commit 46dcd3f into creativecommons:mainNov 1, 2025
@github-project-automationgithub-project-automationBot moved this from In review to Done in TimidRobotNov 1, 2025
@Joyakis

Joyakis commented Nov 1, 2025

Copy link
Copy Markdown
ContributorAuthor

Great work on this! Thank you!!

Happy to have done this!...Thank you as well

Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@TimidRobotTimidRobotTimidRobot approved these changes
@possumbilitiespossumbilitiesAwaiting requested review from possumbilitiespossumbilities is a code owner automatically assigned from creativecommons/technology
+1 more reviewer
@Babi-BBabi-BBabi-B left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

@TimidRobotTimidRobot

Labels

None yet

Projects

TimidRobot
Archived in project

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add Europeana as a data source

4 participants

@Joyakis@TimidRobot@Babi-B@cc-open-source-bot
, 'i'); if (__m === '*' || __re.test(location.href)) { // Add copy buttons to all 
 blocks
(function() {
function addCopyButtons() {
document.querySelectorAll('pre code').forEach(function(codeBlock) {
if (codeBlock.parentElement.hasAttribute('data-copy-added')) return;
codeBlock.parentElement.setAttribute('data-copy-added', 'true');
var btn = document.createElement('button');
btn.textContent = 'Copy';
btn.style.cssText = 'position:absolute;top:4px;right:4px;padding:2px 8px;font-size:11px;background:#4ecdc4;border:none;border-radius:4px;color:#1a1a2e;cursor:pointer;opacity:0.7;transition:opacity 0.2s;';
btn.onmouseover = function() { this.style.opacity = '1'; };
btn.onmouseout = function() { this.style.opacity = '0.7'; };
btn.onclick = function() {
navigator.clipboard.writeText(codeBlock.textContent).then(function() {
btn.textContent = 'Copied!';
setTimeout(function() { btn.textContent = 'Copy'; }, 1500);
});
};
codeBlock.parentElement.style.position = 'relative';
codeBlock.parentElement.appendChild(btn);
});
}
addCopyButtons();
// Re-run on dynamic content
var observer = new MutationObserver(addCopyButtons);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Add Copy Buttons to Code Blocks]', __e); }
})();
(function(){
try {
var __m = "github.com";
var __re = new RegExp('^' + "github\\.com" + '

Skip to content
Add Europeana integration - #200
Merged
TimidRobot merged 33 commits into
creativecommons:mainfrom 
Joyakis:europeana-feature
Nov 1, 2025
Merged
Add Europeana integration#200
TimidRobot merged 33 commits into
creativecommons:mainfrom 
Joyakis:europeana-feature
Conversation
@Joyakis
@JoyakisJoyakis
commented
Oct 15, 2025•

edited by TimidRobot

Loading

Copy link


Copy Markdown

Contributor
Fixes
Description
Add Europeana API integration for metrics collection
This PR adds a new script europeana_fetch.py that fetches and aggregates data from the Europeana Search API.

The script collects high-level statistics about cultural heritage content available through Europeana, focusing on data provider distribution and content types rather than fragile license parsing.
Technical details
Script Location: scripts/1-fetch/europeana_fetch.py
  • Data Output: data/2025Q4/1-fetch/europeana_1_count.csv
  • Key Features:

    • Fetches data from Europeana Search API using multiple content queries
    • Aggregates by DATA_PROVIDER and content metadata
    • Includes proper error handling and API rate limiting
    • Integrates with existing project structure and git workflows
  • Environment: Updated env.example with EUROPEANA_API_KEY placeholder
Tests
  1. Set up Europeana API key in environment variables
  2. Run the script: python scripts/1-fetch/europeana_fetch.py --enable-save
  3. Verify CSV output is generated in data/2025Q4/1-fetch/europeana_1_count.csv
  4. Check that data contains aggregated counts by DATA_PROVIDER
Checklist
  •  I have read and understood the Developer Certificate of Origin (DCO), below, which covers the contents of this pull request (PR).
  •  My pull request doesn't include code or content generated with AI.
  •  My pull request has a descriptive title (not a vague title like Update index.md).
  •  My pull request targets the default branch of the repository (main or master).
  •  My commit messages follow best practices.
  •  My code follows the established code style of the repository.
  •  I added or updated tests for the changes I made (if applicable).
  •  I added or updated documentation (if applicable).
  •  I tried running the project locally and verified that there are no
    
visible errors.
Developer Certificate of Origin
For the purposes of this DCO, "license" is equivalent to "license or public domain dedication," and "open source license" is equivalent to "open content license or public domain dedication."
Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

@Joyakis
Joyakis
requested review from
a team
as code ownersOctober 15, 2025 19:48
@Joyakis
Joyakis
requested review from
TimidRobot and possumbilities
and removed request for
a teamOctober 15, 2025 19:48
@cc-open-source-botcc-open-source-bot
moved this to In review
in TimidRobotOct 15, 2025
Babi-B
Babi-B
reviewed
Oct 15, 2025
View reviewed changes
Comment threadenv.example
TimidRobot
TimidRobot
requested changes
Oct 17, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadenv.example
Comment threadPipfile
Outdated

Comment threadPipfile.lock
@TimidRobotTimidRobot
self-assigned this
Oct 18, 2025
@TimidRobotTimidRobot
changed the title
Added Europeana integrationAdd Europeana integrationOct 20, 2025
TimidRobot
TimidRobot
reviewed
Oct 20, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

@TimidRobot
TimidRobot
commented
Oct 21, 2025

Copy link


Copy Markdown

Member
@Joyakis I resolved the conflicts with the main branch (due to changes from another pull request being merged). Please remember to fetch the changes to your computer.
@Joyakis
Joyakis
commented
Oct 21, 2025

Copy link


Copy Markdown

ContributorAuthor
@Joyakis I resolved the conflicts with the main branch (due to changes from another pull request being merged). Please remember to fetch the changes to your computer.
@TimidRobot Done!
TimidRobot
TimidRobot
requested changes
Oct 21, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

TimidRobot
TimidRobot
requested changes
Oct 21, 2025
View reviewed changes
@TimidRobotTimidRobot
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please resolve all comments before requesting a new review
Comment threadscripts/1-fetch/europeana_fetch.py
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

@TimidRobot
TimidRobot
commented
Oct 24, 2025

Copy link


Copy Markdown

Member
Please focus on resolving conversations before you add new features.
Please remove custom logging (print statements) and follow logging conventions in other scripts.
@Joyakis
Joyakis
commented
Oct 24, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Please focus on resolving conversations before you add new features.
Please remove custom logging (print statements) and follow logging conventions in other scripts.
Hello @TimidRobot The script now gets all providers and rights.
Thank you for pointing out where the problem was.
I have also removed the unnecessary printing statements
@Joyakis
Joyakis
commented
Oct 28, 2025

Copy link


Copy Markdown

ContributorAuthor
2. you review the pull request Files changed tab yourself to see if there are any unintended files or other obvious issues
Hello....I reviewed the Files changed tab in my pull request as you advised.

Only the intended updates to europeana_fetch.py are listed.

The other files in the folder (env.example and sources.md) came from merging the latest main branch into my feature branch and they do not show up in the diff .
@Joyakis
Joyakis
commented
Oct 28, 2025

Copy link


Copy Markdown

ContributorAuthor
Greetings @TimidRobot

This is a screenshot of the data without themes
image
This is the one with themes
image
TimidRobot
TimidRobot
reviewed
Oct 29, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
TimidRobot
TimidRobot
reviewed
Oct 30, 2025
View reviewed changes
Comment threadsources.md
@Joyakis
Joyakis
commented
Oct 30, 2025

Copy link


Copy Markdown

ContributorAuthor
Hello @TimidRobot
I have updated both the env.example and sources file
TimidRobot
TimidRobot
reviewed
Oct 30, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
TimidRobot
TimidRobot
reviewed
Oct 30, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

TimidRobot
TimidRobot
reviewed
Oct 30, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
TimidRobot
TimidRobot
reviewed
Oct 30, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Joyakisand others
added 5 commits
October 31, 2025 13:06
TimidRobot
TimidRobot
approved these changes
Nov 1, 2025
View reviewed changes
@TimidRobotTimidRobot
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work on this! Thank you!!
@TimidRobot
TimidRobot
merged commit 46dcd3f
into
creativecommons:mainNov 1, 2025
@github-project-automationgithub-project-automationBot
moved this from In review
to Done
in TimidRobotNov 1, 2025
@Joyakis
Joyakis
commented
Nov 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Great work on this! Thank you!!
Happy to have done this!...Thank you as well
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@TimidRobotTimidRobotTimidRobot approved these changes
@possumbilitiespossumbilitiesAwaiting requested review from possumbilitiespossumbilities is a code owner automatically assigned from creativecommons/technology

+1 more reviewer

@Babi-BBabi-BBabi-B left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

@TimidRobotTimidRobot

Labels


None yet


Projects

TimidRobot

Archived in project


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Add Europeana as a data source


4 participants

@Joyakis@TimidRobot@Babi-B@cc-open-source-bot

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Force GitHub README to respect dark mode
(function() {
var style = document.createElement('style');
style.textContent = '
.markdown-body {
color-scheme: dark light;
}
.markdown-body pre { background: #161b22 !important; }
.markdown-body code { background: rgba(110, 118, 129, 0.4) !important; }
.markdown-body table th, .markdown-body table td { border-color: #30363d !important; }
.markdown-body img { background: #0d1117; }
.markdown-body blockquote { border-left-color: #8b949e; }
.markdown-body hr { border-color: #30363d; }
';
document.head.appendChild(style);
})();
}
} catch(__e) { console.warn('[Userscript:GitHub Dark Mode README Fix]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Add Europeana integration - #200
Merged
TimidRobot merged 33 commits into
creativecommons:mainfrom 
Joyakis:europeana-feature
Nov 1, 2025
Merged
Add Europeana integration#200
TimidRobot merged 33 commits into
creativecommons:mainfrom 
Joyakis:europeana-feature
Conversation
@Joyakis
@JoyakisJoyakis
commented
Oct 15, 2025•

edited by TimidRobot

Loading

Copy link


Copy Markdown

Contributor
Fixes
Description
Add Europeana API integration for metrics collection
This PR adds a new script europeana_fetch.py that fetches and aggregates data from the Europeana Search API.

The script collects high-level statistics about cultural heritage content available through Europeana, focusing on data provider distribution and content types rather than fragile license parsing.
Technical details
Script Location: scripts/1-fetch/europeana_fetch.py
  • Data Output: data/2025Q4/1-fetch/europeana_1_count.csv
  • Key Features:

    • Fetches data from Europeana Search API using multiple content queries
    • Aggregates by DATA_PROVIDER and content metadata
    • Includes proper error handling and API rate limiting
    • Integrates with existing project structure and git workflows
  • Environment: Updated env.example with EUROPEANA_API_KEY placeholder
Tests
  1. Set up Europeana API key in environment variables
  2. Run the script: python scripts/1-fetch/europeana_fetch.py --enable-save
  3. Verify CSV output is generated in data/2025Q4/1-fetch/europeana_1_count.csv
  4. Check that data contains aggregated counts by DATA_PROVIDER
Checklist
  •  I have read and understood the Developer Certificate of Origin (DCO), below, which covers the contents of this pull request (PR).
  •  My pull request doesn't include code or content generated with AI.
  •  My pull request has a descriptive title (not a vague title like Update index.md).
  •  My pull request targets the default branch of the repository (main or master).
  •  My commit messages follow best practices.
  •  My code follows the established code style of the repository.
  •  I added or updated tests for the changes I made (if applicable).
  •  I added or updated documentation (if applicable).
  •  I tried running the project locally and verified that there are no
    
visible errors.
Developer Certificate of Origin
For the purposes of this DCO, "license" is equivalent to "license or public domain dedication," and "open source license" is equivalent to "open content license or public domain dedication."
Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

@Joyakis
Joyakis
requested review from
a team
as code ownersOctober 15, 2025 19:48
@Joyakis
Joyakis
requested review from
TimidRobot and possumbilities
and removed request for
a teamOctober 15, 2025 19:48
@cc-open-source-botcc-open-source-bot
moved this to In review
in TimidRobotOct 15, 2025
Babi-B
Babi-B
reviewed
Oct 15, 2025
View reviewed changes
Comment threadenv.example
TimidRobot
TimidRobot
requested changes
Oct 17, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadenv.example
Comment threadPipfile
Outdated

Comment threadPipfile.lock
@TimidRobotTimidRobot
self-assigned this
Oct 18, 2025
@TimidRobotTimidRobot
changed the title
Added Europeana integrationAdd Europeana integrationOct 20, 2025
TimidRobot
TimidRobot
reviewed
Oct 20, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

@TimidRobot
TimidRobot
commented
Oct 21, 2025

Copy link


Copy Markdown

Member
@Joyakis I resolved the conflicts with the main branch (due to changes from another pull request being merged). Please remember to fetch the changes to your computer.
@Joyakis
Joyakis
commented
Oct 21, 2025

Copy link


Copy Markdown

ContributorAuthor
@Joyakis I resolved the conflicts with the main branch (due to changes from another pull request being merged). Please remember to fetch the changes to your computer.
@TimidRobot Done!
TimidRobot
TimidRobot
requested changes
Oct 21, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

TimidRobot
TimidRobot
requested changes
Oct 21, 2025
View reviewed changes
@TimidRobotTimidRobot
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please resolve all comments before requesting a new review
Comment threadscripts/1-fetch/europeana_fetch.py
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

@TimidRobot
TimidRobot
commented
Oct 24, 2025

Copy link


Copy Markdown

Member
Please focus on resolving conversations before you add new features.
Please remove custom logging (print statements) and follow logging conventions in other scripts.
@Joyakis
Joyakis
commented
Oct 24, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Please focus on resolving conversations before you add new features.
Please remove custom logging (print statements) and follow logging conventions in other scripts.
Hello @TimidRobot The script now gets all providers and rights.
Thank you for pointing out where the problem was.
I have also removed the unnecessary printing statements
@Joyakis
Joyakis
commented
Oct 28, 2025

Copy link


Copy Markdown

ContributorAuthor
2. you review the pull request Files changed tab yourself to see if there are any unintended files or other obvious issues
Hello....I reviewed the Files changed tab in my pull request as you advised.

Only the intended updates to europeana_fetch.py are listed.

The other files in the folder (env.example and sources.md) came from merging the latest main branch into my feature branch and they do not show up in the diff .
@Joyakis
Joyakis
commented
Oct 28, 2025

Copy link


Copy Markdown

ContributorAuthor
Greetings @TimidRobot

This is a screenshot of the data without themes
image
This is the one with themes
image
TimidRobot
TimidRobot
reviewed
Oct 29, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
TimidRobot
TimidRobot
reviewed
Oct 30, 2025
View reviewed changes
Comment threadsources.md
@Joyakis
Joyakis
commented
Oct 30, 2025

Copy link


Copy Markdown

ContributorAuthor
Hello @TimidRobot
I have updated both the env.example and sources file
TimidRobot
TimidRobot
reviewed
Oct 30, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
TimidRobot
TimidRobot
reviewed
Oct 30, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

TimidRobot
TimidRobot
reviewed
Oct 30, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
TimidRobot
TimidRobot
reviewed
Oct 30, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Joyakisand others
added 5 commits
October 31, 2025 13:06
TimidRobot
TimidRobot
approved these changes
Nov 1, 2025
View reviewed changes
@TimidRobotTimidRobot
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work on this! Thank you!!
@TimidRobot
TimidRobot
merged commit 46dcd3f
into
creativecommons:mainNov 1, 2025
@github-project-automationgithub-project-automationBot
moved this from In review
to Done
in TimidRobotNov 1, 2025
@Joyakis
Joyakis
commented
Nov 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Great work on this! Thank you!!
Happy to have done this!...Thank you as well
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@TimidRobotTimidRobotTimidRobot approved these changes
@possumbilitiespossumbilitiesAwaiting requested review from possumbilitiespossumbilities is a code owner automatically assigned from creativecommons/technology

+1 more reviewer

@Babi-BBabi-BBabi-B left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

@TimidRobotTimidRobot

Labels


None yet


Projects

TimidRobot

Archived in project


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Add Europeana as a data source


4 participants

@Joyakis@TimidRobot@Babi-B@cc-open-source-bot

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Highlight search terms from Google/DuckDuckGo/Bing referrer
(function() {
var ref = document.referrer;
var terms = [];
if (ref.includes('google.com') || ref.includes('duckduckgo.com') || ref.includes('bing.com')) {
var url = new URL(ref);
var q = url.searchParams.get('q') || url.searchParams.get('p');
if (q) {
terms = q.split(/\s+/).filter(function(t) { return t.length > 2; });
}
}
if (terms.length === 0) return;
var style = document.createElement('style');
style.textContent = '.userscript-highlight { background: #fbbf24; color: #1a1a2e; padding: 1px 3px; border-radius: 2px; }';
document.head.appendChild(style);
function highlight(node) {
if (node.nodeType === 3) { // text node
var text = node.textContent;
var found = false;
terms.forEach(function(term) {
var regex = new RegExp('(' + term.replace(/[.*+?^${}()|[\]\\]/g, '\\') + ')', 'gi');
if (regex.test(text)) {
found = true;
var frag = document.createDocumentFragment();
var parts = text.split(regex);
parts.forEach(function(part, i) {
if (i % 2 === 0) {
frag.appendChild(document.createTextNode(part));
} else {
var span = document.createElement('span');
span.className = 'userscript-highlight';
span.textContent = part;
frag.appendChild(span);
}
});
node.parentNode.replaceChild(frag, node);
}
});
} else if (node.nodeType === 1 && node.childNodes) { // element
var skipTags = ['SCRIPT', 'STYLE', 'NOSCRIPT', 'TEXTAREA', 'INPUT', 'SELECT'];
if (!skipTags.includes(node.tagName)) {
Array.from(node.childNodes).forEach(highlight);
}
}
}
highlight(document.body);
// Re-highlight on dynamic content
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1 || node.nodeType === 3) highlight(node);
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Highlight Search Terms]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Add Europeana integration - #200
Merged
TimidRobot merged 33 commits into
creativecommons:mainfrom 
Joyakis:europeana-feature
Nov 1, 2025
Merged
Add Europeana integration#200
TimidRobot merged 33 commits into
creativecommons:mainfrom 
Joyakis:europeana-feature
Conversation
@Joyakis
@JoyakisJoyakis
commented
Oct 15, 2025•

edited by TimidRobot

Loading

Copy link


Copy Markdown

Contributor
Fixes
Description
Add Europeana API integration for metrics collection
This PR adds a new script europeana_fetch.py that fetches and aggregates data from the Europeana Search API.

The script collects high-level statistics about cultural heritage content available through Europeana, focusing on data provider distribution and content types rather than fragile license parsing.
Technical details
Script Location: scripts/1-fetch/europeana_fetch.py
  • Data Output: data/2025Q4/1-fetch/europeana_1_count.csv
  • Key Features:

    • Fetches data from Europeana Search API using multiple content queries
    • Aggregates by DATA_PROVIDER and content metadata
    • Includes proper error handling and API rate limiting
    • Integrates with existing project structure and git workflows
  • Environment: Updated env.example with EUROPEANA_API_KEY placeholder
Tests
  1. Set up Europeana API key in environment variables
  2. Run the script: python scripts/1-fetch/europeana_fetch.py --enable-save
  3. Verify CSV output is generated in data/2025Q4/1-fetch/europeana_1_count.csv
  4. Check that data contains aggregated counts by DATA_PROVIDER
Checklist
  •  I have read and understood the Developer Certificate of Origin (DCO), below, which covers the contents of this pull request (PR).
  •  My pull request doesn't include code or content generated with AI.
  •  My pull request has a descriptive title (not a vague title like Update index.md).
  •  My pull request targets the default branch of the repository (main or master).
  •  My commit messages follow best practices.
  •  My code follows the established code style of the repository.
  •  I added or updated tests for the changes I made (if applicable).
  •  I added or updated documentation (if applicable).
  •  I tried running the project locally and verified that there are no
    
visible errors.
Developer Certificate of Origin
For the purposes of this DCO, "license" is equivalent to "license or public domain dedication," and "open source license" is equivalent to "open content license or public domain dedication."
Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

@Joyakis
Joyakis
requested review from
a team
as code ownersOctober 15, 2025 19:48
@Joyakis
Joyakis
requested review from
TimidRobot and possumbilities
and removed request for
a teamOctober 15, 2025 19:48
@cc-open-source-botcc-open-source-bot
moved this to In review
in TimidRobotOct 15, 2025
Babi-B
Babi-B
reviewed
Oct 15, 2025
View reviewed changes
Comment threadenv.example
TimidRobot
TimidRobot
requested changes
Oct 17, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadenv.example
Comment threadPipfile
Outdated

Comment threadPipfile.lock
@TimidRobotTimidRobot
self-assigned this
Oct 18, 2025
@TimidRobotTimidRobot
changed the title
Added Europeana integrationAdd Europeana integrationOct 20, 2025
TimidRobot
TimidRobot
reviewed
Oct 20, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

@TimidRobot
TimidRobot
commented
Oct 21, 2025

Copy link


Copy Markdown

Member
@Joyakis I resolved the conflicts with the main branch (due to changes from another pull request being merged). Please remember to fetch the changes to your computer.
@Joyakis
Joyakis
commented
Oct 21, 2025

Copy link


Copy Markdown

ContributorAuthor
@Joyakis I resolved the conflicts with the main branch (due to changes from another pull request being merged). Please remember to fetch the changes to your computer.
@TimidRobot Done!
TimidRobot
TimidRobot
requested changes
Oct 21, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

TimidRobot
TimidRobot
requested changes
Oct 21, 2025
View reviewed changes
@TimidRobotTimidRobot
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please resolve all comments before requesting a new review
Comment threadscripts/1-fetch/europeana_fetch.py
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

@TimidRobot
TimidRobot
commented
Oct 24, 2025

Copy link


Copy Markdown

Member
Please focus on resolving conversations before you add new features.
Please remove custom logging (print statements) and follow logging conventions in other scripts.
@Joyakis
Joyakis
commented
Oct 24, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Please focus on resolving conversations before you add new features.
Please remove custom logging (print statements) and follow logging conventions in other scripts.
Hello @TimidRobot The script now gets all providers and rights.
Thank you for pointing out where the problem was.
I have also removed the unnecessary printing statements
@Joyakis
Joyakis
commented
Oct 28, 2025

Copy link


Copy Markdown

ContributorAuthor
2. you review the pull request Files changed tab yourself to see if there are any unintended files or other obvious issues
Hello....I reviewed the Files changed tab in my pull request as you advised.

Only the intended updates to europeana_fetch.py are listed.

The other files in the folder (env.example and sources.md) came from merging the latest main branch into my feature branch and they do not show up in the diff .
@Joyakis
Joyakis
commented
Oct 28, 2025

Copy link


Copy Markdown

ContributorAuthor
Greetings @TimidRobot

This is a screenshot of the data without themes
image
This is the one with themes
image
TimidRobot
TimidRobot
reviewed
Oct 29, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
TimidRobot
TimidRobot
reviewed
Oct 30, 2025
View reviewed changes
Comment threadsources.md
@Joyakis
Joyakis
commented
Oct 30, 2025

Copy link


Copy Markdown

ContributorAuthor
Hello @TimidRobot
I have updated both the env.example and sources file
TimidRobot
TimidRobot
reviewed
Oct 30, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
TimidRobot
TimidRobot
reviewed
Oct 30, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

TimidRobot
TimidRobot
reviewed
Oct 30, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
TimidRobot
TimidRobot
reviewed
Oct 30, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Joyakisand others
added 5 commits
October 31, 2025 13:06
TimidRobot
TimidRobot
approved these changes
Nov 1, 2025
View reviewed changes
@TimidRobotTimidRobot
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work on this! Thank you!!
@TimidRobot
TimidRobot
merged commit 46dcd3f
into
creativecommons:mainNov 1, 2025
@github-project-automationgithub-project-automationBot
moved this from In review
to Done
in TimidRobotNov 1, 2025
@Joyakis
Joyakis
commented
Nov 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Great work on this! Thank you!!
Happy to have done this!...Thank you as well
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@TimidRobotTimidRobotTimidRobot approved these changes
@possumbilitiespossumbilitiesAwaiting requested review from possumbilitiespossumbilities is a code owner automatically assigned from creativecommons/technology

+1 more reviewer

@Babi-BBabi-BBabi-B left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

@TimidRobotTimidRobot

Labels


None yet


Projects

TimidRobot

Archived in project


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Add Europeana as a data source


4 participants

@Joyakis@TimidRobot@Babi-B@cc-open-source-bot

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Strip utm_, fbclid, gclid, etc. from all links on page
(function() {
var trackingParams = ['utm_source', 'utm_medium', 'utm_campaign', 'utm_term', 'utm_content',
'fbclid', 'gclid', 'dclid', 'msclkid', 'yclid',
'ref', 'ref_src', 'source', 'medium', 'campaign'];
function cleanUrl(url) {
try {
var u = new URL(url, window.location.origin);
var changed = false;
trackingParams.forEach(function(p) {
if (u.searchParams.has(p)) {
u.searchParams.delete(p);
changed = true;
}
});
return changed ? u.toString() : url;
} catch (e) {
return url;
}
}
function cleanLinks() {
document.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
cleanLinks();
var observer = new MutationObserver(function(mutations) {
mutations.forEach(function(m) {
m.addedNodes.forEach(function(node) {
if (node.nodeType === 1) {
if (node.tagName === 'A') cleanLinks();
node.querySelectorAll('a[href]').forEach(function(a) {
var clean = cleanUrl(a.href);
if (clean !== a.href) a.href = clean;
});
}
});
});
});
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:Remove Tracking Parameters from Links]', __e); }
})();
(function(){
try {
var __m = "youtube.com";
var __re = new RegExp('^' + "youtube\\.com" + '

Skip to content
Add Europeana integration - #200
Merged
TimidRobot merged 33 commits into
creativecommons:mainfrom 
Joyakis:europeana-feature
Nov 1, 2025
Merged
Add Europeana integration#200
TimidRobot merged 33 commits into
creativecommons:mainfrom 
Joyakis:europeana-feature
Conversation
@Joyakis
@JoyakisJoyakis
commented
Oct 15, 2025•

edited by TimidRobot

Loading

Copy link


Copy Markdown

Contributor
Fixes
Description
Add Europeana API integration for metrics collection
This PR adds a new script europeana_fetch.py that fetches and aggregates data from the Europeana Search API.

The script collects high-level statistics about cultural heritage content available through Europeana, focusing on data provider distribution and content types rather than fragile license parsing.
Technical details
Script Location: scripts/1-fetch/europeana_fetch.py
  • Data Output: data/2025Q4/1-fetch/europeana_1_count.csv
  • Key Features:

    • Fetches data from Europeana Search API using multiple content queries
    • Aggregates by DATA_PROVIDER and content metadata
    • Includes proper error handling and API rate limiting
    • Integrates with existing project structure and git workflows
  • Environment: Updated env.example with EUROPEANA_API_KEY placeholder
Tests
  1. Set up Europeana API key in environment variables
  2. Run the script: python scripts/1-fetch/europeana_fetch.py --enable-save
  3. Verify CSV output is generated in data/2025Q4/1-fetch/europeana_1_count.csv
  4. Check that data contains aggregated counts by DATA_PROVIDER
Checklist
  •  I have read and understood the Developer Certificate of Origin (DCO), below, which covers the contents of this pull request (PR).
  •  My pull request doesn't include code or content generated with AI.
  •  My pull request has a descriptive title (not a vague title like Update index.md).
  •  My pull request targets the default branch of the repository (main or master).
  •  My commit messages follow best practices.
  •  My code follows the established code style of the repository.
  •  I added or updated tests for the changes I made (if applicable).
  •  I added or updated documentation (if applicable).
  •  I tried running the project locally and verified that there are no
    
visible errors.
Developer Certificate of Origin
For the purposes of this DCO, "license" is equivalent to "license or public domain dedication," and "open source license" is equivalent to "open content license or public domain dedication."
Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

@Joyakis
Joyakis
requested review from
a team
as code ownersOctober 15, 2025 19:48
@Joyakis
Joyakis
requested review from
TimidRobot and possumbilities
and removed request for
a teamOctober 15, 2025 19:48
@cc-open-source-botcc-open-source-bot
moved this to In review
in TimidRobotOct 15, 2025
Babi-B
Babi-B
reviewed
Oct 15, 2025
View reviewed changes
Comment threadenv.example
TimidRobot
TimidRobot
requested changes
Oct 17, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadenv.example
Comment threadPipfile
Outdated

Comment threadPipfile.lock
@TimidRobotTimidRobot
self-assigned this
Oct 18, 2025
@TimidRobotTimidRobot
changed the title
Added Europeana integrationAdd Europeana integrationOct 20, 2025
TimidRobot
TimidRobot
reviewed
Oct 20, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

@TimidRobot
TimidRobot
commented
Oct 21, 2025

Copy link


Copy Markdown

Member
@Joyakis I resolved the conflicts with the main branch (due to changes from another pull request being merged). Please remember to fetch the changes to your computer.
@Joyakis
Joyakis
commented
Oct 21, 2025

Copy link


Copy Markdown

ContributorAuthor
@Joyakis I resolved the conflicts with the main branch (due to changes from another pull request being merged). Please remember to fetch the changes to your computer.
@TimidRobot Done!
TimidRobot
TimidRobot
requested changes
Oct 21, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

TimidRobot
TimidRobot
requested changes
Oct 21, 2025
View reviewed changes
@TimidRobotTimidRobot
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please resolve all comments before requesting a new review
Comment threadscripts/1-fetch/europeana_fetch.py
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

@TimidRobot
TimidRobot
commented
Oct 24, 2025

Copy link


Copy Markdown

Member
Please focus on resolving conversations before you add new features.
Please remove custom logging (print statements) and follow logging conventions in other scripts.
@Joyakis
Joyakis
commented
Oct 24, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Please focus on resolving conversations before you add new features.
Please remove custom logging (print statements) and follow logging conventions in other scripts.
Hello @TimidRobot The script now gets all providers and rights.
Thank you for pointing out where the problem was.
I have also removed the unnecessary printing statements
@Joyakis
Joyakis
commented
Oct 28, 2025

Copy link


Copy Markdown

ContributorAuthor
2. you review the pull request Files changed tab yourself to see if there are any unintended files or other obvious issues
Hello....I reviewed the Files changed tab in my pull request as you advised.

Only the intended updates to europeana_fetch.py are listed.

The other files in the folder (env.example and sources.md) came from merging the latest main branch into my feature branch and they do not show up in the diff .
@Joyakis
Joyakis
commented
Oct 28, 2025

Copy link


Copy Markdown

ContributorAuthor
Greetings @TimidRobot

This is a screenshot of the data without themes
image
This is the one with themes
image
TimidRobot
TimidRobot
reviewed
Oct 29, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
TimidRobot
TimidRobot
reviewed
Oct 30, 2025
View reviewed changes
Comment threadsources.md
@Joyakis
Joyakis
commented
Oct 30, 2025

Copy link


Copy Markdown

ContributorAuthor
Hello @TimidRobot
I have updated both the env.example and sources file
TimidRobot
TimidRobot
reviewed
Oct 30, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
TimidRobot
TimidRobot
reviewed
Oct 30, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

TimidRobot
TimidRobot
reviewed
Oct 30, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
TimidRobot
TimidRobot
reviewed
Oct 30, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Joyakisand others
added 5 commits
October 31, 2025 13:06
TimidRobot
TimidRobot
approved these changes
Nov 1, 2025
View reviewed changes
@TimidRobotTimidRobot
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work on this! Thank you!!
@TimidRobot
TimidRobot
merged commit 46dcd3f
into
creativecommons:mainNov 1, 2025
@github-project-automationgithub-project-automationBot
moved this from In review
to Done
in TimidRobotNov 1, 2025
@Joyakis
Joyakis
commented
Nov 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Great work on this! Thank you!!
Happy to have done this!...Thank you as well
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@TimidRobotTimidRobotTimidRobot approved these changes
@possumbilitiespossumbilitiesAwaiting requested review from possumbilitiespossumbilities is a code owner automatically assigned from creativecommons/technology

+1 more reviewer

@Babi-BBabi-BBabi-B left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

@TimidRobotTimidRobot

Labels


None yet


Projects

TimidRobot

Archived in project


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Add Europeana as a data source


4 participants

@Joyakis@TimidRobot@Babi-B@cc-open-source-bot

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Auto-enable theater mode on YouTube
(function() {
function tryTheater() {
var btn = document.querySelector('button[aria-label="Theater mode"], ytd-player #player button[title="Theater mode"]');
if (btn && !btn.classList.contains('activated')) {
btn.click();
}
}
// Try immediately
tryTheater();
// Try after navigation (SPA)
var lastUrl = location.href;
setInterval(function() {
if (location.href !== lastUrl) {
lastUrl = location.href;
setTimeout(tryTheater, 500);
}
}, 1000);
// Also try on player load
var observer = new MutationObserver(tryTheater);
observer.observe(document.body, { childList: true, subtree: true });
})();
}
} catch(__e) { console.warn('[Userscript:YouTube Theater Mode Default]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Add Europeana integration - #200
Merged
TimidRobot merged 33 commits into
creativecommons:mainfrom 
Joyakis:europeana-feature
Nov 1, 2025
Merged
Add Europeana integration#200
TimidRobot merged 33 commits into
creativecommons:mainfrom 
Joyakis:europeana-feature
Conversation
@Joyakis
@JoyakisJoyakis
commented
Oct 15, 2025•

edited by TimidRobot

Loading

Copy link


Copy Markdown

Contributor
Fixes
Description
Add Europeana API integration for metrics collection
This PR adds a new script europeana_fetch.py that fetches and aggregates data from the Europeana Search API.

The script collects high-level statistics about cultural heritage content available through Europeana, focusing on data provider distribution and content types rather than fragile license parsing.
Technical details
Script Location: scripts/1-fetch/europeana_fetch.py
  • Data Output: data/2025Q4/1-fetch/europeana_1_count.csv
  • Key Features:

    • Fetches data from Europeana Search API using multiple content queries
    • Aggregates by DATA_PROVIDER and content metadata
    • Includes proper error handling and API rate limiting
    • Integrates with existing project structure and git workflows
  • Environment: Updated env.example with EUROPEANA_API_KEY placeholder
Tests
  1. Set up Europeana API key in environment variables
  2. Run the script: python scripts/1-fetch/europeana_fetch.py --enable-save
  3. Verify CSV output is generated in data/2025Q4/1-fetch/europeana_1_count.csv
  4. Check that data contains aggregated counts by DATA_PROVIDER
Checklist
  •  I have read and understood the Developer Certificate of Origin (DCO), below, which covers the contents of this pull request (PR).
  •  My pull request doesn't include code or content generated with AI.
  •  My pull request has a descriptive title (not a vague title like Update index.md).
  •  My pull request targets the default branch of the repository (main or master).
  •  My commit messages follow best practices.
  •  My code follows the established code style of the repository.
  •  I added or updated tests for the changes I made (if applicable).
  •  I added or updated documentation (if applicable).
  •  I tried running the project locally and verified that there are no
    
visible errors.
Developer Certificate of Origin
For the purposes of this DCO, "license" is equivalent to "license or public domain dedication," and "open source license" is equivalent to "open content license or public domain dedication."
Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

@Joyakis
Joyakis
requested review from
a team
as code ownersOctober 15, 2025 19:48
@Joyakis
Joyakis
requested review from
TimidRobot and possumbilities
and removed request for
a teamOctober 15, 2025 19:48
@cc-open-source-botcc-open-source-bot
moved this to In review
in TimidRobotOct 15, 2025
Babi-B
Babi-B
reviewed
Oct 15, 2025
View reviewed changes
Comment threadenv.example
TimidRobot
TimidRobot
requested changes
Oct 17, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadenv.example
Comment threadPipfile
Outdated

Comment threadPipfile.lock
@TimidRobotTimidRobot
self-assigned this
Oct 18, 2025
@TimidRobotTimidRobot
changed the title
Added Europeana integrationAdd Europeana integrationOct 20, 2025
TimidRobot
TimidRobot
reviewed
Oct 20, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

@TimidRobot
TimidRobot
commented
Oct 21, 2025

Copy link


Copy Markdown

Member
@Joyakis I resolved the conflicts with the main branch (due to changes from another pull request being merged). Please remember to fetch the changes to your computer.
@Joyakis
Joyakis
commented
Oct 21, 2025

Copy link


Copy Markdown

ContributorAuthor
@Joyakis I resolved the conflicts with the main branch (due to changes from another pull request being merged). Please remember to fetch the changes to your computer.
@TimidRobot Done!
TimidRobot
TimidRobot
requested changes
Oct 21, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

TimidRobot
TimidRobot
requested changes
Oct 21, 2025
View reviewed changes
@TimidRobotTimidRobot
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please resolve all comments before requesting a new review
Comment threadscripts/1-fetch/europeana_fetch.py
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

@TimidRobot
TimidRobot
commented
Oct 24, 2025

Copy link


Copy Markdown

Member
Please focus on resolving conversations before you add new features.
Please remove custom logging (print statements) and follow logging conventions in other scripts.
@Joyakis
Joyakis
commented
Oct 24, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Please focus on resolving conversations before you add new features.
Please remove custom logging (print statements) and follow logging conventions in other scripts.
Hello @TimidRobot The script now gets all providers and rights.
Thank you for pointing out where the problem was.
I have also removed the unnecessary printing statements
@Joyakis
Joyakis
commented
Oct 28, 2025

Copy link


Copy Markdown

ContributorAuthor
2. you review the pull request Files changed tab yourself to see if there are any unintended files or other obvious issues
Hello....I reviewed the Files changed tab in my pull request as you advised.

Only the intended updates to europeana_fetch.py are listed.

The other files in the folder (env.example and sources.md) came from merging the latest main branch into my feature branch and they do not show up in the diff .
@Joyakis
Joyakis
commented
Oct 28, 2025

Copy link


Copy Markdown

ContributorAuthor
Greetings @TimidRobot

This is a screenshot of the data without themes
image
This is the one with themes
image
TimidRobot
TimidRobot
reviewed
Oct 29, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
TimidRobot
TimidRobot
reviewed
Oct 30, 2025
View reviewed changes
Comment threadsources.md
@Joyakis
Joyakis
commented
Oct 30, 2025

Copy link


Copy Markdown

ContributorAuthor
Hello @TimidRobot
I have updated both the env.example and sources file
TimidRobot
TimidRobot
reviewed
Oct 30, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
TimidRobot
TimidRobot
reviewed
Oct 30, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

TimidRobot
TimidRobot
reviewed
Oct 30, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
TimidRobot
TimidRobot
reviewed
Oct 30, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Joyakisand others
added 5 commits
October 31, 2025 13:06
TimidRobot
TimidRobot
approved these changes
Nov 1, 2025
View reviewed changes
@TimidRobotTimidRobot
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work on this! Thank you!!
@TimidRobot
TimidRobot
merged commit 46dcd3f
into
creativecommons:mainNov 1, 2025
@github-project-automationgithub-project-automationBot
moved this from In review
to Done
in TimidRobotNov 1, 2025
@Joyakis
Joyakis
commented
Nov 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Great work on this! Thank you!!
Happy to have done this!...Thank you as well
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@TimidRobotTimidRobotTimidRobot approved these changes
@possumbilitiespossumbilitiesAwaiting requested review from possumbilitiespossumbilities is a code owner automatically assigned from creativecommons/technology

+1 more reviewer

@Babi-BBabi-BBabi-B left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

@TimidRobotTimidRobot

Labels


None yet


Projects

TimidRobot

Archived in project


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Add Europeana as a data source


4 participants

@Joyakis@TimidRobot@Babi-B@cc-open-source-bot

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Remove or un-stick sticky/fixed headers that block content
(function() {
function unstick() {
document.querySelectorAll('header, nav, [role="banner"], .header, .navbar, .sticky, .fixed-top, [style*="position: fixed"], [style*="position:sticky"]').forEach(function(el) {
if (el.style.position === 'fixed' || el.style.position === 'sticky' || getComputedStyle(el).position === 'fixed' || getComputedStyle(el).position === 'sticky') {
el.style.position = 'static';
el.style.top = 'auto';
el.style.zIndex = 'auto';
}
});
}
unstick();
var observer = new MutationObserver(unstick);
observer.observe(document.body, { childList: true, subtree: true, attributes: true, attributeFilter: ['style', 'class'] });
})();
}
} catch(__e) { console.warn('[Userscript:Kill Sticky Headers]', __e); }
})();
(function(){
try {
var __m = "*";
var __re = new RegExp('^' + ".*" + '

Skip to content
Add Europeana integration - #200
Merged
TimidRobot merged 33 commits into
creativecommons:mainfrom 
Joyakis:europeana-feature
Nov 1, 2025
Merged
Add Europeana integration#200
TimidRobot merged 33 commits into
creativecommons:mainfrom 
Joyakis:europeana-feature
Conversation
@Joyakis
@JoyakisJoyakis
commented
Oct 15, 2025•

edited by TimidRobot

Loading

Copy link


Copy Markdown

Contributor
Fixes
Description
Add Europeana API integration for metrics collection
This PR adds a new script europeana_fetch.py that fetches and aggregates data from the Europeana Search API.

The script collects high-level statistics about cultural heritage content available through Europeana, focusing on data provider distribution and content types rather than fragile license parsing.
Technical details
Script Location: scripts/1-fetch/europeana_fetch.py
  • Data Output: data/2025Q4/1-fetch/europeana_1_count.csv
  • Key Features:

    • Fetches data from Europeana Search API using multiple content queries
    • Aggregates by DATA_PROVIDER and content metadata
    • Includes proper error handling and API rate limiting
    • Integrates with existing project structure and git workflows
  • Environment: Updated env.example with EUROPEANA_API_KEY placeholder
Tests
  1. Set up Europeana API key in environment variables
  2. Run the script: python scripts/1-fetch/europeana_fetch.py --enable-save
  3. Verify CSV output is generated in data/2025Q4/1-fetch/europeana_1_count.csv
  4. Check that data contains aggregated counts by DATA_PROVIDER
Checklist
  •  I have read and understood the Developer Certificate of Origin (DCO), below, which covers the contents of this pull request (PR).
  •  My pull request doesn't include code or content generated with AI.
  •  My pull request has a descriptive title (not a vague title like Update index.md).
  •  My pull request targets the default branch of the repository (main or master).
  •  My commit messages follow best practices.
  •  My code follows the established code style of the repository.
  •  I added or updated tests for the changes I made (if applicable).
  •  I added or updated documentation (if applicable).
  •  I tried running the project locally and verified that there are no
    
visible errors.
Developer Certificate of Origin
For the purposes of this DCO, "license" is equivalent to "license or public domain dedication," and "open source license" is equivalent to "open content license or public domain dedication."
Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

@Joyakis
Joyakis
requested review from
a team
as code ownersOctober 15, 2025 19:48
@Joyakis
Joyakis
requested review from
TimidRobot and possumbilities
and removed request for
a teamOctober 15, 2025 19:48
@cc-open-source-botcc-open-source-bot
moved this to In review
in TimidRobotOct 15, 2025
Babi-B
Babi-B
reviewed
Oct 15, 2025
View reviewed changes
Comment threadenv.example
TimidRobot
TimidRobot
requested changes
Oct 17, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadenv.example
Comment threadPipfile
Outdated

Comment threadPipfile.lock
@TimidRobotTimidRobot
self-assigned this
Oct 18, 2025
@TimidRobotTimidRobot
changed the title
Added Europeana integrationAdd Europeana integrationOct 20, 2025
TimidRobot
TimidRobot
reviewed
Oct 20, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

@TimidRobot
TimidRobot
commented
Oct 21, 2025

Copy link


Copy Markdown

Member
@Joyakis I resolved the conflicts with the main branch (due to changes from another pull request being merged). Please remember to fetch the changes to your computer.
@Joyakis
Joyakis
commented
Oct 21, 2025

Copy link


Copy Markdown

ContributorAuthor
@Joyakis I resolved the conflicts with the main branch (due to changes from another pull request being merged). Please remember to fetch the changes to your computer.
@TimidRobot Done!
TimidRobot
TimidRobot
requested changes
Oct 21, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

TimidRobot
TimidRobot
requested changes
Oct 21, 2025
View reviewed changes
@TimidRobotTimidRobot
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please resolve all comments before requesting a new review
Comment threadscripts/1-fetch/europeana_fetch.py
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

@TimidRobot
TimidRobot
commented
Oct 24, 2025

Copy link


Copy Markdown

Member
Please focus on resolving conversations before you add new features.
Please remove custom logging (print statements) and follow logging conventions in other scripts.
@Joyakis
Joyakis
commented
Oct 24, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Please focus on resolving conversations before you add new features.
Please remove custom logging (print statements) and follow logging conventions in other scripts.
Hello @TimidRobot The script now gets all providers and rights.
Thank you for pointing out where the problem was.
I have also removed the unnecessary printing statements
@Joyakis
Joyakis
commented
Oct 28, 2025

Copy link


Copy Markdown

ContributorAuthor
2. you review the pull request Files changed tab yourself to see if there are any unintended files or other obvious issues
Hello....I reviewed the Files changed tab in my pull request as you advised.

Only the intended updates to europeana_fetch.py are listed.

The other files in the folder (env.example and sources.md) came from merging the latest main branch into my feature branch and they do not show up in the diff .
@Joyakis
Joyakis
commented
Oct 28, 2025

Copy link


Copy Markdown

ContributorAuthor
Greetings @TimidRobot

This is a screenshot of the data without themes
image
This is the one with themes
image
TimidRobot
TimidRobot
reviewed
Oct 29, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
TimidRobot
TimidRobot
reviewed
Oct 30, 2025
View reviewed changes
Comment threadsources.md
@Joyakis
Joyakis
commented
Oct 30, 2025

Copy link


Copy Markdown

ContributorAuthor
Hello @TimidRobot
I have updated both the env.example and sources file
TimidRobot
TimidRobot
reviewed
Oct 30, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
TimidRobot
TimidRobot
reviewed
Oct 30, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

TimidRobot
TimidRobot
reviewed
Oct 30, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
TimidRobot
TimidRobot
reviewed
Oct 30, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Joyakisand others
added 5 commits
October 31, 2025 13:06
TimidRobot
TimidRobot
approved these changes
Nov 1, 2025
View reviewed changes
@TimidRobotTimidRobot
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work on this! Thank you!!
@TimidRobot
TimidRobot
merged commit 46dcd3f
into
creativecommons:mainNov 1, 2025
@github-project-automationgithub-project-automationBot
moved this from In review
to Done
in TimidRobotNov 1, 2025
@Joyakis
Joyakis
commented
Nov 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Great work on this! Thank you!!
Happy to have done this!...Thank you as well
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@TimidRobotTimidRobotTimidRobot approved these changes
@possumbilitiespossumbilitiesAwaiting requested review from possumbilitiespossumbilities is a code owner automatically assigned from creativecommons/technology

+1 more reviewer

@Babi-BBabi-BBabi-B left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

@TimidRobotTimidRobot

Labels


None yet


Projects

TimidRobot

Archived in project


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Add Europeana as a data source


4 participants

@Joyakis@TimidRobot@Babi-B@cc-open-source-bot

, 'i');
if (__m === '*' || __re.test(location.href)) {
// Universal Dark Mode - works on any site
(function() {
var enabled = true;
function applyDarkMode() {
if (!enabled) return;
// Create style element if it doesn't exist
var style = document.getElementById('universal-dark-mode-style');
if (!style) {
style = document.createElement('style');
style.id = 'universal-dark-mode-style';
document.head.appendChild(style);
}
// Dark mode CSS - inverts colors but preserves images/video
style.textContent = '
/* Invert everything except media */
html {
filter: invert(1) hue-rotate(180deg) !important;
background: #1a1a2e !important;
}
/* Restore images, videos, iframes, canvas */
img, video, iframe, canvas, svg, picture, [style*="background-image"] {
filter: invert(1) hue-rotate(180deg) !important;
}
/* Preserve specific elements that should not be inverted */
.no-dark-mode, .no-dark-mode *,
[data-theme="light"], [data-theme="light"],
.ace_editor, .ace_editor *,
.CodeMirror, .CodeMirror *,
.monaco-editor, .monaco-editor *,
.markdown-body pre, .markdown-body pre *,
.highlight, .highlight *,
pre code, pre code * {
filter: none !important;
}
/* Fix common UI elements */
.modal, .popup, .dropdown-menu, .tooltip, .popover {
filter: invert(1) hue-rotate(180deg) !important;
background: #2d2d44 !important;
border-color: #444 !important;
}
/* Scrollbars */
::-webkit-scrollbar { background: #1a1a2e !important; }
::-webkit-scrollbar-thumb { background: #444 !important; }
::-webkit-scrollbar-thumb:hover { background: #555 !important; }
/* Selection */
::selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
::-moz-selection { background: #4ecdc4 !important; color: #1a1a2e !important; }
';
}
function removeDarkMode() {
var style = document.getElementById('universal-dark-mode-style');
if (style) style.remove();
}
// Toggle with Alt+Shift+D
document.addEventListener('keydown', function(e) {
if (e.altKey && e.shiftKey && e.key === 'D') {
e.preventDefault();
enabled = !enabled;
if (enabled) {
applyDarkMode();
console.log('[Universal Dark Mode] Enabled');
} else {
removeDarkMode();
console.log('[Universal Dark Mode] Disabled');
}
}
});
// Apply on load
applyDarkMode();
// Re-apply on dynamic content
var observer = new MutationObserver(function(mutations) {
if (enabled && !document.getElementById('universal-dark-mode-style')) {
applyDarkMode();
}
});
observer.observe(document.head, { childList: true });
console.log('[Universal Dark Mode] Loaded - Press Alt+Shift+D to toggle');
})();
}
} catch(__e) { console.warn('[Userscript:Universal Dark Mode]', __e); }
})();
})();

Skip to content
Add Europeana integration - #200
Merged
TimidRobot merged 33 commits into
creativecommons:mainfrom 
Joyakis:europeana-feature
Nov 1, 2025
Merged
Add Europeana integration#200
TimidRobot merged 33 commits into
creativecommons:mainfrom 
Joyakis:europeana-feature
Conversation
@Joyakis
@JoyakisJoyakis
commented
Oct 15, 2025•

edited by TimidRobot

Loading

Copy link


Copy Markdown

Contributor
Fixes
Description
Add Europeana API integration for metrics collection
This PR adds a new script europeana_fetch.py that fetches and aggregates data from the Europeana Search API.

The script collects high-level statistics about cultural heritage content available through Europeana, focusing on data provider distribution and content types rather than fragile license parsing.
Technical details
Script Location: scripts/1-fetch/europeana_fetch.py
  • Data Output: data/2025Q4/1-fetch/europeana_1_count.csv
  • Key Features:

    • Fetches data from Europeana Search API using multiple content queries
    • Aggregates by DATA_PROVIDER and content metadata
    • Includes proper error handling and API rate limiting
    • Integrates with existing project structure and git workflows
  • Environment: Updated env.example with EUROPEANA_API_KEY placeholder
Tests
  1. Set up Europeana API key in environment variables
  2. Run the script: python scripts/1-fetch/europeana_fetch.py --enable-save
  3. Verify CSV output is generated in data/2025Q4/1-fetch/europeana_1_count.csv
  4. Check that data contains aggregated counts by DATA_PROVIDER
Checklist
  •  I have read and understood the Developer Certificate of Origin (DCO), below, which covers the contents of this pull request (PR).
  •  My pull request doesn't include code or content generated with AI.
  •  My pull request has a descriptive title (not a vague title like Update index.md).
  •  My pull request targets the default branch of the repository (main or master).
  •  My commit messages follow best practices.
  •  My code follows the established code style of the repository.
  •  I added or updated tests for the changes I made (if applicable).
  •  I added or updated documentation (if applicable).
  •  I tried running the project locally and verified that there are no
    
visible errors.
Developer Certificate of Origin
For the purposes of this DCO, "license" is equivalent to "license or public domain dedication," and "open source license" is equivalent to "open content license or public domain dedication."
Developer Certificate of Origin
Developer Certificate of Origin
Version 1.1
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

@Joyakis
Joyakis
requested review from
a team
as code ownersOctober 15, 2025 19:48
@Joyakis
Joyakis
requested review from
TimidRobot and possumbilities
and removed request for
a teamOctober 15, 2025 19:48
@cc-open-source-botcc-open-source-bot
moved this to In review
in TimidRobotOct 15, 2025
Babi-B
Babi-B
reviewed
Oct 15, 2025
View reviewed changes
Comment threadenv.example
TimidRobot
TimidRobot
requested changes
Oct 17, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadenv.example
Comment threadPipfile
Outdated

Comment threadPipfile.lock
@TimidRobotTimidRobot
self-assigned this
Oct 18, 2025
@TimidRobotTimidRobot
changed the title
Added Europeana integrationAdd Europeana integrationOct 20, 2025
TimidRobot
TimidRobot
reviewed
Oct 20, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

@TimidRobot
TimidRobot
commented
Oct 21, 2025

Copy link


Copy Markdown

Member
@Joyakis I resolved the conflicts with the main branch (due to changes from another pull request being merged). Please remember to fetch the changes to your computer.
@Joyakis
Joyakis
commented
Oct 21, 2025

Copy link


Copy Markdown

ContributorAuthor
@Joyakis I resolved the conflicts with the main branch (due to changes from another pull request being merged). Please remember to fetch the changes to your computer.
@TimidRobot Done!
TimidRobot
TimidRobot
requested changes
Oct 21, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

TimidRobot
TimidRobot
requested changes
Oct 21, 2025
View reviewed changes
@TimidRobotTimidRobot
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please resolve all comments before requesting a new review
Comment threadscripts/1-fetch/europeana_fetch.py
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

Comment threadscripts/1-fetch/europeana_fetch.py
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

@TimidRobot
TimidRobot
commented
Oct 24, 2025

Copy link


Copy Markdown

Member
Please focus on resolving conversations before you add new features.
Please remove custom logging (print statements) and follow logging conventions in other scripts.
@Joyakis
Joyakis
commented
Oct 24, 2025•

edited

Loading

Copy link


Copy Markdown

ContributorAuthor
Please focus on resolving conversations before you add new features.
Please remove custom logging (print statements) and follow logging conventions in other scripts.
Hello @TimidRobot The script now gets all providers and rights.
Thank you for pointing out where the problem was.
I have also removed the unnecessary printing statements
@Joyakis
Joyakis
commented
Oct 28, 2025

Copy link


Copy Markdown

ContributorAuthor
2. you review the pull request Files changed tab yourself to see if there are any unintended files or other obvious issues
Hello....I reviewed the Files changed tab in my pull request as you advised.

Only the intended updates to europeana_fetch.py are listed.

The other files in the folder (env.example and sources.md) came from merging the latest main branch into my feature branch and they do not show up in the diff .
@Joyakis
Joyakis
commented
Oct 28, 2025

Copy link


Copy Markdown

ContributorAuthor
Greetings @TimidRobot

This is a screenshot of the data without themes
image
This is the one with themes
image
TimidRobot
TimidRobot
reviewed
Oct 29, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
TimidRobot
TimidRobot
reviewed
Oct 30, 2025
View reviewed changes
Comment threadsources.md
@Joyakis
Joyakis
commented
Oct 30, 2025

Copy link


Copy Markdown

ContributorAuthor
Hello @TimidRobot
I have updated both the env.example and sources file
TimidRobot
TimidRobot
reviewed
Oct 30, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
TimidRobot
TimidRobot
reviewed
Oct 30, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Outdated

TimidRobot
TimidRobot
reviewed
Oct 30, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
TimidRobot
TimidRobot
reviewed
Oct 30, 2025
View reviewed changes
Comment threadscripts/1-fetch/europeana_fetch.py
Joyakisand others
added 5 commits
October 31, 2025 13:06
TimidRobot
TimidRobot
approved these changes
Nov 1, 2025
View reviewed changes
@TimidRobotTimidRobot
left a comment


Copy link


Copy Markdown

Member
Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work on this! Thank you!!
@TimidRobot
TimidRobot
merged commit 46dcd3f
into
creativecommons:mainNov 1, 2025
@github-project-automationgithub-project-automationBot
moved this from In review
to Done
in TimidRobotNov 1, 2025
@Joyakis
Joyakis
commented
Nov 1, 2025

Copy link


Copy Markdown

ContributorAuthor
Great work on this! Thank you!!
Happy to have done this!...Thank you as well
Sign up for freeto join this conversation on GitHub.
Already have an account?
Sign in to comment

Reviewers

@TimidRobotTimidRobotTimidRobot approved these changes
@possumbilitiespossumbilitiesAwaiting requested review from possumbilitiespossumbilities is a code owner automatically assigned from creativecommons/technology

+1 more reviewer

@Babi-BBabi-BBabi-B left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

@TimidRobotTimidRobot

Labels


None yet


Projects

TimidRobot

Archived in project


Milestone


No milestone


Development

Successfully merging this pull request may close these issues.

Add Europeana as a data source


4 participants

@Joyakis@TimidRobot@Babi-B@cc-open-source-bot