[DISCARDED] Pull Request: Fix ArXiv Fetch Script - Replace urllib with requests and improve error handling

[Goziee-git]

Fixes

• FixesFixed Issues raised upon reviews for integrating the arxiv fetching functionality in arxiv_fetch.py #189 by @Goziee-git

Description

This PR modernizes the arxiv_fetch.py script by replacing the deprecated urllib implementation with the more robust requests library and implements several critical improvements for reliability and maintainability.

Issues Fixed

• Deprecated urllib usage: Replaced urllib.request with requests library for better HTTP handling
• Poor error handling: Added exponential backoff retry strategy with proper exception handling
• Unnecessary complexity: Removed redundant plan index system from CSV output structure
• License detection bugs: Fixed uppercase conversion logic in license extraction functions
• Rate limiting issues: Simplified to use arXiv's recommended 3-second delay between API calls

Technical Changes

Core Library Migration

• Migrated from urllib.request to requests library
• Added HTTPAdapter with retry strategy (5 retries, exponential backoff)
• Implemented proper session management with timeout handling

Data Structure Improvements

• Removed PLAN_INDEX column from all CSV headers and output
• Simplified CSV structure: HEADER_COUNT, HEADER_CATEGORY, HEADER_YEAR, HEADER_AUTHOR
• Eliminated unnecessary plan indexing logic in save_count_data() function

License Detection Enhancement

• Fixed extract_license_info() function to properly handle case conversion
• Changed from .lower() to .upper() for consistent license matching
• Added explicit "Unknown" fallback for unmatched licenses

Error Handling & Rate Limiting

• Replaced manual retry loops with requests.Session() and HTTPAdapter
• Added requests.RequestException handling
• Simplified rate limiting to single 3-second delay (arXiv recommendation)
• Removed complex exponential backoff in favor of library-handled retries

Files Modified

• scripts/1-fetch/arxiv_fetch.py - Complete refactor with modernized HTTP handling

Checklist

• I have read and understood the Developer Certificate of Origin (DCO), below, which covers the contents of this pull request (PR).
• My pull request doesn't include code or content generated with AI.
• My pull request has a descriptive title (not a vague title like Update index.md).
• My pull request targets the default branch of the repository (main or master).
• My commit messages follow best practices best_practices.
• My code follows the established code style of the repository.
• I added or updated tests for the changes I made (if applicable).
• I added or updated documentation (if applicable).
• I tried running the project locally and verified that there are no
  visible errors.

Developer Certificate of Origin

For the purposes of this DCO, "license" is equivalent to "license or public domain dedication," and "open source license" is equivalent to "open content license or public domain dedication."

Developer Certificate of Origin

Developer Certificate of Origin
Version 1.1
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

[TimidRobot]

Please update your original pull request (PR) instead of creating a new one:

• Add arXiv data fetching functionality #179

Also see:

• Foundational technologies — Creative Commons Open Source

[Goziee-git]

Fixes

• FixesFixed Issues raised upon reviews for integrating the arxiv fetching functionality in arxiv_fetch.py #189 by @Goziee-git

Description

This PR modernizes the arxiv_fetch.py script by replacing the deprecated urllib implementation with the more robust requests library and implements several critical improvements for reliability and maintainability.

Issues Fixed

• Deprecated urllib usage: Replaced urllib.request with requests library for better HTTP handling
• Poor error handling: Added exponential backoff retry strategy with proper exception handling
• Unnecessary complexity: Removed redundant plan index system from CSV output structure
• License detection bugs: Fixed uppercase conversion logic in license extraction functions
• Rate limiting issues: Simplified to use arXiv's recommended 3-second delay between API calls

Technical Changes

Core Library Migration

• Migrated from urllib.request to requests library
• Added HTTPAdapter with retry strategy (5 retries, exponential backoff)
• Implemented proper session management with timeout handling

Data Structure Improvements

• Removed PLAN_INDEX column from all CSV headers and output
• Simplified CSV structure: HEADER_COUNT, HEADER_CATEGORY, HEADER_YEAR, HEADER_AUTHOR
• Eliminated unnecessary plan indexing logic in save_count_data() function

License Detection Enhancement

• Fixed extract_license_info() function to properly handle case conversion
• Changed from .lower() to .upper() for consistent license matching
• Added explicit "Unknown" fallback for unmatched licenses

Error Handling & Rate Limiting

• Replaced manual retry loops with requests.Session() and HTTPAdapter
• Added requests.RequestException handling
• Simplified rate limiting to single 3-second delay (arXiv recommendation)
• Removed complex exponential backoff in favor of library-handled retries

Files Modified

• scripts/1-fetch/arxiv_fetch.py - Complete refactor with modernized HTTP handling

Checklist

• I have read and understood the Developer Certificate of Origin (DCO), below, which covers the contents of this pull request (PR).
• My pull request doesn't include code or content generated with AI.
• My pull request has a descriptive title (not a vague title like Update index.md).
• My pull request targets the default branch of the repository (main or master).
• My commit messages follow best practices best_practices.
• My code follows the established code style of the repository.
• I added or updated tests for the changes I made (if applicable).
• I added or updated documentation (if applicable).
• I tried running the project locally and verified that there are no
  visible errors.

Developer Certificate of Origin

For the purposes of this DCO, "license" is equivalent to "license or public domain dedication," and "open source license" is equivalent to "open content license or public domain dedication."

Developer Certificate of Origin

Developer Certificate of Origin
Version 1.1
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

[TimidRobot]

Please update your original pull request (PR) instead of creating a new one:

• Add arXiv data fetching functionality #179

Also see:

• Foundational technologies — Creative Commons Open Source

[Goziee-git]

Fixes

• FixesFixed Issues raised upon reviews for integrating the arxiv fetching functionality in arxiv_fetch.py #189 by @Goziee-git

Description

This PR modernizes the arxiv_fetch.py script by replacing the deprecated urllib implementation with the more robust requests library and implements several critical improvements for reliability and maintainability.

Issues Fixed

• Deprecated urllib usage: Replaced urllib.request with requests library for better HTTP handling
• Poor error handling: Added exponential backoff retry strategy with proper exception handling
• Unnecessary complexity: Removed redundant plan index system from CSV output structure
• License detection bugs: Fixed uppercase conversion logic in license extraction functions
• Rate limiting issues: Simplified to use arXiv's recommended 3-second delay between API calls

Technical Changes

Core Library Migration

• Migrated from urllib.request to requests library
• Added HTTPAdapter with retry strategy (5 retries, exponential backoff)
• Implemented proper session management with timeout handling

Data Structure Improvements

• Removed PLAN_INDEX column from all CSV headers and output
• Simplified CSV structure: HEADER_COUNT, HEADER_CATEGORY, HEADER_YEAR, HEADER_AUTHOR
• Eliminated unnecessary plan indexing logic in save_count_data() function

License Detection Enhancement

• Fixed extract_license_info() function to properly handle case conversion
• Changed from .lower() to .upper() for consistent license matching
• Added explicit "Unknown" fallback for unmatched licenses

Error Handling & Rate Limiting

• Replaced manual retry loops with requests.Session() and HTTPAdapter
• Added requests.RequestException handling
• Simplified rate limiting to single 3-second delay (arXiv recommendation)
• Removed complex exponential backoff in favor of library-handled retries

Files Modified

• scripts/1-fetch/arxiv_fetch.py - Complete refactor with modernized HTTP handling

Checklist

• I have read and understood the Developer Certificate of Origin (DCO), below, which covers the contents of this pull request (PR).
• My pull request doesn't include code or content generated with AI.
• My pull request has a descriptive title (not a vague title like Update index.md).
• My pull request targets the default branch of the repository (main or master).
• My commit messages follow best practices best_practices.
• My code follows the established code style of the repository.
• I added or updated tests for the changes I made (if applicable).
• I added or updated documentation (if applicable).
• I tried running the project locally and verified that there are no
  visible errors.

Developer Certificate of Origin

For the purposes of this DCO, "license" is equivalent to "license or public domain dedication," and "open source license" is equivalent to "open content license or public domain dedication."

Developer Certificate of Origin

Developer Certificate of Origin
Version 1.1
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

[TimidRobot]

Please update your original pull request (PR) instead of creating a new one:

• Add arXiv data fetching functionality #179

Also see:

• Foundational technologies — Creative Commons Open Source

[Goziee-git]

Fixes

• FixesFixed Issues raised upon reviews for integrating the arxiv fetching functionality in arxiv_fetch.py #189 by @Goziee-git

Description

This PR modernizes the arxiv_fetch.py script by replacing the deprecated urllib implementation with the more robust requests library and implements several critical improvements for reliability and maintainability.

Issues Fixed

• Deprecated urllib usage: Replaced urllib.request with requests library for better HTTP handling
• Poor error handling: Added exponential backoff retry strategy with proper exception handling
• Unnecessary complexity: Removed redundant plan index system from CSV output structure
• License detection bugs: Fixed uppercase conversion logic in license extraction functions
• Rate limiting issues: Simplified to use arXiv's recommended 3-second delay between API calls

Technical Changes

Core Library Migration

• Migrated from urllib.request to requests library
• Added HTTPAdapter with retry strategy (5 retries, exponential backoff)
• Implemented proper session management with timeout handling

Data Structure Improvements

• Removed PLAN_INDEX column from all CSV headers and output
• Simplified CSV structure: HEADER_COUNT, HEADER_CATEGORY, HEADER_YEAR, HEADER_AUTHOR
• Eliminated unnecessary plan indexing logic in save_count_data() function

License Detection Enhancement

• Fixed extract_license_info() function to properly handle case conversion
• Changed from .lower() to .upper() for consistent license matching
• Added explicit "Unknown" fallback for unmatched licenses

Error Handling & Rate Limiting

• Replaced manual retry loops with requests.Session() and HTTPAdapter
• Added requests.RequestException handling
• Simplified rate limiting to single 3-second delay (arXiv recommendation)
• Removed complex exponential backoff in favor of library-handled retries

Files Modified

• scripts/1-fetch/arxiv_fetch.py - Complete refactor with modernized HTTP handling

Checklist

• I have read and understood the Developer Certificate of Origin (DCO), below, which covers the contents of this pull request (PR).
• My pull request doesn't include code or content generated with AI.
• My pull request has a descriptive title (not a vague title like Update index.md).
• My pull request targets the default branch of the repository (main or master).
• My commit messages follow best practices best_practices.
• My code follows the established code style of the repository.
• I added or updated tests for the changes I made (if applicable).
• I added or updated documentation (if applicable).
• I tried running the project locally and verified that there are no
  visible errors.

Developer Certificate of Origin

For the purposes of this DCO, "license" is equivalent to "license or public domain dedication," and "open source license" is equivalent to "open content license or public domain dedication."

Developer Certificate of Origin

Developer Certificate of Origin
Version 1.1
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

[TimidRobot]

Please update your original pull request (PR) instead of creating a new one:

• Add arXiv data fetching functionality #179

Also see:

• Foundational technologies — Creative Commons Open Source

[Goziee-git]

Fixes

• FixesFixed Issues raised upon reviews for integrating the arxiv fetching functionality in arxiv_fetch.py #189 by @Goziee-git

Description

This PR modernizes the arxiv_fetch.py script by replacing the deprecated urllib implementation with the more robust requests library and implements several critical improvements for reliability and maintainability.

Issues Fixed

• Deprecated urllib usage: Replaced urllib.request with requests library for better HTTP handling
• Poor error handling: Added exponential backoff retry strategy with proper exception handling
• Unnecessary complexity: Removed redundant plan index system from CSV output structure
• License detection bugs: Fixed uppercase conversion logic in license extraction functions
• Rate limiting issues: Simplified to use arXiv's recommended 3-second delay between API calls

Technical Changes

Core Library Migration

• Migrated from urllib.request to requests library
• Added HTTPAdapter with retry strategy (5 retries, exponential backoff)
• Implemented proper session management with timeout handling

Data Structure Improvements

• Removed PLAN_INDEX column from all CSV headers and output
• Simplified CSV structure: HEADER_COUNT, HEADER_CATEGORY, HEADER_YEAR, HEADER_AUTHOR
• Eliminated unnecessary plan indexing logic in save_count_data() function

License Detection Enhancement

• Fixed extract_license_info() function to properly handle case conversion
• Changed from .lower() to .upper() for consistent license matching
• Added explicit "Unknown" fallback for unmatched licenses

Error Handling & Rate Limiting

• Replaced manual retry loops with requests.Session() and HTTPAdapter
• Added requests.RequestException handling
• Simplified rate limiting to single 3-second delay (arXiv recommendation)
• Removed complex exponential backoff in favor of library-handled retries

Files Modified

• scripts/1-fetch/arxiv_fetch.py - Complete refactor with modernized HTTP handling

Checklist

• I have read and understood the Developer Certificate of Origin (DCO), below, which covers the contents of this pull request (PR).
• My pull request doesn't include code or content generated with AI.
• My pull request has a descriptive title (not a vague title like Update index.md).
• My pull request targets the default branch of the repository (main or master).
• My commit messages follow best practices best_practices.
• My code follows the established code style of the repository.
• I added or updated tests for the changes I made (if applicable).
• I added or updated documentation (if applicable).
• I tried running the project locally and verified that there are no
  visible errors.

Developer Certificate of Origin

For the purposes of this DCO, "license" is equivalent to "license or public domain dedication," and "open source license" is equivalent to "open content license or public domain dedication."

Developer Certificate of Origin

Developer Certificate of Origin
Version 1.1
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

[TimidRobot]

Please update your original pull request (PR) instead of creating a new one:

• Add arXiv data fetching functionality #179

Also see:

• Foundational technologies — Creative Commons Open Source

[Goziee-git]

Fixes

• FixesFixed Issues raised upon reviews for integrating the arxiv fetching functionality in arxiv_fetch.py #189 by @Goziee-git

Description

This PR modernizes the arxiv_fetch.py script by replacing the deprecated urllib implementation with the more robust requests library and implements several critical improvements for reliability and maintainability.

Issues Fixed

• Deprecated urllib usage: Replaced urllib.request with requests library for better HTTP handling
• Poor error handling: Added exponential backoff retry strategy with proper exception handling
• Unnecessary complexity: Removed redundant plan index system from CSV output structure
• License detection bugs: Fixed uppercase conversion logic in license extraction functions
• Rate limiting issues: Simplified to use arXiv's recommended 3-second delay between API calls

Technical Changes

Core Library Migration

• Migrated from urllib.request to requests library
• Added HTTPAdapter with retry strategy (5 retries, exponential backoff)
• Implemented proper session management with timeout handling

Data Structure Improvements

• Removed PLAN_INDEX column from all CSV headers and output
• Simplified CSV structure: HEADER_COUNT, HEADER_CATEGORY, HEADER_YEAR, HEADER_AUTHOR
• Eliminated unnecessary plan indexing logic in save_count_data() function

License Detection Enhancement

• Fixed extract_license_info() function to properly handle case conversion
• Changed from .lower() to .upper() for consistent license matching
• Added explicit "Unknown" fallback for unmatched licenses

Error Handling & Rate Limiting

• Replaced manual retry loops with requests.Session() and HTTPAdapter
• Added requests.RequestException handling
• Simplified rate limiting to single 3-second delay (arXiv recommendation)
• Removed complex exponential backoff in favor of library-handled retries

Files Modified

• scripts/1-fetch/arxiv_fetch.py - Complete refactor with modernized HTTP handling

Checklist

• I have read and understood the Developer Certificate of Origin (DCO), below, which covers the contents of this pull request (PR).
• My pull request doesn't include code or content generated with AI.
• My pull request has a descriptive title (not a vague title like Update index.md).
• My pull request targets the default branch of the repository (main or master).
• My commit messages follow best practices best_practices.
• My code follows the established code style of the repository.
• I added or updated tests for the changes I made (if applicable).
• I added or updated documentation (if applicable).
• I tried running the project locally and verified that there are no
  visible errors.

Developer Certificate of Origin

For the purposes of this DCO, "license" is equivalent to "license or public domain dedication," and "open source license" is equivalent to "open content license or public domain dedication."

Developer Certificate of Origin

Developer Certificate of Origin
Version 1.1
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

[TimidRobot]

Please update your original pull request (PR) instead of creating a new one:

• Add arXiv data fetching functionality #179

Also see:

• Foundational technologies — Creative Commons Open Source

[Goziee-git]

Fixes

• FixesFixed Issues raised upon reviews for integrating the arxiv fetching functionality in arxiv_fetch.py #189 by @Goziee-git

Description

This PR modernizes the arxiv_fetch.py script by replacing the deprecated urllib implementation with the more robust requests library and implements several critical improvements for reliability and maintainability.

Issues Fixed

• Deprecated urllib usage: Replaced urllib.request with requests library for better HTTP handling
• Poor error handling: Added exponential backoff retry strategy with proper exception handling
• Unnecessary complexity: Removed redundant plan index system from CSV output structure
• License detection bugs: Fixed uppercase conversion logic in license extraction functions
• Rate limiting issues: Simplified to use arXiv's recommended 3-second delay between API calls

Technical Changes

Core Library Migration

• Migrated from urllib.request to requests library
• Added HTTPAdapter with retry strategy (5 retries, exponential backoff)
• Implemented proper session management with timeout handling

Data Structure Improvements

• Removed PLAN_INDEX column from all CSV headers and output
• Simplified CSV structure: HEADER_COUNT, HEADER_CATEGORY, HEADER_YEAR, HEADER_AUTHOR
• Eliminated unnecessary plan indexing logic in save_count_data() function

License Detection Enhancement

• Fixed extract_license_info() function to properly handle case conversion
• Changed from .lower() to .upper() for consistent license matching
• Added explicit "Unknown" fallback for unmatched licenses

Error Handling & Rate Limiting

• Replaced manual retry loops with requests.Session() and HTTPAdapter
• Added requests.RequestException handling
• Simplified rate limiting to single 3-second delay (arXiv recommendation)
• Removed complex exponential backoff in favor of library-handled retries

Files Modified

• scripts/1-fetch/arxiv_fetch.py - Complete refactor with modernized HTTP handling

Checklist

• I have read and understood the Developer Certificate of Origin (DCO), below, which covers the contents of this pull request (PR).
• My pull request doesn't include code or content generated with AI.
• My pull request has a descriptive title (not a vague title like Update index.md).
• My pull request targets the default branch of the repository (main or master).
• My commit messages follow best practices best_practices.
• My code follows the established code style of the repository.
• I added or updated tests for the changes I made (if applicable).
• I added or updated documentation (if applicable).
• I tried running the project locally and verified that there are no
  visible errors.

Developer Certificate of Origin

For the purposes of this DCO, "license" is equivalent to "license or public domain dedication," and "open source license" is equivalent to "open content license or public domain dedication."

Developer Certificate of Origin

Developer Certificate of Origin
Version 1.1
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

[TimidRobot]

Please update your original pull request (PR) instead of creating a new one:

• Add arXiv data fetching functionality #179

Also see:

• Foundational technologies — Creative Commons Open Source

[Goziee-git]

Fixes

• FixesFixed Issues raised upon reviews for integrating the arxiv fetching functionality in arxiv_fetch.py #189 by @Goziee-git

Description

This PR modernizes the arxiv_fetch.py script by replacing the deprecated urllib implementation with the more robust requests library and implements several critical improvements for reliability and maintainability.

Issues Fixed

• Deprecated urllib usage: Replaced urllib.request with requests library for better HTTP handling
• Poor error handling: Added exponential backoff retry strategy with proper exception handling
• Unnecessary complexity: Removed redundant plan index system from CSV output structure
• License detection bugs: Fixed uppercase conversion logic in license extraction functions
• Rate limiting issues: Simplified to use arXiv's recommended 3-second delay between API calls

Technical Changes

Core Library Migration

• Migrated from urllib.request to requests library
• Added HTTPAdapter with retry strategy (5 retries, exponential backoff)
• Implemented proper session management with timeout handling

Data Structure Improvements

• Removed PLAN_INDEX column from all CSV headers and output
• Simplified CSV structure: HEADER_COUNT, HEADER_CATEGORY, HEADER_YEAR, HEADER_AUTHOR
• Eliminated unnecessary plan indexing logic in save_count_data() function

License Detection Enhancement

• Fixed extract_license_info() function to properly handle case conversion
• Changed from .lower() to .upper() for consistent license matching
• Added explicit "Unknown" fallback for unmatched licenses

Error Handling & Rate Limiting

• Replaced manual retry loops with requests.Session() and HTTPAdapter
• Added requests.RequestException handling
• Simplified rate limiting to single 3-second delay (arXiv recommendation)
• Removed complex exponential backoff in favor of library-handled retries

Files Modified

• scripts/1-fetch/arxiv_fetch.py - Complete refactor with modernized HTTP handling

Checklist

• I have read and understood the Developer Certificate of Origin (DCO), below, which covers the contents of this pull request (PR).
• My pull request doesn't include code or content generated with AI.
• My pull request has a descriptive title (not a vague title like Update index.md).
• My pull request targets the default branch of the repository (main or master).
• My commit messages follow best practices best_practices.
• My code follows the established code style of the repository.
• I added or updated tests for the changes I made (if applicable).
• I added or updated documentation (if applicable).
• I tried running the project locally and verified that there are no
  visible errors.

Developer Certificate of Origin

For the purposes of this DCO, "license" is equivalent to "license or public domain dedication," and "open source license" is equivalent to "open content license or public domain dedication."

Developer Certificate of Origin

Developer Certificate of Origin
Version 1.1
Copyright (C) 2004, 2006 The Linux Foundation and its contributors.
1 Letterman Drive
Suite D4700
San Francisco, CA, 94129
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.

[TimidRobot]

Please update your original pull request (PR) instead of creating a new one:

• Add arXiv data fetching functionality #179

Also see:

• Foundational technologies — Creative Commons Open Source