Early iterations of PyCurator included web scraping functionality to gather additional metadata for Dataverse, Dryad, and OpenML objects, as well as metadata for objects in the UC Irvine Machine Learning Repository, which does not have an API at the time of PyCurator development.
Following extensive discussions, we have decided to remove all web scraping functionality from PyCurator. Rationale for removal:
-
Web scraping components are heavily dependent on web site format and prone to breaking. This is especially true for the UCI site, which is in beta during active PyCurator development. Excluding web scraping functionality for PyCurator makes it more stable and will require less just-in-time upkeep to handle web scraping edge cases. Additionally, limiting the potential for errors should create a more user-friendly experience for users.
-
Focusing PyCurator on data repositories with existing APIs is a more narrowly defined scope and limits feature creep. The primary purpose of PyCurator is to enable searching and returning object metadata across multiple repositories without the need to write code specific to each repository API. The target audience of PyCurator is data curation professionals, particularly in libraries, who may not have the time or Python skills needed to query multiple APIs. In cases where additional metadata of interest is only accessible in batch by web scraping, users can retrieve such metadata using tools other than PyCurator, or roll back this repository to the point where web scraping functionality was still enabled (with the understanding that such functionality may or may not work as intended, as websites may have updated in the interim).
Early iterations of PyCurator included web scraping functionality to gather additional metadata for Dataverse, Dryad, and OpenML objects, as well as metadata for objects in the UC Irvine Machine Learning Repository, which does not have an API at the time of PyCurator development.
Following extensive discussions, we have decided to remove all web scraping functionality from PyCurator. Rationale for removal:
Web scraping components are heavily dependent on web site format and prone to breaking. This is especially true for the UCI site, which is in beta during active PyCurator development. Excluding web scraping functionality for PyCurator makes it more stable and will require less just-in-time upkeep to handle web scraping edge cases. Additionally, limiting the potential for errors should create a more user-friendly experience for users.
Focusing PyCurator on data repositories with existing APIs is a more narrowly defined scope and limits feature creep. The primary purpose of PyCurator is to enable searching and returning object metadata across multiple repositories without the need to write code specific to each repository API. The target audience of PyCurator is data curation professionals, particularly in libraries, who may not have the time or Python skills needed to query multiple APIs. In cases where additional metadata of interest is only accessible in batch by web scraping, users can retrieve such metadata using tools other than PyCurator, or roll back this repository to the point where web scraping functionality was still enabled (with the understanding that such functionality may or may not work as intended, as websites may have updated in the interim).