MedCrawler is a Python package that provides asynchronous interfaces for crawling medical literature databases. It currently supports crawling data from PubMed (via NCBI E-utilities) and ClinicalTrials.gov (via their API v2).
- Asynchronous HTTP requests for efficient data retrieval
- Built-in rate limiting and retry strategies with exponential backoff
- Caching with time-based expiration
- Batch processing capabilities
- Comprehensive error handling
- Date-based filtering for both PubMed and ClinicalTrials.gov
- Well-defined abstract interfaces for easy extension to other sources
pip install git+https://github.com/yourusername/MedCrawler.gitAdd the repository as a submodule to your project:
git submodule add https://github.com/yourusername/MedCrawler.git
git submodule update --init --recursiveInstall the package in editable mode:
pip install -e ./MedCrawlerimportasynciofromcrawlersimportPubMedCrawlerasyncdefmain():
asyncwithPubMedCrawler() ascrawler:
# Search for articlesasyncforpmidincrawler.search("cancer treatment", max_results=5):
# Fetch metadata for each articlemetadata=awaitcrawler.get_item(pmid)
print(f"Title: {metadata['title']}")
print(f"Authors: {', '.join(metadata['authors'])}")
print(f"Abstract: {metadata['abstract'][:100]}...")
print("\n"+"-"*50+"\n")
if__name__=="__main__":
asyncio.run(main())The package includes a demonstration script that showcases its functionality:
python main.py --source pubmed --query "diabetes" --max 10
python main.py --source clinicaltrials --query "covid" --max 5 --recentAvailable options:
--source:pubmed,clinicaltrials, orall(default:all)--query: Search query string (default:cancer)--max: Maximum number of results (default:5)--from-date: Start date for filtering results (format depends on source)--to-date: End date for filtering results (format depends on source)--recent: Short for setting from-date to 90 days ago
The BaseCrawler class provides core functionality used by all crawler implementations:
fromcrawlersimportCrawlerConfigfrommedcrawler.baseimportBaseCrawler# Create a custom configurationconfig=CrawlerConfig(
user_agent="YourApp/1.0",
email="your@email.com",
api_key="your-api-key", # Optionalmin_interval=0.5# Seconds between requests
)
# Use the crawler with the configurationasyncwithYourCrawler(config) ascrawler:
# Your code herefromcrawlersimportPubMedCrawlerasyncwithPubMedCrawler() ascrawler:
# Search with date filtering (YYYY/MM/DD format)asyncforpmidincrawler.search(
query="cancer treatment",
max_results=10,
from_date="2023/01/01",
to_date="2023/12/31"
):
metadata=awaitcrawler.get_item(pmid)
# Batch retrieval for efficiencypmids= ["12345678", "23456789", "34567890"]
results=awaitcrawler.get_items_batch(pmids)fromcrawlersimportClinicalTrialsCrawlerasyncwithClinicalTrialsCrawler() ascrawler:
# Search with date filtering (YYYY-MM-DD format)asyncfornct_idincrawler.search(
query="covid vaccine",
max_results=10,
from_date="2023-01-01",
to_date="2023-12-31"
):
metadata=awaitcrawler.get_item(nct_id)You can implement your own crawler by extending the BaseCrawler class:
frommedcrawler.baseimportBaseCrawlerfromtypingimportDict, Any, AsyncGenerator, Set, OptionalclassYourCrawler(BaseCrawler):
def__init__(self, config=None):
super().__init__("https://your-api-base-url.com", config)
asyncdefsearch(
self, query: str, max_results: Optional[int] =None,
old_item_ids: Optional[Set[str]] =None,
from_date: Optional[str] =None,
to_date: Optional[str] =None
) ->AsyncGenerator[str, None]:
# Implementation hereasyncdefget_metadata_request_params(self, item_id: str) ->Dict:
# Implementation hereasyncdefget_metadata_endpoint(self) ->str:
# Implementation heredefextract_metadata(self, response_data: Any) ->Dict[str, Any]:
# Implementation heremedcrawler/
├── __init__.py # Package version and exports
├── base.py # Base crawler implementation
├── pubmed.py # PubMed crawler
├── clinical_trials.py # ClinicalTrials.gov crawler
└── config.py # Configuration handling
pytestThis project follows PEP 8 style guidelines and uses:
- Black for code formatting
- isort for import sorting
- pytest for testing
- Fork the repository
- Create your feature branch (
git checkout -b feature/amazing-feature) - Commit your changes (
git commit -m 'Add amazing feature') - Push to the branch (
git push origin feature/amazing-feature) - Open a Pull Request