Skip to content

Add BWIKI lineup archives and avatar scraping - #6

Merged
AofeiLi-code merged 1 commit into
AofeiLi-code:mainfrom
bupt-lxc:feat/bwiki-lineups-avatars-squashed
May 9, 2026
Merged

Add BWIKI lineup archives and avatar scraping#6
AofeiLi-code merged 1 commit into
AofeiLi-code:mainfrom
bupt-lxc:feat/bwiki-lineups-avatars-squashed

Conversation

@bupt-lxc

Copy link
Copy Markdown
Contributor

Summary

  • Add lineup_scraper.py to archive BWIKI player lineups into data/lineups.json and data/lineups.csv, with incremental refresh based on list-page update dates.
  • Extend rocom_scraper.py to parse upload-lineup avatar images from the live BWIKI page, download them through the existing URL pipeline, and support --avatars-only / --skip-avatars modes.
  • Refresh generated data indexes, including data/urls.csv avatar entries and updated sprite/skill datasets.

Fixes / Safety

  • Preserve existing lineup output if all detail fetches fail, and record failed lineup/image URLs for retry.
  • Skip already-downloaded avatar images via the existing _add_url cache/path behavior.
  • Keep avatar assets separate from large sprite art under data/images/avatars, avoiding data/images/sprites mixing.
  • Add tracked avatar parser fixture so tests do not depend on ignored data/*.html samples.

Test Plan

  • python -m pytest tests/test_rocom_scraper.py tests/test_lineup_scraper.py -q -> 29 passed
  • python -m py_compile rocom_scraper.py lineup_scraper.py tests/test_rocom_scraper.py tests/test_lineup_scraper.py
  • python rocom_scraper.py --avatars-only -> parsed and saved 200 avatar URL rows, 0 failures

@AofeiLi-code
AofeiLi-code merged commit ab6658b into AofeiLi-code:mainMay 9, 2026
@AofeiLi-code

Copy link
Copy Markdown
Owner

Thank you for your contribution!

AofeiLi-code pushed a commit that referenced this pull request May 9, 2026
Sign up for freeto join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

@bupt-lxc@AofeiLi-code