Home/Microcenter-Monitor
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Implement resilient hybrid scraper for AMD and Intel bundles with SKU tracking #1

Merged
clanker merged 3 commits from feature/resilient-bundle-scraper into main 2026-09-13 21:01:50 +00:00
Conversation 0 Commits 3 Files changed 4 +367 -189
clanker commented 2026-09-13 20:54:20 +00:00
Member
Copy link

Overview

Resolves scraper breakages caused by Micro Center marketing redesigns and Cloudflare anti-bot challenges, and provides 100% bundle coverage across both AMD and Intel.

Key Changes

  • Hybrid Ingestion: Scrapes both promotional landing pages (bundle-and-save.aspx) and backend catalog listings (N=4294833320), de-duplicating by permanent SKU (5007xxx).
  • Semantic Invariant Parser: Eliminates brittle dependencies on transient marketing IDs (#BW2M, #cpu, h2.B) by anchoring on product link patterns (/product/5007xxx/), parent container traversing, and price/parts list selectors.
  • Intel & AMD Bundle Support: Full visibility into both AMD Ryzen and Intel Core / Core Ultra / Xeon bundles without relying on hidden UI tab clicks.
  • Cloudflare Bypass: Replaces blocked Chrome TLS profiles with Safari TLS impersonation (safari17_0 / safari15_5) in curl_cffi.
  • SKU-Keyed History & Database Migration: Adds sku column to price_history table for durable price drop tracking regardless of component title renamings.
  • CLI Enhancements: Added --list to inspect all live bundles, prices, and component lists without database writes, and --store to query specific local stores.
### Overview Resolves scraper breakages caused by Micro Center marketing redesigns and Cloudflare anti-bot challenges, and provides 100% bundle coverage across both AMD and Intel. ### Key Changes - **Hybrid Ingestion**: Scrapes both promotional landing pages (`bundle-and-save.aspx`) and backend catalog listings (`N=4294833320`), de-duplicating by permanent SKU (`5007xxx`). - **Semantic Invariant Parser**: Eliminates brittle dependencies on transient marketing IDs (`#BW2M`, `#cpu`, `h2.B`) by anchoring on product link patterns (`/product/5007xxx/`), parent container traversing, and price/parts list selectors. - **Intel & AMD Bundle Support**: Full visibility into both AMD Ryzen and Intel Core / Core Ultra / Xeon bundles without relying on hidden UI tab clicks. - **Cloudflare Bypass**: Replaces blocked Chrome TLS profiles with Safari TLS impersonation (`safari17_0` / `safari15_5`) in `curl_cffi`. - **SKU-Keyed History & Database Migration**: Adds `sku` column to `price_history` table for durable price drop tracking regardless of component title renamings. - **CLI Enhancements**: Added `--list` to inspect all live bundles, prices, and component lists without database writes, and `--store` to query specific local stores.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Home/Microcenter-Monitor!1
No description provided.