Changelog

Fixed

• **Homepage crash** — `getStats()` wrapped in try/catch; Supabase connection failure no longer throws to root error boundary ("Something went wrong")
• **CSP blocking images** — added `https://img.gettcgsignal.com` and `https://cards.scryfall.io` to `connect-src` directive
• **Ticker 524 timeout** — added 8s `AbortController` on all price_history/exchange_rates queries; limit reduced 200→100
• **PriceTicker crash guard** — `if (!r.ok) return` before `.json()` prevents 524 HTML body from triggering silent parse error
• **Footer `/api/docs` 404** — corrected link to `/api-docs` (was 404ing on every page load via RSC prefetch)

Removed

• **Google AdSense** — removed `adsbygoogle.js` Script tag and `NEXT_PUBLIC_ADSENSE_CLIENT_ID` usage from layout (was blocked by CSP, causing console errors on every page)
• **Google Tag Manager** — removed GA Script tags from layout and stripped `googletagmanager`/`google-analytics` from CSP `script-src` and `connect-src`

---

Changed

• **Card images now served from R2 CDN** (`img.gettcgsignal.com`) — eliminates Vercel image optimization quota usage
• **`src/lib/image.ts`** (new) — `getCardImageUrl()` rewrites TCGPlayer CDN URLs to R2, upgrades `_200w→_400w` for widths >200px, passes Scryfall URLs through unchanged
• **`CardImage.tsx`** — `next/image <Image>` replaced with plain `<img>`, imports `getCardImageUrl()`
• **`CardImageZoom.tsx`** — same swap; `upgradeImageUrl` import removed
• **CSP `img-src`** — removed all `tcgplayer-cdn` / `tcgplayer.com` origins, added `https://img.gettcgsignal.com`
• **`next.config.mjs` `remotePatterns`** — cleared (no `next/image` usage for card images)

---

Fixed

• **Homepage crash** — `RegionalHeatmap` division guard: `weekAvg === 0` → `weekAvg <= 0` (catches near-zero floats)
• **Deals page exchange_rates query** — `updated_at` → `date` (column doesn't exist in exchange_rates)
• **Search results order** — `updated_at` → `last_updated` (correct price_summary column)
• **Market page dead RPCs** — removed `get_game_stats`/`get_platform_stats` calls (RPCs never existed, always fell through to slow fallback)
• **DealFilters hardcoded 3 platforms** — now dynamically built from `db-enums.ts` PLATFORMS (all 10 platforms visible)
• **Missing hareruya in PLATFORM_FEES** — hareruya deals were silently excluded from deal finder
• **Missing hareruya + tcgrepublic in PLATFORM_DISPLAY** — platform names showed raw slugs instead of labels
• **Deals empty state text** — updated from "TCGplayer, eBay, or Cardmarket" to generic "2+ platforms"

Changed

• **Middleware auth optimization** — skip `getUser()` for public routes (was running on every request)
• **Deck pricing API** — batch card name resolution with `.in('name', ...)` before per-name fallback (N+1 → batch)
• **Sets page** — added `.limit(2000)` safety cap on rarity query
• **WatchlistMovers** — added `.limit(100)` on watchlist query
• **Compare page** — parallelized card fetches with `Promise.all`
• **Game page** — merged two sequential `Promise.all` blocks into one
• **Collection page** — parallelized set count queries within batches
• **Centralized platform labels** — compare and market pages now use `platformLabel()` from `format.ts`

Added

• **YUYU-TEI One Piece** — `onepieceExtractor` + `YUYUTEI_OP_SET_MAP` (39 slugs: OP01-OP15, ST01-ST20, EB01-EB04, PRB01-PRB02, all probed 2026-03-20). Slug probe via `yuyu-tei.jp/sell/opc/s/{slug}` confirmed 200 for all except OP14 (local timeout, included for VPS). Collector number format: `OP01-024`. Direct CN match (no set_code filter — CN encodes set). `YUYUTEI_GAMES=pokemon-jp,yugioh,onepiece` on DO VPS.
• **TCG Republic scraper** (`vps-scraper/src/scrapers/tcgrepublic.ts`) — new JP price source. USD-native pricing (unique among JP sources — no JPY conversion). HTML scraper using cheerio, parses `li.product_thumbnail` elements. Dynamic subcategory discovery from category pages (`category_page_{id}.html`). Set codes extracted from bracket notation in subcategory titles (e.g. `[M4]` → `M4`). Pagination via `?p=N`. Circuit breaker (3 failures per game), 30-min timeout, Level 1 response cache.

- Games: `pokemon` (cat 35, pokemon-jp), `onepiece` (cat 67), `yugioh` (cat 34), `mtg` (cat 64, disabled by default)

- Condition: corner_ribbon `New` → NM, `Pre-Order` → skip, else → LP

- Matching: Pokemon (set_code + CN fraction), YGO (lang-swap), One Piece (direct CN)

- Dedup: by `card_id:condition`, keep lowest price

- Env: `TCGREPUBLIC_ENABLED`, `TCGREPUBLIC_GAMES` (default: `pokemon,onepiece`), `TCGREPUBLIC_MAX_SUBCATEGORIES_PER_GAME` (50), `TCGREPUBLIC_MAX_PAGES_PER_SUBCATEGORY` (10), `TCGREPUBLIC_REQUEST_DELAY_MS` (3000)

- Smoke test probe added to orchestrator

• **Migration 051** (`supabase/migrations/051_add_tcgrepublic_platform.sql`) — adds `tcgrepublic` to `price_summary` platform CHECK constraint. Updates `find_linked_deals` and `find_same_card_deals` RPCs to include `tcgrepublic` in `jp_platforms` array.
• **Platform integrations**: `db-enums.ts`, `fees.ts`, `shipping.ts`, `source-diversity.ts`, `cross-region-deals.ts`, `vps-scraper/src/cross-region.ts` all updated.

Research findings (probe 2026-03-20)

• 246 Pokemon subcategories with bracket codes (e.g. `[M4]`, `[SV10]`); 209+ products per subcategory
• YGO: bracket codes match OCG collector number prefixes (e.g. `[PAC1]` → `PAC1-JP012`)
• One Piece: `[OP-01]` bracket but CN is `OP01-120` (no hyphen — matches DB format directly)
• robots.txt: category pages allowed; inventory/cart/watchlist disallowed
• Pricing in USD (no exchange rate needed — unique JP source)

Deployed (2026-03-20)

• Migration 051 applied, `ANALYZE price_summary;` done
• DO VPS: `TCGREPUBLIC_ENABLED=true`, `TCGREPUBLIC_GAMES=pokemon,onepiece`, `YUYUTEI_GAMES=pokemon-jp,yugioh,onepiece`
• YUYU-TEI One Piece dry-run: 2,654 parsed, 2,505 matched (94%)
• TCGRepublic first live run: **3,968 rows, 0 errors** — Pokemon 68% (1,321/1,943), One Piece 94.7% (2,647/2,796)
• Hetzner: git pull + build (no TCGRepublic needed)

Stats

• Tests: 903 across 38 files (+36 new: 8 YUYU-TEI One Piece + 28 TCG Republic)
• Migration: 051
• Data sources: 10 registered + 4 VPS-only (bigweb, surugaya, hareruya, **tcgrepublic**)

---

Added / Changed

• **YUYU-TEI YGO set map** (`YUYUTEI_YGO_SET_MAP` in `yuyutei.ts`) — 47 slugs confirmed via slug probe from DO VPS (2026-03-20). Covers OCG sets 2014-2025: major sets (LEDE, LOCH, AGOV, DUNE, CYAC, PHHY, DABL, DIFO, POTE, BLZD, DAMA, BODE, IGAS, ETCO, ROTD), premium packs (PRB, PRE, WPP1-WPP6, TTP1, CP17-CP20), classic sets (SAST, SOFU, FLOD, INOV, RATE, CORE, SECE, NECH, DUEA), and structure decks SD30-SD48.
• **MTG noted as absent** — YUYU-TEI does not carry MTG (no `/top/mtg` page). `YUYUTEI_MTG_SET_MAP` stays empty with explanatory comment.
• **DO VPS**: `YUYUTEI_GAMES=pokemon-jp,yugioh` added to `.env`.

Verified

• YGO dry-run: `lede` set → 81 parsed, 81 matched (100%). Parser + collector number format confirmed working.
• TCGdex backfill dry-run: 3,280 cards would be inserted across 98 mapped sets.

---

Added

• **Migration 050** (`supabase/migrations/050_catalog_backfill_columns.sql`) — adds `catalog_source` and `ygo_passcode` columns to `cards` table. Indexes on both (partial, WHERE NOT NULL). Apply in Supabase SQL Editor, then run `ANALYZE cards;`.
• **`vps-scraper/src/scripts/catalog-backfill.ts`** — standalone backfill script. Fetches all JP sets from TCGdex API (`https://api.tcgdex.net/v2/ja/`), maps set IDs to DB set_codes via case-insensitive match against known Pokemon-JP codes (mirroring YUYU-TEI set map). Inserts missing cards with `catalog_source='tcgdex'`, updates `name_jp` on existing cards where NULL. Rate limited at 100ms between API calls.
• **`vps-scraper/src/scripts/tcgdex-utils.ts`** — pure utility functions for testability: `mapTcgdexSetId()`, `parseTcgdexCard()`, and TypeScript types `TcgdexSetSummary`, `TcgdexSet`, `TcgdexCard`, `CardInsert`.
• **`catalog-backfill` npm script** in `vps-scraper/package.json`
• **CLI safety:** Non-dry-run execution requires explicit `--confirm` flag. `--rollback --confirm` deletes all `catalog_source='tcgdex'` rows. `--set sv1s` for single-set targeting.
• **Tests:** `mapTcgdexSetId` (7 tests: case-insensitive match, null for unknown, no uppercase guessing), `parseTcgdexCard` (5 tests: valid insert, null for empty localId/name, set_code from argument).

Owner Action Required

1. Apply migration 050 in Supabase SQL Editor

2. `ANALYZE cards;`

3. Deploy to DO VPS

4. Dry-run: `node dist/scripts/catalog-backfill.js --source tcgdex --game pokemon-jp --dry-run`

5. Production: `node dist/scripts/catalog-backfill.js --source tcgdex --game pokemon-jp --confirm`

Stats

• Tests: 867 across 38 files (+38 new tests: 12 YUYU-TEI multi-game + 12 TCGdex utils)
• Migration: 050

---

Added

• **YUYU-TEI multi-game scraper** (`vps-scraper/src/scrapers/yuyutei.ts`) — refactored from single-game (Pokemon-JP only) to game-loop architecture following BigWeb pattern.

- `YUYUTEI_GAMES` env var (default: `pokemon-jp`) — comma-separated list: `pokemon-jp,mtg,yugioh`

- `YUYUTEI_MAX_SETS_PER_GAME` (default: `50`) — cap sets per game per run

- `YUYUTEI_MAX_REQUESTS` (default: `0` = unlimited) — global HTTP request cap

- Per-game circuit breaker (3 failures → skip to next game, not abort run)

- MTG/YGO set maps start **empty** — zero risk to existing Pokemon scraping

• **Collector number extractors** (exported for testing):

- `pokemonExtractor` — full fraction "025/414" format

- `ygoExtractor` — OCG format "SAST-JP067"

- `mtgExtractor` — fraction-first, bare-number fallback

• **Per-game DB matching:**

- Pokemon-JP: `set_code + collector_number` (unchanged)

- MTG: `set_code (lowercase) + collector_number` (Scryfall-style codes)

- YGO: `collector_number` only + language-swap (SAST-JP067 → SAST-EN067) + strip fallback (same strategy as BigWeb)

• **`parseSetPage` refactored** — optional `gameConfig` param (backward compatible, defaults to Pokemon behavior)
• **Unmatched card logging** — `/var/log/tcgsignal-cache/yuyutei-unmatched.json` with per-game counts, top 10 unmatched slugs, 5 sample cards
• **Telegram per-game breakdown** — `validation.perGameBreakdown` populated (e.g. `pokemon-jp: 5200/5400 (96%) P0:5200`)
• **Tests** (new describe blocks):

- "YUYU-TEI collector number extractors" — 18 unit tests for `pokemonExtractor`, `ygoExtractor`, `mtgExtractor`

- "YUYU-TEI parseSetPage with YGO config" — 3 tests with OCG-format HTML fixtures

- "YUYU-TEI parseSetPage with MTG config" — 3 tests with fraction/bare-number fixtures

Notes

• Pokemon scraping is **unchanged** — existing behavior, same match rates
• MTG/YGO activation: populate set maps in code after slug probe, then set `YUYUTEI_GAMES=pokemon-jp,yugioh,mtg` on DO VPS
• For MTG: dry-run first before populating map — if 0 cards parsed, `mtgExtractor` may need adjustment for actual YUYU-TEI HTML format

Stats

• Tests: 867 across 38 files (829 → 867, +38)
• No migration needed

---

Fixed

• **`db-health-check.ts`** — Health report crash when `run_health_check_storage` fails (migration 049 not yet applied): `h.row_counts` and `h.retention_leak` were accessed without null guards, crashing `formatHealthReport` and suppressing both the health AND coverage Telegram reports.
• **Coverage report not sent** — `formatHealthReport` and `formatCoverageReport` were called sequentially in a shared try/catch; a crash in the former suppressed the latter. Now wrapped in independent try/catch blocks — coverage report sends even if health report formatting fails.
• **Unguarded `.length` accesses** — Added `?.` optional chaining to all unguarded array `.length` checks and iteration in `formatHealthReport`: `cards_per_game`, `card_coverage`, `price_by_platform`, `staleness`, `conditions_by_platform`, `cross_region_by_game`, `card_links`, `zero_prices`, `high_prices`, `sync_freshness`, `exchange_rates`. Guards `h.cross_region` for crossregion RPC failure.

Note

• Permanent fix for storage RPC: apply migration 049 in Supabase SQL Editor (owner action). The code hardening here is a belt-and-suspenders guard for any future RPC failure.

Stats

• Tests: 829 across 38 files (unchanged)
• No migration needed (scraper-only)

---

Fixed

• **`run_health_check_storage` RPC** — `records_written` → `records_synced` (actual column name from migration 011). Also fixed SQL aliases to match TypeScript `WriteVolume` interface: `avg_daily` → `avg_daily_writes`, added `total_records_7d`. Was causing health report write-volume section to crash/show blank. Migration 049.
• **`find_linked_deals` RPC** — Table-qualified `link_card_ids` CTE (`cl.card_id_a`, `cl.card_id_b`) to fix `card_id_a` ambiguity with the `RETURNS TABLE` output parameter. Was causing the linked deals section to fail. Migration 049.
• **Cross-region junk deals** — Added `min_cheap_price` parameter (default `$0.50`) to both `find_linked_deals` and `find_same_card_deals` RPCs. Previously a deal with €0.02 EU → $5 US would pass because the expensive-side check (`>= $2.00`) was satisfied. Both client-side fallbacks updated with matching `if (low < 0.50) continue` guard.
• **Hareruya missing from cross-region** — Added `hareruya: 'JP'` to `PLATFORM_REGION` map in both `vps-scraper/src/cross-region.ts` and `src/lib/cross-region-deals.ts`. Was causing hareruya prices to fall through to `'US'` default, suppressing all JP→US deals from hareruya since v0.98.0.
• **`find_same_card_deals` timeout** — Restored `idx_price_summary_platform_condition_card_price` (dropped in migration 048). Without it the 3 per-region CTEs do full price_summary scans on free tier, hitting statement timeout. Migration 049.

Hardened

• **`db-health-check.ts`** — Added `?.` optional chaining on `table_sizes`, `history_dates`, `dead_tuples`, `write_volume` in `formatHealthReport()`. Prevents cascade crash if the storage RPC fails for any reason.
• **`tcgplayer-card-map.ts`** — Retry failed pagination page once before breaking (silent partial map was the worst-case). Warn if map < 80K entries. Skip cache save if map < 50K (prevents poisoning 7-day cache with a corrupted partial map).

Docs

• Added Standing Rule 11: version numbering (`v0.10x.0` format, not `v1.x.x`)
• Fixed version `1.0.2` → `0.106.0` in `VERSION` and `package.json`

Stats

• Tests: 829 across 38 files (unchanged)
• Migration: 049 applied (index restored + 3 RPC fixes)

---

Maintenance

• **Deleted `tests/db-indexes.test.ts`** (27 tests) — 100% structural: read SQL migration files and checked for index pattern strings. No functional coverage; Standing Rule 8 violation.
• **Deleted `tests/tcgcsv.test.ts`** (16 tests) — merged into `tests/vps-scraper.test.ts` where it belongs (same domain: VPS scraper data source layer).
• **Rewrote `tests/vps-scraper.test.ts`** — removed ~350 `src()` anti-pattern tests that called `fs.readFileSync()` on source files and asserted `expect(code).toContain('string')`. Kept all fixture-based and function-call tests (241 tests remaining). Standing Rule 8 compliance.

Stats

• Tests: 829 across 38 files (was 1234/40 — net reduction of 405 structural/grep tests)

---

Performance

• **Bulk preload price history** — `preloadPriceHistory(supabase, platform, date)` in `history.ts` paginates all yesterday NM rows for a platform into memory before the group/set loop. Eliminates ~2600 per-group DB round trips in tcgcsv + tcgtracking (~8 MB/day saved).
• **Bulk preload tcgplayer card map** — `loadTcgplayerCardMap(supabase)` in `scrapers/tcgplayer-card-map.ts` paginates all `cards WHERE tcgplayer_id IS NOT NULL` once at startup. Disk-cached (7-day TTL, invalidated on new cards). Eliminates ~1790 per-group/per-set DB queries (~7 MB/day saved).
• **`filterChangedPrices` backward-compatible** — new optional `preloaded?: PreloadedHistory` param; legacy callers (cardtrader) are unaffected.
• **Expected savings** — DO VPS sync egress: ~35 MB/day → ~20 MB/day after these two fixes + migration 047 (cross-region RPC).

Stats

• Tests: 1234 across 40 files

---

Fixed

• **Cross-region RPC timeout** — `find_linked_deals` rewritten to replace 46K correlated subqueries (6 per card_link × 7719 links) with a single JOIN-based scan of the ~15K relevant card_ids. `find_same_card_deals` updated to include `hareruya` in JP platforms (was missing since v0.98.0). Migration 047.
• **Index cleanup** — `idx_price_summary_platform_condition_card_price` added in 047 then dropped in 048. Existing `idx_price_summary_condition_card_price (condition, card_id, market_price)` covers the same queries adequately; platform-first index saved ~3× scan reads but was not load-bearing for the timeout fix and cost ~10–15 MB at 421/500 MB free tier.

Stats

• Tests: 1220 across 40 files

---

Fixed

• **Telegram report HTML parse failure** — Error messages and card names from scrapers are now escaped (`escHtml()`) before embedding in Telegram HTML-mode messages. Raw HTTP error responses (e.g. `<!DOCTYPE html>`) in scraper errors no longer break the daily report.
• **Migration 046 Postgres compat** — `ORDER BY` moved inside subquery `t` instead of on `jsonb_agg()` to avoid syntax errors on Supabase's Postgres version.
• **Health check leak detection** — `run_health_check_storage` RPC used 2-day threshold but tiered retention keeps $5+ cards for 7 days. Updated to `interval '7 days'` to match actual policy (migration 046). No actual data leak — 141K flagged rows were legitimate.
• **Supabase client refresh** — `resetSupabaseClient()` called before cross-region and health check stages to prevent stale connections after 5+ hour full-sync runs.
• **Cache dir unbounded growth** — daily cron now cleans response cache files (`.json`, `.html`) older than 3 days, capping ~180 MB max.
• **Orphan purge on fresh DB** — Identified blind spot: MTG orphan purge deletes all MTG cards with no prices on a fresh DB (before prices are written). Tracked in backlog for code fix.

Ops

• **New Supabase DB migration** — Switched to new DB after lockout. Full data re-seed: 253K cards (MTG 152K, YGO 38K, Pokemon 28K), 285K+ price rows. All scrapers updated on DO + Hetzner VPS.
• **Migration 046 applied** to new DB via SQL editor.

Stats

• Tests: 1220 across 40 files

---

Added

• **`get_top_movers` RPC** — server-side price change ranking (migration 045)
• **`get_game_market_stats` RPC** — aggregate game market statistics (migration 045)
• **Consolidated `getTopMoversForGame()`** — single source of truth for trending logic with RPC-first + fallback
• **Price floor enforcement**: $5+ for 7d period, $1+ for 24h (matches tiered retention)
• **`src/lib/price-trends.ts`** — tier-aware `getCardPriceTrend()` and `getGameMarketStats()`
• **Card detail page** — tier-aware trend display (7d for $5+, 24h for $1-5, hidden for sub-$1)
• **Game hub** — Market Overview section with total market value, avg/median price, cards tracked

Changed

• **Removed 30d period option** from trending page (only 7-day max retention available)
• **Trending page** uses shared `getTopMoversForGame()` (removed ~170 lines of inline duplicate)

Stats

• Tests: 1219 across 40 files

---

Added

• **`game` column on `data_source_sync_log`** for per-game match rate tracking (migration 044)
• **Coverage RPC updated** to GROUP BY source_id, game
• **All scrapers now report per-game results** (perGameResults on multi-game scrapers, game field on single-game)
• **Card map cache** (Egress Phase 3) — `vps-scraper/src/card-map-cache.ts` with 7-day TTL + incremental refresh
• **tcgcsv + cardtrader** use cache-first pattern for card map loading
• **`--no-cache` CLI flag** to force fresh card map fetch
• **Health check** displays per-game match rates in Telegram report

Stats

• Tests: 1219 across 40 files

---

Added

• **Source diversity RPCs** (migration 043): 4 server-side Postgres RPCs replace heavy client-side `price_summary` scans in `source-diversity.ts`. `checkOriginDiversity`, `checkConditionCoverage`, `checkStaleness`, and `checkRegionalCoverage` now aggregate in Postgres and return ~1 KB each instead of scanning 350K+ rows
• **RPC fallback**: All 4 functions try RPC first, fall back to existing client-side queries if RPCs unavailable (same pattern as cross-region RPCs in v0.97.1)
• **Hareruya regional mapping**: Added `hareruya: 'JP'` to `PLATFORM_REGIONS` in source-diversity checks
• **Source diversity tests** (`tests/source-diversity.test.ts`): 16 tests covering RPC response mapping, fallback behavior, and null client handling

Changed

• **Egress reduction**: Estimated savings ~200-400 MB/day (~6-12 GB/month) once migration 043 is applied. `checkConditionCoverage` and `checkStaleness` were full-table scans — worst offenders

---

Added

• **BigWeb per-game match rate breakdown**: Telegram report now shows matched/fetched per game (Pokemon, MTG, YGO, One Piece) with match percentages
• **BigWeb match-tier logging**: Tracks which matching tier (P0–P4, LangSwap, Name) matched each card. Breakdown shown in Telegram report per game
• **BigWeb unmatched card logging**: Unmatched products collected per run with game, set name, card name, collector number. Top 10 unmatched sets and 5 sample cards shown in Telegram. Full list written to `/var/log/tcgsignal-cache/bigweb-unmatched.json`
• **YGO language swap utility** (`vps-scraper/src/utils/ygo-language-swap.ts`): Shared utility for OCG→TCG collector number conversion (`-JP`→`-EN`). Used by BigWeb and Suruga-ya scrapers
• **Suruga-ya YGO support**: Added category `501080040` (YGO) and `501080047` (vintage Pokemon) to `SURUGAYA_CATEGORY_GAME_MAP`. YGO cards matched via language swap (`-JP`→`-EN`) + stripped language code fallback

Changed

• **BigWeb YGO matching**: Now uses `swapJpToEn()` from shared utility + `stripLanguageCode()` fallback for modern sets. JP-exclusive sets (Structure Decks, Premium Packs) remain unmatched as expected
• **BigWeb `mapYgoJpToEn()`**: Now delegates to shared `swapJpToEn()`. Kept for backward compatibility
• **Suruga-ya batch card lookup**: YGO categories include EN-mapped and stripped collector numbers in the batch query for cross-region matching

---

Added

• **Hareruya scraper** (`vps-scraper/src/scrapers/hareruya.ts`): New JSON API scraper for Hareruya MTG EN retail inventory. Clean API, no auth, no Cloudflare. DO Singapore VPS, post-vercel phase.

- 57K+ EN retail cards, full catalog daily (~19 min at 2s delay)

- NM condition only, foils skipped, consignment items filtered

- Collector number + set_code matching (card-format-guide priority)

- Exchange rate inversion (USD→JPY stored, inverted for JPY→USD)

- Circuit breaker (3 consecutive failures), 30-min timeout guard

- Response caching (Level 1), dedup by card_id:condition (lowest price)

- 500-row batch upserts, `data_source='hareruya_api'`

- ScraperValidation metrics, smoke test probe registered

• **Migration 042**: Add `'hareruya'` to price_summary platform CHECK constraint
• **db-enums.ts**: `'hareruya'` added to PLATFORMS array
• **REGION_MAP**: `hareruya: 'JP'` for Telegram daily report
• **Test fixture**: `tests/fixtures/hareruya-page.json` with 8 sample products

Fixed

• **Hareruya API response parsing**: API returns Solr-style `{ response: { numFound, docs } }` not `{ total, items }`. Scraper was hitting circuit breaker on every run due to `items` being undefined.
• **Hareruya pagination**: API ignores `start` offset param — uses `page=N` (1-based). Was processing the same 100 items on every page.
• **Hareruya collector number leading zeros**: Strip `"021"` → `"21"` to match DB format.
• **Hareruya 1000-row Supabase limit**: Query cards per set_code instead of bulk `IN(collector_number)` to avoid cap on common numbers.

Deployed

• Migration 042 applied ✅
• DO VPS .env configured ✅ (`HARERUYA_ENABLED=true`, `HARERUYA_MAX_PAGES=600`, `HARERUYA_REQUEST_DELAY_MS=2000`)
• First full run: 9.4K valid NM non-foil → 6.4K matched → 1,216 unique cards upserted (68% match rate)

---

Fixed

• **Retention cleanup egress**: Replaced 5 SELECT-then-DELETE loops with direct `.delete()` calls for MP/HP/DMG, tcgtracking non-NM, cheap stale, stale 30d, and zero-price purges. Saves ~0.2-0.3 GB/month Supabase egress.
• **Cardmarket N+1 queries**: Replaced per-card `.single()` set_code lookups with batched `.in('id', [...])` query. Saves ~0.03-0.09 GB/month.

Added

• **Card-link skip stable sets**: Sets with >=90% EN cards already linked skip JP card queries entirely. Saves ~0.06-0.09 GB/month.
• **Cross-region RPCs (migration 041)**: Two server-side PL/pgSQL functions (`find_linked_deals`, `find_same_card_deals`) compute deals in Postgres. Returns only ~50-200 deal rows instead of full table scans (~8-15 MB). Saves ~0.3-0.6 GB/month. Both VPS and Vercel callers try RPCs first, fall back to client-side logic if migration not applied.
• **Egress reduction plan**: Full plan with remaining phases saved to `docs/egress-reduction-plan.md`.

Owner Action Required

• Apply migration 041 in Supabase SQL Editor, then run `ANALYZE price_summary; ANALYZE card_links; ANALYZE cards;`

---

Fixed

• **Cardmarket matching order**: Collector number + set_code now runs before name matching (per card-format-guide priority). Prevents false matches like Underground Sea Revised → wrong expansion.
• **Expansion-size validation**: New safety net rejects expansion mappings where CM expansion size vs DB set size ratio is >5x or <0.1x, preventing wrong-expansion assignments.

Added

• **BigWeb page count logging**: First page fetch per game now logs total available pages and estimated product count for observability.
• **YUYU-TEI SM expansion**: Set map expanded from 69 → 95 sets — all SM sub-sets (SM1-SM12) confirmed via HTTP probe.

Changed (env/infra)

• **DO VPS**: `BIGWEB_MAX_PAGES` 80 → 250 (MTG full cycle: 33 → 11 days)
• **Hetzner VPS**: Added `SURUGAYA_CATEGORIES=501080033,501080034` for MTG
• **Cron**: Hetzner 17:00 → 14:00 UTC (22:00 SGT); DO stays at 20:30 UTC (04:30 SGT)

Owner Action Required

• **Fix 1C**: After first DO cron completes post-deploy (~21:30 SGT), run MTG cardmarket_product_id purge SQL (see plan)

---

Fixed

• **CardTrader `recordsFetched`**: Now counts unique blueprints instead of 14.7M seller listings, giving accurate match rate percentages in coverage report.
• **Cross-region "?" card names**: Deals with missing card names (orphan `card_links` refs) are now skipped instead of showing "?" in Telegram.
• **Removed 999% gap cap**: Cross-region deals show real price gaps. One-time SQL cleanup deleted pre-ceiling CardTrader outliers (>$5K) from `price_summary` and `price_history`.
• **eBay freshness**: Platforms with fewer than 50 rows show "sparse" instead of misleading 0% fresh in health report.

Changed

• **PriceCharting backlog**: Expanded with specific revisit triggers (revenue, graded pricing, eBay API).

---

Fixed

• **Pokemon-JP label**: Coverage report showed "Pokemon" for both `pokemon` and `pokemon-jp` in deals/history lines. Now displays "PKM-JP" for `pokemon-jp`.
• **Lorcana in history**: Lorcana appeared in the HISTORY line but was excluded from Game Health, Deals, and Cross-Region. Now filtered from history for consistency.
• **Grammar**: "1 days" → "1 day" in history line.

Added

• **Price ceiling ($5K)**: CardTrader prices above $5,000 USD are skipped at scrape time with a warning log. Garbage data (e.g. $44K Lord of Atlantis, $9K Yaranzo) was inflating deal counts and suspicious gaps. Migration 039 adds the same ceiling to `deal_candidates` and `suspicious_gaps` RPCs to filter historical bad data.
• **Backlog**: CardTrader One Piece coverage gap flagged — OP has 0 EU cards because CardTrader doesn't scrape it yet.

---

Added

• **Migration 035**: 2 new indexes (`idx_price_summary_condition_card_price` composite, `idx_price_summary_price_5plus` partial) + 3 new RPCs replacing `coverage_deals`.
• **3 split RPCs**: `run_health_check_price_tiers` (MAX GROUP BY, no DISTINCT ON), `run_health_check_cross_region_5plus` (partial index pre-filter), `run_health_check_deal_candidates` (LP-aware condition logic). Each completes within 55s timeout.
• **BigWeb YGO**: Game 9 added to `BIGWEB_GAMES` on DO VPS (`170,177,1,9`).

Fixed

• **TCGCSV circuit breaker**: Two-tier pattern — backoff at 5 failures (10s→30s), hard stop at 15. Was: global `break` after 5 failures killed all remaining groups (76.9% freshness).
• **Coverage deals timeout**: `run_health_check_coverage_deals` exceeded 55s on free tier. Cross-region $5+ overlap, price tiers, and deal candidates now report via dedicated RPCs.
• **Score denominator**: All coverage_deals sections now have dedicated RPCs, score denominator is stable across runs.
• **Exchange rate false alarm**: Health check showed rates 82-418h stale because it reported ALL historical rows and used business day `date` (Friday on Monday = 72h). Migration 037: cleanup old rows, only show latest per currency pair. Threshold raised from 48h to 96h (weekend buffer).

Changed

• **Health check total**: Now 9 RPCs (was 6): storage, cards, pricing, crossregion, coverage, game_health, price_tiers, cross_region_5plus, deal_candidates.
• **Migration 036**: Split `game_health` from coverage RPC (was causing timeout). Rewrote `price_tiers` without LEFT JOIN on 304K cards table. Dropped old `run_health_check_coverage_deals()`.
• **`--source db-health-check`**: New CLI shortcut to run health check standalone.

---

Added

• **Coverage health check RPCs**: 2 new Supabase RPCs — `run_health_check_coverage` (maturity scoring with 20 platform×game targets) and `run_health_check_coverage_deals` (linked + same-card deal counts by game). Migration 034.
• **Maturity scoring**: Each of the 20 targets defines expected card counts (e.g., tcgplayer/pokemon = 10K). Coverage RPC compares actual vs target, computes maturity percentage.
• **`records_fetched` in sync log**: All scrapers (tcgcsv, cardtrader, tcgtracking, yuyutei, bigweb, surugaya, cardmarket) now report total records fetched from source before card matching. VPS orchestrator writes this to `sync_log`.
• **Telegram coverage report**: Coverage maturity and deal counts included in daily Telegram health check message.

Fixed

• **Health check RPCs**: Fixed all 4 existing RPCs — resolved nested aggregate errors, added statement timeout guards, corrected column name mismatches.

Changed

• **Health check total**: Now 6 RPCs (was 4): storage, cards, pricing, crossregion, coverage, coverage_deals.
• **VPS orchestrator**: Writes sync log entries with `records_fetched` counts for all scraper runs.

---

Added

• **Design system**: CSS custom properties for colors (navy backgrounds, green/red signals), fonts (Sora sans, Geist Mono for prices).
• **Homepage redesign**: Price ticker, alert banner, top arbitrage deals, biggest movers, featured comparison, regional heatmap.
• **Ticker API**: `/api/homepage/ticker` — top 5 price movers + EUR/USD, JPY/USD rates (1h cache).
• **Homepage components**: PriceTicker, TopArbitrage, BiggestMovers, AlertBanner, FeaturedComparison, RegionalHeatmap in `src/components/features/homepage/`.

Changed

• **Global theme**: Dark navy backgrounds (#0c0f14), Sora font for text, Geist Mono for prices/stats. Replaces Inter + gray-950 scheme.
• **Navigation**: Simplified — removed GameMegaMenu and Events link. Added Movers and Alerts. Logo styled as TCG (white) + Signal (green). Sign In CTA now green.
• **Search results**: Larger thumbnails (64x90), game badges, themed panels/borders, green pagination.
• **Card detail page**: Navy panels, mono prices, themed related cards, green accent links.
• **Footer**: Updated links (Movers, Regions), branded logo, trading terminal messaging.
• **All pages**: Bulk theme sweep — 100+ files updated from gray-950/gray-900/gray-800 to CSS variable system.
• **CTA buttons**: Blue-600 → green throughout.
• **InlinePriceComparison**: Green-soft "Best" badge, themed borders.

Refactored (post-release cleanup)

• **AlertBanner**: replaced 75-line custom spike query with `getTopMoversForGame()` from `trending.ts`
• **FeaturedComparison**: replaced 112-line custom cross-region query with `findSameCardDeals()` from `cross-region-deals.ts`
• **BiggestMovers**: extracted `MoverList` sub-component to DRY duplicate gainers/losers JSX
• **RegionalHeatmap**: parallelized 28 sequential DB queries into nested `Promise.all` (games × regions concurrent)
• **PriceTicker**: added division-by-zero guard on exchange rate inversion
• **page.tsx**: fixed `totalCards` stat (was duplicating `totalPrices` from same query); removed unnecessary `<div>` wrapper
• **MobileNav**: fixed unread alerts badge — was checking AUTH_LINKS for `/alerts` which lives in PUBLIC_LINKS

Stats

• Tests: 1034 passing across 39 files
• No new DB tables or migrations — frontend only
• Net: -181 lines from refactoring (reuse existing lib functions)

Fixed

• **Homepage stats**: Card count now shows priced cards (price_summary with market_price > 0) instead of all cards in DB.
• **$0.00 NM prices in search**: Zero-price entries excluded from NM cheapest calculation and allPrices map.
• **Profit Calculator default**: Sell platform now defaults to TCGplayer instead of eBay.
• **Price history empty state**: Shows "$5+ threshold" message for sub-$5 cards; "after next daily sync" for new cards with no data yet.
• **Game hub garbage deals**: ROI capped at 999% in `getTopDealsForGame()`, minimum price raised from $0.50 to $2.00.
• **Autocomplete race condition**: Search form submit now sets a `submittedRef` flag that prevents debounced autocomplete from showing dropdown after navigation.

Added

• **UI_GAME_SLUGS**: New enum in `db-enums.ts` excluding games with no data (Lorcana). `USER_GAME_SLUGS` preserved for DB compatibility.
• **Sealed product filter**: Search results filter out sealed products (booster boxes, ETBs, theme decks, etc.) by name matching.

Changed

• **Lorcana removed from UI**: Game hubs, footer, homepage, search, sets, decks, tournaments, calendar, help, and API docs no longer show Lorcana. DB enum unchanged.
• **Game hub empty states**: Trending and sets sections hidden entirely when no data exists, instead of showing "No data yet" shells.

Stats

• Tests: 1034 passing across 39 files

Added

• **Search relevance scoring**: New `src/lib/search-relevance.ts` with 5-factor composite scoring (price, freshness, set era, source diversity, rarity) plus exact-match bonus. Log-scaled price curve spreads $1–$50 range. Freshness uses `price_summary.updated_at` so reprints rank correctly.
• **Default sort = relevance**: Search results now default to relevance sort instead of popularity. Users get the best match first, not the most listed.
• **Relevance pool fetch**: Relevance sort fetches top 200 cards (vs 24) for client-side scoring and pagination. Pre-scored pool means pages 2+ are instant.
• **Pagination indicator**: "Showing top N results ranked by relevance" replaces generic count when relevance sort is active.
• **Tests**: 1034 (44 new: price scoring, freshness tiers, set era mapping, source diversity, rarity scoring, exact match, integration, weight validation)

Changed

• **SortSelect**: Relevance is now first option and default when a search query is present. Popular remains default for browsing without a query.
• **Scoring weights**: Configurable via `WEIGHTS` const — price 0.35, freshness 0.15, set era 0.15, source count 0.2, rarity 0.15.

Removed

• **Monolithic health check fallback**: Removed `run_db_health_check()` fallback from VPS orchestrator. Migration 033 drops the function from Supabase. Split RPCs (032) are the sole path now.

Fixed

• **Cross-region minPrice filter**: `findLinkedDeals()` now filters out sub-$2 cards (matching `findSameCardDeals()` pattern). Eliminates $0.01 junk deals.
• **Gap cap at 999%**: Cross-region gap percentages capped at 999% in both VPS and Next.js. Prevents misleading 54,825% gaps from near-zero prices.
• **Error logging**: Supabase query errors in cross-region card/price lookups now logged instead of silently swallowed.

Added

• **Orphan card purge**: Daily retention deletes MTG cards with no price_summary rows and no FK refs (card_links, portfolio). ~27K orphans from old catalog imports. Other games excluded (BigWeb/YUYU-TEI match rates still improving).
• **Zero-price purge**: Removes price_summary rows with market_price = 0 or NULL (~4,338 rows).
• **Tiered stale retention**: Sub-$1 price_summary rows deleted after 14 days (vs 30 days for $1+ cards). Targets long-tail bulk from rotated sets.
• **Split health check RPCs**: Migration 032 splits monolithic `run_db_health_check()` into 4 functions (storage, cards, pricing, crossregion) — each under 30s timeout. Uses `reltuples` estimates for row counts. Fallback to old RPC if migration not yet applied.
• **Tests**: 990 (18 new: cross-region minPrice, gap cap, error logging, orphan purge, zero-price, tiered stale, health check split)

Owner Action

• ~~Run migration 032 in Supabase SQL Editor~~ ✅
• ~~Run migration 033 in Supabase SQL Editor (drops monolithic run_db_health_check)~~ ✅
• Add `PURGE_MAX_BATCHES=200` to DO VPS `.env` for first run, then remove
• **Decision needed**: Supabase paid tier (~4-5 weeks until 500 MB, see #8a in owner-action-items.md)

Added

• **Sub-$1 history purge**: Daily retention now deletes legacy `price_history` rows with `avg_price < 1.0` — safety net for rows written before v0.90.0 write filter.

Changed

• **Stale cleanup capacity**: Batch limit increased from 10 → 50 iterations (5K → 25K rows/run) to prevent cleanup falling behind accumulation.

Owner Action

• **URGENT**: Run emergency SQL in Supabase SQL Editor to delete legacy sub-$1 history rows and reclaim space. See `docs/owner-action-items.md` top section. DB at 512 MB / 500 MB — may go read-only.

Added

• **LP in same-card deals**: `findSameCardDeals()` now queries JP platforms (yuyutei, bigweb, surugaya) with NM+LP conditions. Compares all three region pairs (US↔EU, US↔JP, EU↔JP). US/EU queries stay NM-only. Keeps best gap per card.
• **BigWeb Pokemon match rate improvements**: Expanded `BIGWEB_POKEMON_SET_MAP` from 30 to 85+ entries (SV, SWSH, SM, XY, BW eras). P0 tier: Pokemon `cardset.slip` direct match. P3 tiebreaker prefers known set codes.
• **BigWeb YGO name fallback**: `name_jp`-based matching for unmatched YGO products when collector number fails. Queries cards by JP name with dedup.
• **Cardmarket S3 larger samples**: Sample size increased from 5 to 15 for non-Pokemon games (MTG/YGO/OP) — better expansion mapping for large sets.
• **Centralized collector number module**: `vps-scraper/src/collector-number.ts` — `parseCollectorNumber()`, `normalizeCollectorNumber()`, `normalizeSetCode()` for all 5 game formats.
• **Cross-region game filter**: `/cross-region` page now has game tabs (All, Pokemon, MTG, YGO, One Piece) via searchParams.
• **Tiered price history retention**: Sub-$1 cards skip `price_history` writes entirely. $1-5 cards: 1-day retention. $5+ cards: 7-day retention. Cleanup uses two-step delete with avg_price threshold.
• **Card format guide**: `docs/card-format-guide.md` — authoritative reference for card identity across all 5 games.
• **Tests**: 935 → 972 (37 new: collector-number parsing, BigWeb set map, YGO fallback, tiered retention, game filter)

Changed

• `findSameCardDeals()` restructured in both Next.js and VPS to use `trackBest()` per-region pattern.
• Retention cleanup in orchestrator now tiered: `PRICE_HISTORY_EXTENDED_RETENTION_DAYS` (default 7) for max retention, `PRICE_HISTORY_RETENTION_DAYS` (default 1) for base.
• `filterChangedPrices()` type accepts optional `market_price` field for price-tier filtering.

Owner Action

• Set `CARD_LINKS_ENABLED=true` on DO VPS and run `--source card-links` (#7o in owner-action-items.md)

Added

• **BigWeb Pokemon set matching**: Manual `BIGWEB_POKEMON_SET_MAP` (30+ JP set names), `extractDenominator()` for NNN/NNN format, `setSizeMap` for denominator-hint disambiguation. 4-tier matching priority: manual map → set name map → denominator hint → fallback. Target: 1% → 30%+ match rate.
• **BigWeb YGO JP→EN mapping**: `mapYgoJpToEn()` converts "IGAS-JP001" → "IGAS-EN001". Batch pre-conversion at page level for efficient DB lookups. Target: 0% → 20%+ match rate.
• **LP in cross-region linked deals**: JP LP (Excellent) ≈ Western NM quality. `findCrossRegionDeals()` and VPS `findLinkedDeals()` now query NM+LP, filtering LP to JP platforms only (`yuyutei`, `bigweb`, `surugaya`). Same-card deals (US↔EU) stay NM-only. LP badge shown in Telegram report and `/cross-region` page UI.
• **Card link automation**: `vps-scraper/src/card-link-sync.ts` — automated EN↔JP Pokemon card linking. 3-tier matching: unique name → name+collector number tiebreaker → collector number only. Registered in orchestrator as post-scraper step, gated on `CARD_LINKS_ENABLED=true`. Supports `--source card-links` for standalone testing.
• **Cardmarket S3 multi-game matching**: `LIKE '%/%'` collector number filter now conditional on Pokemon only (MTG/YGO/OP use different formats). Dynamic threshold for small sets (`Math.min(3, sampleNames.length)`). Collector-number fallback for products that fail name matching within mapped expansions. `normalizeCardName()` handles V-UNION, ex/EX/GX case variants.

Changed

• `CrossRegionDeal` interface: added `conditionA?`, `conditionB?` fields
• `CrossRegionResult` interface: added `cheapCondition?`, `expensiveCondition?` fields
• `ProductFields` interface: added `denominator` field
• Tests: 893 → 933 (40 new tests for all 5 items)

Owner Action

• Set `CARD_LINKS_ENABLED=true` in DO VPS `.env` after running manual `populate-card-links.ts` backfill (#7o)
• Run `--source card-links --dry-run` to verify stats before enabling

Added

• **Automated daily health check**: Comprehensive DB audit runs after every `--full-sync` daily report. Sends a separate Telegram message covering storage (size, table breakdown, retention leak, bloat, write volume), card coverage (per-game, priced %, orphans), pricing (platform×game NM matrix, freshness, condition policy violations), cross-region overlap (US/EU/JP by game), and data quality (zero prices, high prices, sync failures, exchange rate age).
• **Migration 031**: `run_db_health_check()` SECURITY DEFINER RPC — runs all 18 health queries in a single call, returns JSONB. Includes dead tuple check, cross-region CTE, condition policy audit, exchange rate age, and write volume estimation.
• **`vps-scraper/src/db-health-check.ts`**: Calls the RPC, formats a detailed Telegram report with status icons per section.
• **Health check prompt updated**: `prompt/tcgsignal-database-health-check.md` — fixed 4 critical issues (RPC-based queries, column name mismatches, missing tables, realistic storage targets) and added 8 improvements (retention effectiveness, growth rate, per-platform conditions, exchange rate age, One Piece JP targets).

Owner Action

• Apply migration 031 in Supabase SQL Editor before next daily sync.

Changed

• **price_history retention default**: Code default changed from 7 days to 1 day (`vps-scraper/src/index.ts`). Both VPS `.env` updated to `PRICE_HISTORY_RETENTION_DAYS=1`. Expected savings: ~87 MB after next sync cycle.
• **Migration 030**: `get_table_sizes()` RPC function for daily health reporting.

Removed

• **Lorcana cards**: Deleted 2,782 Lorcana cards from DB (no active scrapers, zero FK refs, no user data). Game enum kept in codebase for future use. Saves ~2-3 MB after autovacuum.

Fixed

• **VERSION/package.json**: Synced to v0.88.0 (was stale at v0.86.1).

Added

• **Same-card cross-platform deals**: New `findSameCardDeals()` compares prices for the SAME card_id across US (tcgplayer/ebay) and EU (cardtrader/cardmarket) platforms. No card_links needed — instantly surfaces MTG, YGO, Pokemon, and One Piece cross-region gaps. Minimum $2 price threshold filters out sub-dollar noise.
• **`findAllCrossRegionDeals()`**: Merges card_links-based EN↔JP deals with same-card US↔EU deals, deduplicates by card_id pair, sorts by gap %.
• **VPS same-card deals**: `findSameCardDeals()` + `findLinkedDeals()` in `vps-scraper/src/cross-region.ts` mirror the Next.js logic for Telegram reporting.
• **Cross-region in daily Telegram report**: `sendDailyReport()` now includes a cross-region deals section (was only in deprecated `sendCombinedReport`).
• **Pokemon SET_CODE_MAP expansion**: Added ~40 sets covering SM (Sun & Moon), XY, and BW (Black & White) eras. Total: ~65 sets (up from 25). Owner needs to re-run `populate-card-links.ts` to expand from 2,674 → ~15K+ links.
• **Tests**: 10 new tests (893 total) — same-card deals, SM/XY/BW set mappings, daily report integration.

Changed

• **`/cross-region` page**: Now calls `findAllCrossRegionDeals()` (was `findCrossRegionDeals()`) to show both deal types.
• **`checkCrossRegionArbitrage()`** (both Next.js + VPS): Now runs linked + same-card queries in parallel and merges results.
• **Telegram cross-region section**: Label changed from "Cross-Region Arbitrage" to "Cross-Region Deals" to reflect broader scope.

Owner Action

• Run `npx tsx scripts/populate-card-links.ts` to expand EN↔JP links with new set mappings (see #7o in owner-action-items.md).
• Run `npx tsx scripts/sync-tcgcsv-catalog.ts` to bootstrap One Piece cards (see #7n in owner-action-items.md).

Fixed

• **BigWeb circuit breaker was global, not per-game**: `consecutiveFailures` counter was shared across all games in the game loop. When Pokemon hit pages past its catalog end, 3 consecutive failures would abort the entire scraper — MTG and One Piece never ran. Circuit breaker now resets per game and uses `break` (skip to next game) instead of `return` (abort all).

Changed

• **BIGWEB_MAX_PAGES is per-game**: Each game independently gets up to MAX_PAGES pages per run. With `BIGWEB_MAX_PAGES=80` and 3 games: Pokemon 1-80, One Piece 1-46 (full catalog), MTG 1-80. Total up to 240 pages/run (~20 min at 5s/page).
• **Page rotation state already per-game**: `state/bigweb-rotation.json` tracks `gameOffsets` independently — Pokemon on page 161 doesn't affect MTG starting at page 1.

Owner Action

• Set `BIGWEB_MAX_PAGES=80` on DO VPS `.env` (was 50, but MTG/OP never ran due to this bug).

Fixed

• **eBay intermittent sync failures**: Gate now checks BOTH `EBAY_CLIENT_ID` AND `EBAY_CLIENT_SECRET` (was only checking ID, causing silent 503 when secret missing). Token cache cleared on any auth failure (prevents stale token reuse on Vercel warm instances). Added logging when eBay skipped.
• **Cardmarket €0.00 prices**: Skip price entries where trend/avg/low are all < 1 cent (no sales data). Logs count of skipped zero-price entries per game.

Changed

• **Hetzner cleanup**: Removed duplicate `CARDMARKET_SOURCE=csv` from Hetzner `.env` (S3 mode is now the only entry).
• **owner-action-items.md**: Marked 7i (Cardmarket S3 on Hetzner) and 7j (eBay Vercel) as completed.

Added

• **Cardmarket S3 multi-game support**: `CARDMARKET_S3_GAMES=6,1,3,15` enables all 4 games (Pokemon, MTG, YGO, One Piece) via plain HTTP S3 JSON downloads. ~2-3 min vs 30-60 min Camoufox. Replaces Camoufox pipeline entirely. Set `CARDMARKET_SOURCE=s3` on Hetzner.
• **BigWeb MTG + YGO field extraction**: Game-aware `getProductFields()` helper handles different field assignments per game (Pokemon/OP/MTG/YGO). `extractMtgCollectorNumber()` and `extractYgoCollectorNumber()` for game-specific parsing.
• **YUYU-TEI game filter safety fix**: Added `.eq('game', 'pokemon-jp')` to card DB queries to prevent cross-game collector number collisions when multi-game is added later.
• **Cross-region deals page**: `/cross-region` browse page shows cards with significant price gaps across US/EU/JP regions. Sortable by gap percentage. Added to Header + MobileNav.
• **Cross-region arbitrage in Telegram**: Daily combined report now includes "Cross-Region Arbitrage" section showing top 5 deals with 30%+ price gaps across regions.
• **`src/lib/cross-region-deals.ts`**: `findCrossRegionDeals()` and `checkCrossRegionArbitrage()` query card_links + price_summary for cross-region price comparisons.
• **`vps-scraper/src/cross-region.ts`**: VPS-side cross-region check for Telegram report integration.
• **Cardmarket S3 smoke test**: Updated smoke test to probe S3 price guide URL instead of cardmarket.com (no Cloudflare).
• **Test fixtures**: `cardmarket-products-sample.json` and `cardmarket-price-guide-sample.json` from real S3 data.
• **Tests**: 42 new tests (868 total) — Cardmarket S3 multi-game config, fixture parsing, BigWeb MTG/YGO extraction, YUYU-TEI game filter, cross-region deals.

Changed

• **Cardmarket data_source**: `cardmarket_csv` → `cardmarket_s3` in price_summary rows.
• **Cardmarket scraper source name**: Reports as `cardmarket-s3` instead of `cardmarket-csv`.
• **Cardmarket stub**: `s3` mode alias added (backward-compatible with `csv`).
• **Response caching**: Cardmarket S3 downloads cached as `cardmarket-s3-price-guide-{gameId}.json` and `cardmarket-s3-products-{gameId}.json`.

Owner Action

• Switch Hetzner VPS to `CARDMARKET_SOURCE=s3` and `CARDMARKET_S3_GAMES=6,1,3,15` (see owner-action-items.md #7i).
• Add `EBAY_CLIENT_ID` + `EBAY_CLIENT_SECRET` to Vercel dashboard (see #7j).
• Dry-run BigWeb MTG+YGO on DO VPS: `BIGWEB_GAMES=170,177,1,9 node dist/index.js --source bigweb --dry-run` (see #7k).

Fixed

• **Telegram daily report not sending**: Combined sync report exceeded Telegram's 4096-character limit, causing silent failure. `sendTelegram()` now auto-splits long messages at paragraph boundaries and logs API error responses for debugging.
• **Orchestrator Telegram logging**: Now logs "Telegram report sent ✓" or "FAILED ✗" after each run for visibility.
• **Purged 39 MP + 2 LP residual rows** from price_summary/price_history (condition policy enforcement).

Added

• **Write-on-change history** (`history.ts`): `filterChangedPrices()` compares today's prices against yesterday's snapshot and only writes rows where avg_price or low_price actually changed. Applied to tcgcsv, cardtrader, tcgtracking. Expected 50-70% reduction in daily price_history writes (~300K → ~90-150K rows/day).
• **Platform filter for history**: Only tcgplayer + cardtrader write to price_history. Removed history writes from yuyutei, bigweb, surugaya, cardmarket (low-volume sources not driving trend features).
• **DB size monitoring in Telegram report**: Shows actual DB size (MB), percentage of 500 MB Supabase free tier, with color-coded status icon (🟢 <70%, 🟡 70-89%, 🔴 ≥90%).
• **`get_db_size_mb()` RPC** (migration 029): Postgres function to query actual database size.
• **Multi-recipient Telegram reports**: Set `TELEGRAM_EXTRA_CHAT_IDS` (comma-separated) in VPS `.env` to send the daily report to additional users.
• **Tests**: 8 new tests for write-on-change logic and platform filter (826 total).

Owner Action

• Apply migration 029 (`get_db_size_mb`) in Supabase SQL editor.
• To add a second Telegram recipient: get their chat ID (they must `/start` the bot first), then add `TELEGRAM_EXTRA_CHAT_IDS=<their_chat_id>` to DO VPS `.env` and Hetzner `.env`.
• Monitor DB size in Telegram report for 3-5 days. If still above 80%, reduce `PRICE_HISTORY_RETENTION_DAYS` from 7 to 3.

Added

• **BigWeb One Piece game mapping**: Added `game_id 177 → onepiece` to BigWeb scraper's `gameIdToGame` config. Enable with `BIGWEB_GAMES=170,177` on DO VPS.
• **Tests**: `gameIdToGame` mapping coverage for all 4 game IDs (170, 1, 9, 177).

Owner Action

• Set `BIGWEB_GAMES=170,177` in DO VPS `.env` to enable One Piece alongside Pokemon.
• Dry-run recommended first: `BIGWEB_GAMES=177` with `--source bigweb --dry-run` to validate OP collector number format.

Added

• **SET_CODE_MAP expanded to 25 entries**: Added 13 SWSH-era mappings (SWSH01–SWSH12 + CRZ) and 2 new SV-era mappings (DRI, JTG), covering Sword & Shield through Scarlet & Violet eras. JP_TO_EN_SET auto-derives.
• **Tier 2.5 matching** (`scripts/populate-card-links.ts`): Within mapped sets, uses `collector_number` as tiebreaker when multiple candidates share the same name (confidence 0.85).
• **Tier 3 fallback matching** (`scripts/populate-card-links.ts`): Cards with no set mapping can now link via exact name + exact collector_number match across catalogs (confidence 0.7).
• **CrossRegionPrices component** (`src/components/features/prices/CrossRegionPrices.tsx`): Displays linked card prices grouped by card with region badge (EN/JP), platform rows, USD + original currency, and exchange rate footnote. Hidden when no linked prices exist.
• **Cross-region prices on card detail page**: `getLinkedPrices()` fetched in parallel with existing data, rendered after the main price panel. JPY and other linked currencies merged into exchange rate fetching.
• **"View JP/EN version" link** on card detail page: Links to the counterpart card in the card info area, styled in purple to distinguish from external buy links.

Changed

• Populate script now fetches `collector_number` alongside `id, name, set_code` for improved matching accuracy.

Fixed

• **Cardmarket Camoufox price extraction** (`cardmarket_scraper.py`): Price Trend regex was broken — `[^\d]*?` stopped at digits in CSS class names (e.g. `col-6`), causing the extractor to fall through to the generic "first EUR price on page" (cheapest listing, any condition). Now anchors on `</dt>` HTML structure to reliably extract the Trend price. Verified against cached HTML: correctly extracts €0.04 trend instead of €0.02 cheapest listing.
• **Purged 7,079 stale `cardmarket_csv` price_summary rows**: S3-sourced data with broken `?idProduct=` URLs and outdated prices. 37 real Camoufox rows retained with correct prices and working slug URLs.

Added

• **Backlog items**: TCGPlayer stale ID cleanup (Bug 3), Cardmarket Phase 1 coverage expansion.

Added

• **card_links table** (migration 028): Cross-region card matching — links cards that represent the same physical card across different game catalogs (EN↔JP).
• **populate-card-links script** (`scripts/populate-card-links.ts`): 3-tier name-based matching — Tier 1: unique name (1,041 links, confidence 1.0), Tier 2: set-mapped duplicates (1,633 links, confidence 0.9), Tier 3: skip ambiguous (2,189 skipped). Total: 2,674 links.
• **getLinkedPrices() helper** (`src/lib/card-links.ts`): Given any card_id, finds linked cards via card_links and returns NM prices from all regions/platforms. Foundation for cross-regional price comparison UI.
• **hasLinkedCards() helper** (`src/lib/card-links.ts`): Lightweight check for cross-region links without fetching prices.
• **normalizeCardNameForMatching()** (`src/lib/card-links.ts`): Strips collector number suffixes ("028 142", "028/142"), attack brackets, diacritics, punctuation for cross-region name matching.

Fixed

• **Card link matching strategy**: Collector-number matching failed (EN/JP Pokemon sets have different card ordering at same numbers). Switched to name-based matching with skip-duplicates approach.
• **Mobile card page crash** (`usePriceUpdates.ts`): WebSocket errors (Realtime unavailable, insecure context) were propagating uncaught to the error boundary, showing "Could not load card" on mobile. Now fails silently — page loads with cached data.

Fixed

• **CRITICAL: Suruga-ya N+1 queries** (`surugaya.ts`): Replaced per-product DB queries (~4,000/run) with batch lookup per page via `.in('collector_number', [...])`. ~95% query reduction.
• **YUYU-TEI circuit breaker** (`yuyutei.ts`): HTTP 403/429 errors now increment `consecutiveFailures`. Previously only exceptions tripped the breaker.
• **YUYU-TEI upsert resilience** (`yuyutei.ts`): price_summary and price_history upserts wrapped in per-batch try/catch. One transient error no longer kills the entire slug's remaining batches.
• **YUYU-TEI price_history NM filter** (`yuyutei.ts`): Added explicit NM-only filter (was relying on all rows being NM).
• **BigWeb ambiguous matching** (`bigweb.ts`): Pre-loads set name→set_code map, resolves `product.cardset` to exact set_code before picking candidates. Falls back to substring heuristic only when mapping fails.
• **Suruga-ya blind matching** (`surugaya.ts`): When multiple cards share a collector number, picks the candidate with the best name similarity instead of taking the first arbitrary result.

Added

• **CardTrader price_history** (`cardtrader.ts`): NM-only price_history writes after price_summary upsert. CardTrader prices now participate in trend analysis, alerts, and history charts.

Removed

• **195 structural tests** removed across 26 files (985 → 790 tests, 46 → 38 files). Tests checked code shape (file contents, enum counts, config values) rather than behavior. All behavioral tests retained. Rollback doc: `docs/structural-tests-removed.md`.
• 8 test files deleted entirely: skeleton, diversity, image-optimization, isr-revalidation, medium-fixes, rate-limit-tiers, realtime-prices, search-filter-improvements.

Removed

• **5 unused/low-usage database indexes** dropped to save ~25-35 MB:

- `idx_cards_name` (full-text GIN, 3 scans — trigram index covers all search)

- `cards_name_jp_trgm_idx` (JP trigram GIN, 22 scans — full scan acceptable at this frequency)

- `idx_cards_cardtrader_id` (0 scans — card matching uses unique constraint instead)

- `idx_cards_cardmarket_product_id` (0 scans — card matching uses unique constraint instead)

- `idx_price_summary_source_url` (0 scans — never queried)

Changed

• **`.gitignore` cleanup**: Added `prompt/`, `ct_games.json`, `vps-scraper/dist/`, `__pycache__/` to reduce git status noise (12 untracked files → clean)

Changed

• **YUYU-TEI set map expanded 20 → 59 sets** (`yuyutei.ts`): Added complete SV era (SV1–SV4), 29 Sword & Shield era sets, and 9 Sun & Moon top sets. Projected JP coverage 962 → 3,000–5,000 cards.
• **BigWeb default MAX_PAGES 10 → 50** (`bigweb.ts`): 5x more cards per run, full catalog cycle in 1 day instead of 5.
• **Surugaya default MAX_PAGES 5 → 20** (`surugaya.ts`): 4x more products scraped per run.

Fixed

• **YUYU-TEI m1L/m1S case mismatch** (`yuyutei.ts`): Set codes were `M1L`/`M1S` but DB has `m1L`/`m1S` — 0 matches for those sets. Fixed casing.

Changed

• **TCGTracking NM-only in price_summary** (`tcgtracking.ts`): TCGTracking's value is fresher NM SKU-level pricing. LP/MP/HP/DMG are redundant with TCGcsv. Drops ~398K rows (~160 MB).
• **NM+LP only for all sources in price_summary** (`tcgcsv.ts`, `tcgcsv/route.ts`, `cardtrader.ts`, `surugaya.ts`): MP/HP/DMG cards are low-value, not useful for cross-regional arbitrage. Drops ~237K rows (~95 MB).

Fixed

• **tcgcsv price_history NM-only filter** (`tcgcsv.ts`, `tcgcsv/route.ts`): Both VPS and Vercel tcgcsv scrapers were writing ALL conditions to `price_history`, violating NM-only policy from v0.82.1.

Added

• **Retention cleanup enhancements** (orchestrator): Purges non-NM from `price_history`, MP/HP/DMG from `price_summary`, non-NM tcgtracking rows, and stale rows (>30 days). Runs on every `--full-sync`.

Added

• **Cardmarket bulk price download via S3 JSON** (`cardmarket-csv.ts`): Downloads public Price Guide + Product List JSON from `downloads.s3.cardmarket.com` — no Camoufox needed. Covers all 65k+ Pokemon products (vs 500 cards with browser scraping). ~30 seconds vs ~58 minutes. Matches products to cards by normalized name within expansion, stores `cardmarket_product_id` for fast subsequent lookups.
• **Migration 027**: `cardmarket_product_id` integer column on cards table for CSV join key.

Changed

• **CARDMARKET_SOURCE=csv**: Now routes to the JSON bulk downloader. Camoufox mode (`camoufox`) remains as fallback.

Fixed

• **Remove non-existent `last_sync_date` from upsert payloads** (`cardtrader.ts`, `cardtrader/route.ts`, `yuyutei/route.ts`): The `last_sync_date` field was included in `price_summary` upserts but the column was never created via migration. Supabase PostgREST schema cache rejection caused 100% CardTrader upsert failure (0 rows, 1063 errors). The `last_updated` timestamp already serves the same purpose.

Fixed

• **CardTrader Yu-Gi-Oh! game_id corrected: 6 → 4** (`cardtrader-constants.ts`, `cardtrader.ts`): Game ID 6 is Flesh and Blood, not Yu-Gi-Oh! Verified against CardTrader `GET /games` endpoint. Previous mapping caused 0 YGO prices to be synced (F&B expansions never matched yugioh DB sets). No DB cleanup needed — no bad rows were written.
• **Cardmarket scraper: browser-only mode when no cf_clearance** (`cardmarket_scraper.py`): Cardmarket doesn't issue CF challenges to Camoufox browser visitors from Hetzner, so `cf_clearance` cookie never exists. `curl_cffi` HTTP requests without it got 429'd on every request. Now auto-detects: no cf_clearance → Camoufox for all pages (~7s/page); cf_clearance present → curl_cffi HTTP fast path (~2-3s/page).

Changed

• **Cardmarket stub timeout: 20 min → 90 min** (`cardmarket-stub.ts`): Browser-only mode at ~7s/page for 500 cards needs ~58 min. Hetzner→DO window is 5 hours (17:00→22:00 UTC), so 90 min is safe.

Changed

• **Cardmarket scraper uses cookie harvest + curl_cffi HTTP** (`vps-scraper/python/cardmarket_scraper.py`): Camoufox solves Cloudflare challenge once during Phase 1 discovery, then cookies + UA are harvested. Phase 2 pricing uses `curl_cffi` with `impersonate="firefox"` (matching Camoufox's Firefox TLS fingerprint) for all page fetches — ~2-3s per page vs 7-10s with full browser. Falls back to Camoufox for one page if CF challenges HTTP, then continues HTTP.
• **Default CARDMARKET_MAX_CARDS increased 100 → 500** — still env-var overridable. HTTP mode handles 500 cards in ~15 min vs 100 in ~12 min with browser.
• **Idle pause interval widened to 15-25 pages** (was 5-10) for HTTP mode since individual fetches are faster.

Added

• `harvest_cookies(context)` — extracts cf_clearance + all cardmarket cookies + UA from Camoufox session
• `fetch_with_http(url)` — curl_cffi HTTP fetch with Firefox impersonation, returns None for CF challenge, "" for other errors
• `refresh_cookies_via_browser(context, url)` — refreshes cookies by loading one page via Camoufox
• Two-mode circuit breaker: 3 cookie refreshes per run OR 3 consecutive browser failures → hard stop
• Validation report: `httpFetches`, `browserFetches`, `cookieRefreshes` fields in Telegram report
• `curl_cffi>=0.7.0` added to `vps-scraper/python/requirements.txt`
• CF challenge HTML fixture + 17 new tests for cookie harvest + HTTP fetch path

Deploy Notes

• Hetzner VPS: `pip install -r python/requirements.txt` to install `curl_cffi`
• `CARDMARKET_MAX_CARDS=500` is now the default (no .env change needed unless overriding)

Fixed

• **CardTrader set matching rewritten — code-first with name fallback** (`vps-scraper/src/scrapers/cardtrader.ts`): Previous approach ran 3,562 individual `ilike` queries (one per expansion), producing 0.8% match rate in 89 minutes. New approach loads all distinct `(set_code, set_name)` per game in 3 paginated queries, builds in-memory lookup maps, matches expansions by `set_code` first then `normalizeName(set_name)` fallback. Unmatched expansions skip the products API call entirely.
• **CardTrader card name matching uses `normalizeName()`**: Both DB card names and CardTrader product names now go through `normalizeName()` — handles diacritics, apostrophe variants, punctuation, whitespace differences.
• **TCGTracking moved from DO → Hetzner VPS**: DO VPS (Singapore) returns 403 from TCGTracking (Cloudflare blocks). Hetzner (Germany) returns 200. Docstring updated.

Added

• `normalizeName()` in `vps-scraper/src/utils.ts` — shared fuzzy name normalizer for set and card matching (strips diacritics, punctuation, normalizes quotes/apostrophes, collapses whitespace)
• 12 new tests: `normalizeName` (7) + CardTrader matching logic (5)

Added

• 7-day retention cleanup for `price_history` — runs before exchange rates in `--full-sync`. Configurable via `PRICE_HISTORY_RETENTION_DAYS` env var (default 7).
• NM-only `price_history` writes across TCGTracking, Suruga-ya, eBay — LP/MP/HP/DMG excluded from history (still written to `price_summary`). Reduces daily growth by ~75%.

Deploy Notes

• Hetzner VPS: ensure `TCGTRACKING_ENABLED` is not `false` (defaults to `true`)
• DO VPS: add `TCGTRACKING_ENABLED=false` to `.env`
• Both VPSes: retention cleanup runs automatically on `--full-sync`

Changed

• **TCGTracking moved from Vercel to DO VPS** (`vps-scraper/src/scrapers/tcgtracking.ts`): 45s Vercel time budget only covered ~70% of sets per game. VPS scraper runs without time constraint — processes all sets for all 3 games (pokemon, mtg, yugioh) to completion. Runs as "pre-vercel" phase after TCGCSV. Circuit breaker on 5 consecutive failures.
• **CardTrader moved from Vercel to DO VPS** (`vps-scraper/src/scrapers/cardtrader.ts`): 50s Vercel budget only covered ~30 expansions. VPS scraper processes all expansions per game with 1s rate limit. Same EUR→USD conversion. Circuit breaker on 5 consecutive failures.
• **TCGCSV timeout increased 30min → 60min**: Full catalog (1,553 groups) daily with generous safety margin.
• **Vercel route chain reduced to 4 routes** (`vps-scraper/src/vercel-sync.ts`): Removed TCGTracking (3 routes) + CardTrader (1 route). Remaining: eBay, alerts, digest, history.
• **Orchestrator registers 7 scrapers**: Added tcgtracking + cardtrader as pre-vercel. Order: tcgcsv → tcgtracking → cardtrader.
• **Smoke test expanded**: Added tcgtracking + cardtrader probe URLs.

Added

• `vps-scraper/src/scrapers/tcgtracking.ts` — VPS TCGTracking scraper
• `vps-scraper/src/scrapers/cardtrader.ts` — VPS CardTrader scraper
• `vps-scraper/src/constants/tcgtracking-constants.ts` — Shared constants
• `vps-scraper/src/constants/cardtrader-constants.ts` — Shared constants

Deploy Notes

• DO VPS: `BIGWEB_MAX_PAGES=80`
• Hetzner VPS: `SURUGAYA_MAX_PAGES=200`, crontab `0 21` → `0 17`

Fixed

• **TCGCSV timeout increased 15min → 30min** (`vps-scraper/src/scrapers/tcgcsv.ts`): 15-min timeout only covered ~400/1553 groups (26%) per run — full catalog took 4 days. 30-min covers ~800 groups (52%), full catalog in 2 days of rotation. TCGplayer prices don't change hourly so 2-day rotation is sufficient.
• **Camoufox pip version pinned** (`vps-scraper/python/requirements.txt`): Changed `camoufox[geoip]>=0.4.0` → `camoufox[geoip]==0.4.11`. Camoufox auto-updated its browser binary mid-run on Hetzner (v135.0.1-beta.24), download progress text went to stdout and corrupted JSON output. Pinning prevents surprise upgrades.
• **Hetzner Cardmarket env var** (`.env` on VPS, not in repo): Added `CARDMARKET_ENABLED_GAMES=pokemon` — without this, the scraper returned 0 rows in 11.2s on the March 10 cron run.

Deployed

• DO VPS: TCGCSV 30-min timeout
• Hetzner VPS: Camoufox pin + CARDMARKET_ENABLED_GAMES=pokemon

Changed

• **TCGCSV moved from Vercel to DO VPS** (`vps-scraper/src/scrapers/tcgcsv.ts`): Vercel route consistently timed out at 60s (1,600+ groups = 27s loading alone). New VPS scraper runs without time constraint (~8-12 min) with 15-min timeout guard and 5-failure circuit breaker. Runs as "pre-vercel" phase so TCGTracking's fresher data overwrites where they overlap. Vercel route `/api/sync/tcgcsv` kept as manual fallback.
• **Orchestrator phase system** (`vps-scraper/src/index.ts`): Added `phase` concept — `pre-vercel` (TCGCSV, runs before Vercel chain) and `post-vercel` (YUYU-TEI, BigWeb, Cardmarket, Suruga-ya). Full-sync sequence: exchange → pre-vercel → Vercel routes → post-vercel.
• **TCGCSV removed from Vercel route chain** (`vps-scraper/src/vercel-sync.ts`): 8 routes remain (was 9).
• **Combined Telegram report** expanded: new "Pre-Vercel Scrapers" section between exchange rates and Vercel routes.

Fixed

• **Dry-run forces offline for Camoufox sources**: `--dry-run` now automatically uses cached HTML for cardmarket and surugaya — no Camoufox launch, no network requests. Enforced at two layers: orchestrator (`index.ts`) and Python scraper (`cardmarket_scraper.py`). Direct `python3 cardmarket_scraper.py --dry-run` from SSH is also safe.
• **Manual runs now logged** (`daily-scrape.sh`): All output auto-tees to `/var/log/tcgsignal-scraper.log` via `exec > >(tee -a)`. Previously only cron runs were logged (via shell redirect). Also skips jitter for `--source` flag.
• **daily-scrape.sh execute permission** restored on both VPSes (`chmod +x`).

Added

• `vps-scraper/src/constants/tcgcsv-constants.ts` — shared GAME_CATEGORIES, SUBTYPE_CONDITION, TcgcsvPrice type
• TCGCSV smoke test probe in orchestrator
• VPS CLI reference in `docs/architecture.md` (all flags, dry-run vs live-run table)

Docs

• `docs/architecture.md`: Updated sync chain, file tree, added CLI reference with dry-run comparison table
• `docs/DATA_SOURCES.md`: Updated TCGCSV entry (VPS scraper details, validated metrics), scrapers table with phase column, dry-run safety docs

Tests

• 874 total (46 files), all passing. Added 22 new tests: TCGCSV constants (11), TCGCSV scraper (16), cardmarket dry-run enforcement (1), orchestrator phase/registration updates, daily-scrape.sh logging.

Deployed

• DO VPS: TCGCSV scraper + logging fix + dry-run safety
• Hetzner VPS: logging fix + dry-run safety + cardmarket enforcement

Validated

• TCGCSV dry-run on DO VPS: 1,626/1,626 groups, 284,931 parsed, 264,359 matched (93%), 188,751 rows

Fixed

• **YUYU-TEI regex reverted to full fraction** (`vps-scraper/src/scrapers/yuyutei.ts`): v0.80.0 change was based on wrong assumption that DB stores numerator only. True for `game='pokemon'` but **false** for `game='pokemon-jp'` (TCGCSV category 85) which stores `"025/414"`. YUYU-TEI matches pokemon-jp cards, so regex must capture full fraction. Reverted from `/(\d{3})\/\d{3,4}/` back to `/(\d{3}\/\d{3,4})/`.
• **Cardmarket Cloudflare detection now language-agnostic** (`cardmarket_scraper.py`, `fetch_url.py`): Old check (`"Just a moment"` + `"cf-browser-verification"`) was English-only and missed German CF pages served to Hetzner. Replaced with `"cf_challenge"` + `"cf-turnstile-response"` markers found in actual cached blocked pages.
• **Cardmarket comment corrected** (`cardmarket_scraper.py`): Clarified that `cn.split('/')[0]` operates on English `game='pokemon'` cards (usually bare numerators), unrelated to YUYU-TEI's pokemon-jp full-fraction matching.

Tests

• Updated 4 YUYU-TEI collector number tests to expect full fraction format (829 total, all passing)

Fixed

• **YUYU-TEI collector number bug** (`vps-scraper/src/scrapers/yuyutei.ts`): Regex captured full fraction `"025/414"` but DB stores numerator only `"025"`. Changed regex from `(/(\d{3}\/\d{3,4})/)` to `(/(\d{3})\/\d{3,4}/)` — moves `/` outside capture group. Was causing 0 DB matches despite 962 parsed rows. BigWeb unaffected (already extracted numerator only).
• **Cardmarket comment typo** (`vps-scraper/python/cardmarket_scraper.py`): Fixed misleading comment claiming DB stores `"025/191"` format.
• **Cardmarket SyntaxError** (`cardmarket_scraper.py`): `try/else/except` block was in wrong order — Python requires `except` before `else`. Was preventing all Cardmarket scraping.
• **Cardmarket get_urls() timeout** (`cardmarket_scraper.py`): MTG/YGO queries hit Supabase statement timeout (57014) scanning 293K cards with no `cardmarket_id`. Added `CARDMARKET_ENABLED_GAMES` filter — only queries games with expansion map entries (Pokemon only). Eliminates unnecessary full-table scans.
• **daily-scrape.sh execute permission**: `git pull` was resetting file mode to 644, silently breaking cron on both VPSes. Set mode to 755 via `git update-index --chmod=+x`.

Added

• **Cardmarket expansion map expanded** (3→15 sets): Fusion Strike, Ascended Heroes, Paldea Evolved, Paradox Rift, Scarlet & Violet, Cosmic Eclipse, Unified Minds, Paldean Fates, Sword & Shield, Evolving Skies, Destined Rivals, Chilling Reign. Target: 3,000+ cards with `cardmarket_id` (up from 653).

Verified (no code changes)

• **Cardmarket pipeline** code reviewed: `needs_discovery()` 80% skip threshold correct, circuit breaker counters independent between discovery/pricing phases, `get_urls()` correctly targets cards with 2+ platform overlap.
• **CardTrader sync** route exists since v0.75.0, gated by `CARDTRADER_JWT_TOKEN`. Env var confirmed on Vercel + DO VPS.

Deployed

• DO VPS: YUYU-TEI fix + CardTrader token + expansion map
• Hetzner VPS: YUYU-TEI fix + expansion map (15 sets)

Tests

• 4 new YUYU-TEI collector number extraction tests via `parseSetPage()` (829 total, all passing)

Added

• **DO VPS full-sync mode** (`--full-sync` flag): Orchestrates entire daily sync from DO VPS — exchange rates (local Frankfurter API call) → Vercel routes sequentially (each gets full 60s) → local scrapers. Eliminates Vercel 60s timeout issues with TCGTracking Pokemon.
• **Local exchange rate sync** (`vps-scraper/src/exchange.ts`): Calls Frankfurter API directly and upserts to Supabase. No Vercel round-trip needed.
• **Vercel sync client** (`vps-scraper/src/vercel-sync.ts`): HTTP client calling 9 Vercel routes sequentially with 120s timeout each. Env-var gated (eBay, CardTrader). Continues on error.
• **Heartbeat route** (`/api/sync/heartbeat`): Dead man's switch — Vercel cron at 23:30 UTC checks if price data is stale (>6h). Sends Telegram alert to `ADMIN_TELEGRAM_CHAT_ID`. Arrives at 7:30 AM SGT (wake-up time).
• **Combined Telegram report** (`sendCombinedReport`): Three sections — exchange rates, Vercel routes, local scrapers. Flags TCGTracking Pokemon as known fragile point when it fails or returns <100 rows.

Changed

• **Vercel crons replaced**: Two crons (`/api/sync/daily` at 0:00 UTC, `/api/sync/exchange` at 17:00 UTC) replaced with single heartbeat cron at 23:30 UTC. Routes remain callable manually.
• **Cron script** (`daily-scrape.sh`): Passes through all CLI args. Jitter strategy: 0-10 min for `--full-sync`, 0-30 min for local-only, no jitter for `--dry-run`.
• **Schedule**: Hetzner 21:00 UTC (local scrapers), DO VPS 22:00 UTC (full sync), heartbeat 23:30 UTC.

Added

• **BigWeb API adapter** (`vps-scraper/src/scrapers/bigweb.ts`): JSON API scraper for BigWeb JP retail prices. Fetches features (sets) per game, paginates products, matches by `set_code + collector_number`. Writes `platform: 'bigweb', data_source: 'bigweb_api'`. Volume controls: `BIGWEB_MAX_FEATURES` (default 3), `BIGWEB_MAX_PAGES` (default 10). Rotation state persisted in `state/bigweb-rotation.json`. Circuit breaker: 3 consecutive failures stops the run. Updates `cards.name_jp` from BigWeb's `fname` field.
• **Suruga-ya HTML scraper** (`vps-scraper/src/scrapers/surugaya.ts`): Plain HTTP + cheerio scraper for JP used goods. Condition mapping: Near Mint→NM, Excellent→LP, Good→MP, etc. Japanese grade codes supported (SA/S/A→NM, B→MP, C→HP). Match by collector number or fuzzy name. Writes `platform: 'surugaya', data_source: 'surugaya_scrape'`. Volume controls: `SURUGAYA_MAX_CATEGORIES` (default 1), `SURUGAYA_MAX_PAGES` (default 5).
• **Migration 026** (`supabase/migrations/026_add_bigweb_surugaya.sql`): Expands `price_summary.platform` CHECK constraint to 8 platforms (added `bigweb`, `surugaya`). No `bigweb_id` column — matching via Feature ID mapping + collector number.
• **PLATFORMS expanded** (6→8): `bigweb` and `surugaya` added to `db-enums.ts`, `fees.ts`, `shipping.ts`, `affiliate.ts`, `source-diversity.ts`.
• **Per-source validation report** in Telegram: each scraper returns match rate, price range (min/median/max), sample card, overlap divergence. Price sanity check flags suspicious medians (JPY >¥500K, EUR/USD >€10K).
• **Env-var gating** in VPS orchestrator: yuyutei enabled by default, bigweb/surugaya require `=true`, cardmarket checks `CARDMARKET_SOURCE`. `--source` flag overrides gating for explicit testing.
• **Cardmarket card selection strategy**: Targets cards with `cardmarket_id` AND 2+ existing `price_summary` rows from other platforms, ordered by `last_updated DESC`. Ensures scraped cards overlap with existing data for meaningful cross-platform divergence.
• **Camoufox persistent context**: Browser context reused across page navigations (cookie reuse for cf_clearance). Accept-Language: `de-DE` for Hetzner IP locale matching.
• **Cardmarket combined discovery + pricing** (`python/cardmarket_scraper.py`): Single Camoufox session — Phase 1 discovers `cardmarket_id` slugs from expansion listing pages (CARDMARKET_EXPANSION_MAP), Phase 2 scrapes prices. Skips sets >80% populated. `cardmarket-ids.ts` deprecated, removed from orchestrator (5→4 scrapers).
• **Two-level response caching** (`vps-scraper/src/response-cache.ts`): Unified cache at `/var/log/tcgsignal-cache/`. Level 1 (all scrapers): save last response per label, overwritten each run. Level 2 (Cardmarket + Suruga-ya): `--offline` flag reads from cache instead of live requests. Prevents IP blocking during debugging. Standing Rule 10 added.
• **Suruga-ya CloudFront identified**: Returns 403 from DO Singapore, 200 from Hetzner DE. Promoted to Level 2 source with offline replay support.

Fixed

• **VPS scrapers missing `price_history` writes**: All 4 VPS scrapers (yuyutei, bigweb, surugaya, cardmarket) only wrote to `price_summary` — historical price data was lost every sync. Added `price_history` upsert to all 4 scrapers matching the Vercel sync pattern (`onConflict: card_id,platform,condition,date`, first write per day wins, non-fatal errors).
• **YUYU-TEI dedup** (`vps-scraper/src/scrapers/yuyutei.ts`): Deduplicate parsed cards by collector number before DB match — keeps lowest-priced entry (standard version) over alt art/SAR variant. Fixes non-fatal upsert conflicts in sets with duplicate entries (m02a, sv11b, sv11w). 6 tests added.

Changed

• **VPS orchestrator** (`vps-scraper/src/index.ts`): 4 scrapers registered (yuyutei, bigweb, cardmarket, surugaya). `--source` is primary CLI flag, `--test-source` deprecated with warning.
• **VPS config** (`vps-scraper/src/config.ts`): Source configs for bigweb (3-5s delay) and surugaya (3-5s delay).
• **VPS notify** (`vps-scraper/src/notify.ts`): Enhanced `ScraperValidation` with price range, median, CF clearance, overlap divergence. `formatValidation()` renders per-source detail in Telegram reports.
• **Source diversity** (`src/lib/source-diversity.ts`): bigweb→JP, surugaya→JP in `PLATFORM_REGIONS`. New divergence pairs: `tcgplayer:bigweb`, `tcgplayer:surugaya`, `yuyutei:bigweb`.

Added

• **TCGTracking adapter** (`src/lib/data-sources/tcgtracking.ts`): Data source for condition-specific TCGPlayer pricing. Writes `platform: 'tcgplayer', data_source: 'tcgtracking'` — enriches TCGCSV NM rows with SKU-level data and adds LP/MP/HP/DMG condition rows. Game categories: pokemon → [3, 85], mtg → [1], yugioh → [2].
• **`/api/sync/tcgtracking` route**: Accepts `?game=pokemon|mtg|yugioh`. 45s time budget, per-set try/catch. Upsert conflict key `(card_id, platform, condition)` overwrites TCGCSV NM prices with accurate SKU data. Cross-platform ID extraction (cardtrader_id) in separate outer try/catch.
• **VPS scraper project** (`vps-scraper/`): Standalone Node.js + Python project for scraping sites that block Vercel IPs (YUYU-TEI, Cardmarket). Full Cloudflare escalation ladder: direct → cached cf_clearance cookie → FlareSolverr → proxy. Key files:

- `src/proxy.ts`: 4-step fetch escalation

- `src/scrapers/yuyutei.ts`: Ports full YUYU-TEI parser

- `src/scrapers/cardmarket-csv.ts`: S3 CSV download + parse + upsert

- `python/cardmarket_scraper.py`: Camoufox-based fallback scraper

- `docker/docker-compose.yml`: FlareSolverr container

- `cron/daily-scrape.sh`: Bash wrapper with random jitter + log rotation

• **Price diversity verification** (`src/lib/source-diversity.ts`): 6 check functions appended to daily Telegram report — origin diversity, price divergence, condition coverage, regional coverage, cross-platform match rate, staleness. All non-blocking (Promise.allSettled, graceful DB failure defaults).
• **"By Condition" view in PriceComparisonPanel**: Platform × condition matrix table with horizontal scroll on mobile. New `PriceView` type: `'all' | 'conditions' | 'simple'`.
• **Migration 025** (`supabase/migrations/025_add_cardtrader_id.sql`): Adds `cardtrader_id text` column with partial index.
• **`docs/vps-scraper-setup.md`**: SSH assessment commands, network tests, decision tree, installation steps, incremental testing (5 steps), cron setup, troubleshooting table.
• **~88 new tests** in `tests/v077-tcgtracking-vps.test.ts`.

Changed

• **Cardmarket gated behind `CARDMARKET_VERCEL_SYNC=true`** in daily sync: VPS scraper handles Cardmarket by default. Set env var to re-enable Vercel-side sync.
• **Daily sync chain updated**: 3 parallel TCGTracking jobs (pokemon/mtg/yugioh) added after TCGCSV phase. Diversity checks block added post-Phase 3.
• **`notifyAdminDailySummary()`** extended: accepts optional `DiversityReport`, appends `📊 Price Diversity` section to Telegram body with severity logic.
• **`src/lib/data-sources/init.ts`**: 10 adapters total (added tcgtracking).
• **Existing tests updated**: adapter count 9→10 in v036/v041/v075 tests; sync-daily test updated for CARDMARKET_VERCEL_SYNC gate; PriceComparisonPanel test updated for PriceView type.

---

Full details for older versions are preserved in git history (`git log -- CHANGELOG.md`).

| Version | Date | Summary |

|---------|------|---------|

| v0.76.2 | 2026-03-07 | Functional health checks for all 9 adapters |

| v0.76.1 | 2026-03-06 | Japanese card names (`name_jp`) + sync health report + YUYU-TEI selector hotfix |

| v0.76.0 | 2026-03-06 | Cross-source deals & search enhancements |

| v0.75.0 | 2026-03-06 | CardTrader integration (adapter + sync route + platform CHECK) |

| v0.74.0 | 2026-03-03 | Search page filter improvements |

| v0.73.1 | 2026-03-03 | Community decks: visual builder |

| v0.73.0 | 2026-03-03 | Community decks: save + browse + view |

| v0.72.0 | 2026-03-03 | Deck parser enhancement + cross-platform deck pricing |

| v0.71.0 | 2026-03-03 | Nav consolidation + game hub enhancement |

| v0.70.3 | 2026-03-03 | Autocomplete performance + data source resilience |

| v0.70.2 | 2026-03-03 | Code quality audit + error handling hardening |

| v0.70.1 | 2026-03-03 | Rate limiter resilience + platform constraint fix |

| v0.70.0 | 2026-03-02 | Price history charts + buy signal |

| v0.69.0 | 2026-03-02 | Buyer price comparison + total cost |

| v0.68.0 | 2026-03-02 | Filter sidebar + inline multi-source prices |

| v0.67.0 | 2026-03-02 | Multi-source autocomplete + summary panels |

| v0.66.1 | 2026-03-02 | Pokemon Japan catalog (category 85) + JustTCG rate limit fix |

| v0.66.0 | 2026-03-02 | Multi-source integration (PriceComparisonPanel + YUYU-TEI) |

| v0.65.0 | 2026-03-02 | Game mega menu & hub polish |

| v0.64.0 | 2026-03-02 | Next.js 16 + React 19 + Tailwind CSS 4 upgrade |

| v0.63.0 | 2026-03-02 | Currency-aware deals |

| v0.62.0 | 2026-03-02 | Cardmarket EUR→USD fix & search defaults |

| v0.61.1 | 2026-03-02 | Consistent price formatting & URL cleanup |

| v0.61.0 | 2026-03-02 | Clickable prices & multi-source foundation (source_url, currency) |

| v0.60.0 | 2026-03-02 | Game hub pages, game-scoped search, search fixes |

| v0.59.0 | 2026-03-01 | Claude AI decklist analysis, database performance review |

| v0.58.0 | 2026-03-01 | Realtime prices, tiered rate limits, ISR tuning |

| v0.57.0 | 2026-03-01 | Smoke tests, image optimization, cache stats, type audit |

| v0.56.0 | 2026-03-01 | Data integrity fixes, performance optimizations |

| v0.55.0 | 2026-03-01 | Calendar export, sets search, help troubleshooting & glossary |

| v0.54.0 | 2026-03-01 | Alert export, deal presets, market freshness |

| v0.53.0 | 2026-03-01 | Search view toggle, tournament controls, pricing feature matrix |

| v0.52.0 | 2026-03-01 | Price insights, portfolio performance, context-aware dashboard |

| v0.51.0 | 2026-03-01 | Watchlist filters, deal stats, trending game breakdown |

| v0.50.0 | 2026-03-01 | Collection sort/search, saved search stats, PriceTable a11y |

| v0.49.0 | 2026-03-01 | Set sort, compare export/history, share button with prices |

| v0.48.0 | 2026-03-01 | Portfolio condition filter, alert timeline, preferences export |

| v0.47.0 | 2026-03-01 | Deal sort & time filter, trending summary stats, recent searches |

| v0.46.0 | 2026-03-01 | Searchable FAQ, alert stats & game filters, market volatility index |

| v0.45.0 | 2026-03-01 | Watchlist value widget, deal freshness, card quick stats, admin scorecard |

| v0.44.0 | 2026-03-01 | API docs, search price filter, enhanced exports, tournament trending, digest email |

| v0.43.0 | 2026-03-01 | Notification test buttons, AdSense, CSV import, welcome email |

| v0.42.0 | 2026-03-01 | Affiliate links, portfolio analytics, price chart moving averages |

| v0.41.0 | 2026-03-01 | fetchPrices on all adapters, Scryfall catalog adapter, sync log recording |

| v0.40.0 | 2026-03-01 | MTGMelee integration, admin scoring panel, fetchPrices adapter |

| v0.39.0 | 2026-03-01 | Admin Telegram alerts, CSP headers, source scoring, login anomaly detection |

| v0.38.0 | 2026-03-01 | Snipe UX, deal filters, volatility view, dashboard security |

| v0.37.0 | 2026-03-01 | Admin source health, arbitrage alerts, payment failure handling |

| v0.36.0 | 2026-03-01 | Data source adapter system |

| v0.35.0 | 2026-03-01 | Anti-abuse detection, alert insights, shareable portfolios |

| v0.34.0 | 2026-03-01 | Smart volatility alerts, per-card frequency, watchlist notes |

| v0.33.0 | 2026-03-01 | Tournament scraping, alert frequency enforcement, login history |

| v0.32.0 | 2026-03-01 | Cardmarket migration, JustTCG integration, data source overhaul |

| v0.31.0 | 2026-03-01 | Saved searches, price alert overlay, portfolio sets widget |

| v0.30.0 | 2026-02-28 | Bulk price check, set collection tracker, user onboarding |

| v0.29.0 | 2026-02-28 | Notification center, market overview, portfolio timeline |

| v0.28.0 | 2026-02-28 | Dashboard overview, daily sync chain, cron consolidation |

| v0.27.0 | 2026-02-28 | Deal finder, price trend badges, portfolio insights |

| v0.26.0 | 2026-02-28 | PWA offline support, price history cron, Vercel deploy fixes |

| v0.25.0 | 2026-02-28 | Next.js Image optimization, input sanitization, PWA support |

| v0.24.0 | 2026-02-28 | Mobile portfolio edit, GA analytics, tournament sync stub |

| v0.23.0 | 2026-02-28 | Search filters, table sorting, compare swap, portfolio editing |

| v0.22.0 | 2026-02-28 | Form validation, type safety, timing-safe auth, query optimization |

| v0.21.0 | 2026-02-28 | Accessibility, CSV security, OG image, polish |

| v0.20.0 | 2026-02-28 | JSON-LD, admin tools, API key UI, tournament expand |

| v0.19.0 | 2026-02-28 | SEO metadata, decklist export, pricing page, sets optimization |

| v0.18.0 | 2026-02-28 | Bug fixes, feature gaps, Cardmarket sync |

| v0.17.0 | 2026-02-28 | Navigation polish, loading states, cron fixes |

| v0.16.0 | 2026-02-28 | Cardmarket, dealer API, notification preferences |

| v0.15.0 | 2026-02-28 | Tournaments, multi-platform charts, affiliate links |

| v0.14.0 | 2026-02-28 | eBay integration, Stripe billing, MVP-1 features |

| v0.13.0 | 2026-02-28 | Sparklines, watchlist bulk actions, tests |

| v0.12.0 | 2026-02-28 | Set pagination, rarity filter, SEO metadata |

| v0.11.0 | 2026-02-28 | Home stats, popular searches, condition filter |

| v0.10.0 | 2026-02-28 | Recently viewed, alert badges, portfolio breakdown |

| v0.9.0 | 2026-02-28 | Image zoom, keyboard shortcuts, portfolio stats |

| v0.8.0 | 2026-02-28 | Share, breadcrumbs, alert presets, 55 new tests |

| v0.7.0 | 2026-02-28 | Help, changelog, loading states, accessibility |

| v0.6.0 | 2026-02-28 | Calendar, decklists, SEO, tests |

| v0.5.0 | 2026-02-28 | Set browser, card comparison, UX polish |

| v0.4.0 | 2026-02-28 | Multi-channel alerts, portfolio, admin |

| v0.3.1 | 2026-02-28 | Post-MVP fixes |

| v0.3.0 | 2026-02-28 | MVP-0 |

| v0.0.0 | 2026-02-28 | Initial commit |