AI systems
LeadRadar
A Threads lead pipeline that survives a crash
- 2
- BullMQ queues
- 24 ч
- score cache TTL
Problem
Scraping and scoring fail in different ways: a Playwright page crash shouldn't lose already-scraped posts, and a transient Gemini 429 shouldn't kill the whole run. A naive loop loses both.
System
Two BullMQ queues on Redis sit between the stages. The scraper (headless Chromium) runs at low concurrency — it's browser- and I/O-bound; the scorer runs high — it's mostly waiting on an API. Each stage retries independently with exponential backoff and survives a process restart, because state lives in Redis, not memory. Scores are cached by sha256(post text) with a 24h TTL, so the same post surfacing under several keywords costs one Redis GET instead of a second billed Gemini call. The niche isn't in source — it's a JSON config resolved from an env path, so switching verticals or running a second instance is configuration, not a deploy.
Role
Solo build, including the queue architecture decisions.
Outcome
Deployed on Railway from a Dockerfile built on Playwright's official image. A relevance threshold gates Telegram delivery, while the cache is filled by every scored post, not just the ones that alert.