270+
news · YouTube · podcasts · social
1 000-2 000
one strict-format feed
~10 000
weekly newsletter each
Crypto market intelligence - 270+ feeds, one personalised briefing
The crypto market generates more signal in an hour than a serious reader can process in a week. We built a continuous-ingestion pipeline that pulls from 150+ news sites, 50+ YouTube channels, 20+ podcast feeds, and hundreds of social-network groups and influencer accounts; normalises everything into a single labelled, deduplicated, strict-format feed; and produces personalised briefings, push alerts, weekly newsletters, daily AI-generated podcast episodes, and a chatbot that already knows the reader's portfolio context.
270+
news · YouTube · podcasts · social
1 000-2 000
one strict-format feed
~10 000
weekly newsletter each
Last updated:
Information firehose with no editorial gatekeeper
The crypto market is a special case of an information firehose. It runs 24/7 across dozens of jurisdictions, on dozens of platforms, with no editorial gatekeeper. Every minute produces:
- News - published across hundreds of sites of varying quality, frequently duplicated within minutes and frequently rewriting one another.
- Long-form audio and video - multiple flagship YouTube channels and podcasts publish daily; a meaningful trade-affecting move usually surfaces there before it lands in a headline.
- Social signal - token-specific Telegram groups, Discord servers, X (Twitter) influencer threads, Reddit subs. Often where breaking news actually breaks.
A serious crypto reader spends hours per day skimming sources to extract the small set of moves that actually affect their portfolio. Information overload is the problem; lossy "top 10 daily" newsletters are a partial answer; a personalised, multi-format pipeline is the better one.
The Brief
Build it.
270+ feeds in, one strict-format feed out
Ingestion layer (continuous, 24/7)
150+ news sites
RSS where available, headless-browser scraping where not. Per-site adapters handle paywall logic, bot-detection, and rate limits.
50+ YouTube channels
Automated transcript extraction; audio-language detection so non-English channels route through translation downstream; falls back to ASR on the raw audio when no transcript is published.
20+ podcast feeds
RSS for the feed, audio download, automatic speech-to-text on the audio.
Hundreds of social-network groups + influencer accounts
Telegram channels, Discord servers, X timelines, Reddit subs. Read-only ingestion; no engagement signal sent back.
Normalisation pipeline
Once a piece of content lands, it goes through a strict pipeline:
Language detection + translation
Everything ends up in English in the storage layer; the original source text is preserved alongside.
Topic + entity tagging
Extracts the tokens, exchanges, regulators, projects, and people mentioned. Classifies the type of event - price move, regulatory news, exchange listing, security incident, partnership announcement, technical-roadmap update, and so on.
Deduplication
Semantic similarity across the day's input. The same story rewritten on twelve sites collapses into one canonical item with twelve source pointers.
Compression to a standard format
Every surviving item is rewritten to a strict shape: a one-line title, a three-sentence summary, and a small set of bullet points in a standardised order - what happened · why it matters · who is affected · what to watch next. Same shape across news, social, video, and podcast.
Confidence and gravity scoring
Each item gets a confidence score (how many independent sources, how reputable) and a market-gravity score (how big the implied move is). The personalisation layer reads both.
User-facing surfaces
The aggregation app
The reader picks the tokens, themes, and event types they care about. The feed shows only items that pass their filter, ordered by gravity.
Push alerts
When a high-gravity item lands inside the reader's filter, they get a push. The per-user threshold is tuned by the system from what the reader actually opens versus dismisses.
A custom app surface per reader
The home screen reorganises around the reader's portfolio - tokens held, tokens watched, event types they care about. Same backend, very different home screens.
Personalised weekly newsletter
Every Sunday the system produces a newsletter for that reader - a recap of the week's moves that mattered to them, written in the standardised compression format, plus an editorial opener reflecting that reader's interest profile.
Chatbot with portfolio context
A conversational interface that knows what the reader holds, what they've been clicking, and what's been published this week. The most-asked question, predictably, is "Why did X drop so much today?" - and the bot answers from the actual normalised feed plus the source pointers. Citations are not optional.
Five phases - plumbing, normalisation, personalisation, chat, podcast
- Shipped01Phase 1 · Ingestion plumbing
Built the per-source adapters. The hardest cases were YouTube channels with no published transcript (audio download + ASR fallback) and invite-only Telegram channels. Adapters are versioned per source - when a site updates its layout, only that adapter breaks; the rest of the pipeline keeps running.
- Shipped02Phase 2 · Normalisation pipeline
"Topic tagging, entity extraction, semantic dedup, the strict-format compression. The compression step took the most iteration; "title + three sentences + standardised bullets" is easy to specify and surprisingly hard to make consistent across sources at production volume. The output now passes a schema validator before it reaches storage."
- Shipped03Phase 3 · Personalisation layer
Filter language, per-user push thresholds, per-reader home-screen layout, weekly newsletter generation. The personalisation reads from the same labelled feed the public surfaces read from; the difference is purely in which items each reader sees and how they are ordered.
- Shipped04Phase 4 · The chatbot with portfolio context
The reader's holdings, watched tokens, click history, and the past two weeks of normalised items go into the conversation context. The chatbot does not generate market opinions from thin air; it pulls from the feed and cites the source items.
- Shipped05Phase 5 · The daily NotebookLM podcast
Source-material assembly, NotebookLM-driven generation, automated publishing to the podcast feed. The end-of-day workflow runs at the same time every market day; the listener gets a consistent slot.
Friction we hit
- Adapter breakage at scale. With 150+ scraping adapters, something is broken most weeks. Built a self-healing layer that detects drops in expected ingestion volume per source and routes the adapter to a holding queue while the human fixes it; the rest of the pipeline keeps running.
- Translation cost on the long tail. A small number of high-signal sources publish in languages where translation cost runs faster than the rest. The pipeline now front-loads English-first sources and only translates the long tail when a topic clears a relevance threshold.
- The "why did X drop" answer needed source citations to be trustworthy. Early chatbot responses sounded plausible but couldn't be verified. The current bot answers always cite the canonical normalised item plus the contributing sources; "trust me" answers are not allowed.
- NotebookLM-driven audio needed an editorial voice scaffold. Out-of-the-box generation drifts in tone day to day. We built a per-day prompt scaffold that fixes the voice and pacing.
270+ sources in, ~10 000 personalised readers out
A multi-source intelligence pipeline for your domain?
The pattern - many noisy feeds in, one strict-format normalised feed out, personalised reader surfaces on top - works in any market with information overload. Finance, regulatory tracking, supply-chain, OSINT, scientific literature, internal company knowledge. A 30-minute diagnostic walks through your sources, your readers, and what a personalised pipeline would cost.