2026-05-02
How AI-Powered News Aggregation Works at WorldIntelligence
From RSS ingestion to embedding-based clustering to LLM-written briefings: the pipeline behind every intelligence summary on the site.
The pipeline
- Ingest. Roughly 435 RSS, Atom and JSON feeds are polled on category-specific cadences.
- Normalise. Items are de-duplicated, language-tagged, and stripped of tracking parameters.
- Cluster. Stories from multiple outlets are merged into events using cosine-similarity over sentence embeddings, then thresholded against a recency-decay window.
- Summarise. A free-tier LLM (currently Gemma-3-27B via OpenRouter) produces a 3-paragraph intelligence brief at the cluster level, with full source attribution.
- Translate. Briefings are translated on demand into the seven non-English locales, cached to disk for cost efficiency.
Hard rate-limiting
We never exceed the free-tier rate limit. The OpenRouter client uses a token-bucket algorithm (1 token per 4.2 seconds, capped at 15 tokens/minute, hard-capped at 1000 requests/day) and an exponential-backoff retry loop for transient failures. If the daily quota is exhausted, the site falls back to pre-cached summaries rather than dropping requests.
Why we publish methodology
Intelligence products that hide their pipeline cannot be challenged. Every claim on the site should be traceable to a citable upstream source or to a transparent aggregation step. If it is not, it is a bug — please report it.