Contents
  1. Background: why announcements are hard
  2. From proof of concept to production
  3. The production design
    1. Stage 1: ingest
    2. Stage 2: extract text
    3. Stage 3: extract fields
    4. Stage 4: classify
    5. Idempotent stages through the market-data service
  4. The backfill
  5. How research uses it
  6. What I’d do differently
  7. Conclusion

āagman’s research agent answers questions like “how did promoter holding change across private banks over the last year?” or “which NIFTY 50 companies announced buybacks this quarter?” Answering those well depends less on the model than on the documents behind it. For Indian listed companies, the primary record of almost everything material is the stream of corporate announcements filed with the exchanges: results, board meeting outcomes, shareholding patterns, insider trading disclosures, dividends, acquisitions, and a long tail of everything else.

This part describes the pipeline for that stream. It covers:

  • Why announcements are hard to ingest
  • From proof of concept to production
  • Four idempotent stages
  • Classification, extraction and the cost cap on the LLM fallback
  • The backfill
  • How the output feeds research

This part describes the design and its trade-offs. It doesn’t include request-level details of how the exchange website is accessed. If you build something similar, read the exchange’s terms of use and keep request rates conservative. The terms of use were a review item before go-live.

Background: why announcements are hard#

Each announcement is a small metadata record (symbol, a description, a timestamp) plus, usually, a PDF attachment. The difficulty is in the details:

  • The category is unreliable. The same event appears under different descriptions across companies. “Outcome of Board Meeting” can contain quarterly results, a dividend, a fundraise, or all three.
  • Attachments vary. Text PDFs, scanned PDFs and files that fail to parse.
  • Volume is skewed. Most announcements are routine. A small minority carry most of the information value.
  • Corrections and revisions. A company files results, then a revised version. Both exist, and only one is current.

The cost structure follows from that skew. Fetching metadata is cheap. Downloading and parsing an attachment is expensive, and running an LLM over text is more expensive again, so the design treats every model call as a budget line.

From proof of concept to production#

The proof of concept proved the hard part: reliably reading the exchange’s announcements API at a polite rate, parsing timestamps, normalizing PDF links, extracting text, and classifying announcements with rules and regexes across the planned categories. That code was worth keeping.

It couldn’t go to production as it was, because it didn’t follow the platform’s own patterns. The production version fixes that:

In the proof of concept In production
The orchestrator wrote to Postgres directly The orchestrator posts to the market-data HTTP API, which owns the writes
Table names from an AI-generated PRD Two tables in the market-data migrations
PDFs uploaded to object storage that didn’t exist yet PDFs on a mounted volume
One raw SQL file, run once, with no history Tracked migrations with review for schema changes

The production design#

The pipeline has four stages, each idempotent so re-running is always safe. Prefect triggers them, and each stage writes through a market-data endpoint.

graph LR
    A["1. Ingest\nmetadata + PDF"] --> B["2. Extract text\nPyMuPDF, then pdfplumber"]
    B --> C["3. Extract fields\nregex first"]
    C -->|low confidence| L["LLM fallback\ncapped per day"]
    L --> C
    C --> D["4. Classify\nrule-based, 13 tags"]
    A -->|fails| Q["Dead-letter\nqueue"]
    B -->|fails| Q
    C -->|fails| Q

    class A,B blue
    class C,D amber
    class L violet
    class Q red
Figure 1: The stages. Each stage is idempotent and writes through the market-data service. The LLM fallback has a hard daily cap, and failures go to a dead-letter queue for manual review.

Stage 1: ingest#

A flow calls the announcements API for a date window, downloads the PDF to a mounted volume, and posts a metadata row to an upsert endpoint. The row is unique on the symbol, the date and the attachment key, so rerunning a window is idempotent. The volume path includes the symbol and a content hash.

Stage 2: extract text#

PyMuPDF first, with pdfplumber as the fallback. The extracted text is stored on the announcement row (the first part of it, to bound row size) along with the extraction method and a character count, so a poor extraction is visible.

Stage 3: extract fields#

Fields are typed and depend on the category: for a dividend, the amount per share, the type, the record date and the ex-date; for earnings, revenue, profit and earnings per share; for shareholding, the promoter, FII, DII, public and pledged percentages; for a buyback, price and size; for bonuses and splits, the ratio and dates; for order wins, the value and the client; for mergers, the target and the deal value; for insider trades, the name, relationship, value and shares. Every row carries an extraction confidence of high, medium, low or failed.

A regex pass runs first. Only low-confidence rows go to an LLM, and the number of model calls per day is capped by an environment variable that defaults to zero, meaning off. This is the cost-control idea in one setting: an uncapped fallback across a large backfill could cost hundreds of dollars, so the cap must be enabled on purpose per run.

Stage 4: classify#

Classification is rule-based and assigns one of 13 tags: earnings, board outcome, dividend, buyback, stock split, bonus, shareholding, rights issue, mergers and acquisitions, order win, guidance, insider buy and insider sell. Anything not matched stays UNCLASSIFIED. The 13 were a priority list, with other categories left to a second phase. The acceptance test for the classifier was a manual review of a sample of rows with under 10% wrong tags.

The shape matters: classification comes after extraction, by rules, and there is no small model deciding whether to download an attachment. The principle is the one in the title: decide cheaply what a document is before paying to read it. In the plan the cheap decision is the regex and rule pass, and the expensive one, the LLM, is capped.

Idempotent stages through the market-data service#

Each stage is its own flow that reads and writes through endpoints, which gave the pipeline three properties:

  • Independent reprocessing. Improving classification means re-running one stage over existing rows.
  • Visible state. “How many rows have no extracted text?” is a SQL query.
  • Isolation of failures. A bad attachment fails one row. A dead-letter queue collects failed rows for review by hand.

The backfill#

The exchange’s announcements go back to roughly 2005, and the API takes at most a 90-day window per request. So the backfill walks the history in 90-day sliding windows for every instrument, with an estimated run of three to seven days, and a daily incremental flow at 20:00 IST. The acceptance criteria were concrete: over 500,000 rows after the backfill, and three consecutive trading days of the daily flow with no errors.

The backfill covers the NIFTY 500 for the last 20 years, and it starts with the NIFTY 50. I review the output of each stage before widening the universe, because a failure on fifty symbols is debuggable and a failure on two thousand is a fog. Progress is checkpointed, so a run resumes from the last completed window. It is the same pattern as the OHLCV side of the platform, which keeps a per-symbol checkpoint in Postgres: the last successful timestamp for each provider, symbol and timeframe.

The risks were known up front: the exchange can block automated access or change the API without notice, a per-process rate limiter doesn’t hold across multiple workers (so NSE flows run on a single worker), the shareholding endpoint needed its own handling, the PDF volume needs a backup policy, and automated access needs a terms-of-use review before go-live.

How research uses it#

The pipeline’s output is meant for two kinds of question:

  • Structured facts answer quantitative questions directly. “Promoter holding change across private banks” is a query over the structured fields, not a retrieval problem.
  • Text answers qualitative questions: “what did the company say about the acquisition?” The extracted text is stored per announcement, and the research agent retrieves it with hybrid search (Part 3).

Every finding should link back to its source announcement, and when a structured field and the text disagree, the structured field should win and the disagreement should be logged, because it usually means an extraction bug.

What I’d do differently#

Version the taxonomy from day one. The tag list grows as you learn what users ask about, and adding a tag means reclassifying history. Stamping every label with the taxonomy version that produced it makes that targeted instead of a full rerun.

Measure extraction quality with a labeled set per category. Confidence flags and a manual review of fifty rows are a start. A small hand-labeled set per category, scored on every extraction change, would make improvements measurable instead of anecdotal.

Conclusion#

The announcements pipeline is a cost-control design as much as a data design: use cheap, inspectable methods first, cap the expensive one, and give every failure somewhere to go. That principle runs through the rest of āagman, from the intent router to the screener. The next part moves from information to money: the pre-trade risk gates the blueprint put in front of every order.

I built āagman as Founding Software Engineer at KotiLabs, July 2025 to April 2026.