Contents
  1. Why you can’t point an LLM at a broker API
  2. The five-layer architecture
    1. End to end: the voice-to-trade pipeline
  3. Mastra as the orchestrator (L2)
    1. Why Mastra
    2. The orchestrator and its agents
    3. User memory: how the system improves
  4. The neuro-symbolic boundary: DSL and sandbox (L3)
    1. The Strategy Intermediate Representation
    2. The SIR compiler
    3. The sandbox: isolated execution
    4. The explainability engine
  5. The compliance firewall: OPA and the risk engine (L5)
    1. OPA and Rego: compliance as code
    2. The risk engine: dynamic market risk
    3. Mapping SEBI requirements to enforcement
    4. The hard question: black-box algo classification
  6. Backtesting and the data backbone
    1. Deterministic strategy validation with VectorBT
    2. ClickHouse as the analytics backbone
  7. The connector layer: abstracting broker fragmentation (L4)
  8. What I’d do differently, and open questions
    1. Constrained decoding instead of post-hoc validation
    2. The semantic cache and OPA re-evaluation
    3. Scoping the “zero hallucinated tool calls” claim
    4. Where the guarantee stops
    5. A note on the “neuro-symbolic” label
    6. What’s next
  9. Conclusion

Most LLM demos end at “the model called a function.” In regulated finance, that’s where the problems start.

When I set out to build āagman, an AI trading platform for Indian markets, the core question wasn’t whether an LLM could generate a trading strategy. It was whether we could prove to a regulator that a language model never caused a non-compliant trade. The first is an ML problem. The second is a systems design problem, and it’s the subject of this post.

āagman is a SEBI-registered Investment Adviser (INA000021951) and an NSE-empanelled algo provider under NSE Circular Ref. 40/2026. Real money flows through it, and real brokers depend on it. The architecture described here runs in production, and it exists because the alternatives were unacceptable.

The thesis in one line: LLMs propose, deterministic engines decide, and the architecture (not the prompt) enforces the split. This post covers:

  • The failure modes that rule out pointing an LLM at a broker API
  • A five-layer architecture in which the LLM has no network path to the broker
  • A typed Strategy Intermediate Representation (SIR) that’s compiled and run in a sandbox
  • A compliance firewall built from Open Policy Agent (OPA) and a separate risk engine
  • Backtesting, auditing, and broker connectivity
  • Where the guarantees stop

This is the overview. The eight-part series Inside āagman takes it apart one layer at a time, from the strategy language to the execution relay.

Why you can’t point an LLM at a broker API#

Four failure modes shaped every architectural decision:

Hallucinated parameters. A user says “buy 10 Reliance,” and the LLM emits qty: 100. The order is structurally valid and passes any naive schema check. It’s wrong, at 10× the intended exposure.

Prompt injection reaching execution. If the LLM has direct access to the broker API, adversarial input can make it place trades the user never intended. This isn’t theoretical: any system in which a language model holds credentials to a side-effecting API is one injection away from an incident.

No audit trail. SEBI’s algo trading framework requires every algo order to carry a unique identifier, and exchanges keep kill-switch authority over every algo ID. If the execution path runs through an LLM’s unstructured output, you can’t guarantee that order metadata is present, correct, or immutable.

Regulatory liability lands on the broker. Under SEBI’s February 2025 circular on safer participation of retail investors in algo trading, algo providers are agents of the broker. The broker is the principal and carries the regulatory exposure. A broker connecting to your system needs a guarantee, not “the model usually gets it right.”

Prompt engineering can’t fix these failure modes. Architecture can.

The five-layer architecture#

The platform uses a layered microservices architecture that decouples the high-latency work of the AI agents (reasoning and planning) from the low-latency demands of market data processing and trade execution. Each layer enforces a different class of safety property; remove any one, and a specific failure mode becomes possible.

graph TD
    A["L1 · User Surfaces\nIntent Capture · Visualization"] --> B["L2 · Agentic Co-pilot\nMastra AI · Reasoning · Planning"]
    B --> C["L3 · Trading Intelligence\nDSL Engine · Sandbox · Explainability"]
    C --> D["L5 · Compliance Firewall\nOPA/Rego · Risk Engine"]
    D -->|allow| E["L4 · Connector Layer\nBroker Adaptors · Execution"]
    D -->|deny| F["Rejection\n+ Reason Logging"]

    class A,B,C,D amber
    class E green
    class F red
Figure 1: The five-layer pipeline. L5 (Compliance Firewall) sits in the blocking path between L3 (Trading Intelligence) and L4 (Connector Layer), so every broker-bound action must pass through it. Read-only queries exit earlier.

The following table shows what each layer enforces:

Layer Components Core responsibility What breaks without it
L1: User Surfaces Web (Next.js), mobile (React Native), browser extension Intent capture, visualization, voice input No structured intent reaches the agents; raw text hits the planner directly.
L2: Agentic Co-pilot Mastra AI, LLMs, pgvector Reasoning, memory, multi-step planning Complex requests aren’t decomposed. “Hedge my Nifty position with BankNifty calls” becomes one unmanageable LLM call.
L3: Trading Intelligence DSL engine, E2B sandbox, explainability engine Translation, validation, sandboxed execution The LLM’s probabilistic output reaches the broker without deterministic translation, and nothing explains in plain language why a trade happened.
L4: Connector Layer Broker adaptors (Strategy pattern), market data aggregator Protocol normalization, execution Broker-specific fragmentation leaks into the upper layers; a Zerodha error code means something different from an Angel One error code.
L5: Compliance Firewall OPA/Rego (static rules), risk engine (dynamic risk) Deterministic rule enforcement, real-time margin checks A structurally valid but non-compliant order reaches the broker, and position limits, segment restrictions, and kill switches don’t fire.

The key property is that the LLM process holds no broker credentials and has no network path to the broker. Its only output channel is a string that the orchestrator parses. “The LLM only suggests” is a property of the network topology, not an instruction in a prompt.

End to end: the voice-to-trade pipeline#

To make this concrete, suppose a user who is driving speaks into the mobile app: “Buy 50 shares of Tata Motors if it breaks the 15-minute high.” Each layer handles the request in turn:

  1. L1 (User Surfaces). The app’s on-device voice activity detection (VAD) module analyzes speech in 20 ms frames, computing zero-crossing rate and energy. Only when VAD probability exceeds 0.8 does the app open a WebSocket and stream Opus-encoded audio, which cuts bandwidth by more than 90% compared with continuous streaming. A circular buffer keeps the last 1,000 ms so the first consonant isn’t lost.
  2. L2 (Agentic Co-pilot). Mastra Voice transcribes the audio. The intent router, a lightweight distilled model (a fine-tuned Llama-3 8B) rather than the reasoning model, classifies the request as STRATEGY_GEN with a CONDITIONAL_ENTRY subtype in under 50 ms. Before calling the reasoning model, the router checks a Redis semantic cache (cosine similarity above 0.92) and misses. The orchestrator retrieves the user’s context from pgvector (preferences, position sizing, and risk-reward ratio) and builds a multi-step plan: Get_15min_High → Create_Trigger → Execute_Buy.
  3. L3 (Trading Intelligence). The DSL engine turns the abstract plan into a concrete JSON definition: {"symbol": "TATAMOTORS", "trigger": "PRICE > HIGH_15M", "action": "BUY", "qty": 50}. It’s validated against a strict schema, with Zod on the TypeScript side and Pydantic on the Python side. The explainability engine sends a push notification: “I will place a trigger order for Tata Motors at ₹980 (15-min high). I verified this fits your ₹1L limit. Is this okay?”
  4. L5 (Compliance Firewall). The JSON object goes to OPA, deployed as a sidecar to the execution service. The policy engine checks whether the user may trade equity, whether the order value is within free cash, whether Tata Motors is on the ban list, and whether the F&O segment is activated if needed. If everything passes, it returns ALLOW.
  5. L4 (Connector Layer). The user confirms through the push notification. The connector layer selects the right broker adaptor (for example, ZerodhaAdaptor), authenticates with the stored daily token, and places a good-till-triggered (GTT) order. The explainability engine confirms: “I’ve placed a trigger order for Tata Motors at ₹980 (15-min high).”

The whole pipeline, from voice to confirmed order, completes without the user taking their hands off the steering wheel.

Mastra as the orchestrator (L2)#

Choosing the orchestration framework was one of the more consequential decisions in the stack. We picked Mastra, a TypeScript-native agent framework, over LangGraph, CrewAI, and a hand-rolled solution.

Why Mastra#

We chose Mastra for three reasons, in order of importance:

Durable workflows with typed schemas. Mastra defines a workflow as a directed acyclic graph (DAG) in which each step has typed input and output schemas. If an agent emits output that doesn’t match its schema, the step fails before the next step sees it. We needed the orchestrator itself to enforce type boundaries rather than relying on downstream validation alone.

First-class agent separation. Each Mastra agent is a discrete unit with its own system prompt, tool access, and output schema. Instead of one monolithic agent that does everything, you get composable specialists with explicit interfaces. This maps directly to our security model: the strategy agent can’t call execution tools, and the execution agent can’t reason about markets.

One TypeScript codebase. Our backend and Next.js frontend are TypeScript, so running the orchestrator in the same language removes the serialization boundary between the agent layer and the execution layer. When OPA returns a deny, the error propagates through the same type system as the rest of the application. Mastra’s strong typing, from workflow definitions to tool schemas, catches integration bugs at compile time.

The orchestrator and its agents#

The orchestrator decomposes complex trading requests into a DAG of tasks and routes each one to a domain-specific agent. When a user says “Build a strategy to buy Nifty if it drops 1% but hedge it with BankNifty calls,” the orchestrator produces Market_Scanner → Option_Chain_Analyzer → Strategy_Compiler, and Mastra’s workflow engine passes typed state between the steps.

sequenceDiagram
    participant U as User
    participant IR as Intent Router
    participant O as Orchestrator
    participant SA as Strategy Agent
    participant RA as Risk Analyst
    participant OPA as OPA (Sidecar)
    participant EA as Execution Agent
    participant BA as Broker Adaptor

    U->>IR: "Buy 50 Tata Motors if it breaks 15m high"
    IR->>IR: Classify: STRATEGY_GEN
    IR->>O: intent + user context (pgvector)
    O->>SA: Plan: Get_15m_High → Create_Trigger → Execute_Buy
    SA->>SA: Formulate DSL via JSON Schema
    SA->>RA: Validate against portfolio
    RA->>RA: VaR check · sector correlation · margin
    RA->>U: "Confirm BUY 50 TATAMOTORS trigger at ₹980?"
    U->>RA: ✓ Confirmed
    RA->>OPA: DSL + user context + market state
    OPA-->>OPA: Evaluate Rego policies
    OPA->>EA: ALLOW + reasons
    EA->>BA: ZerodhaAdaptor.placeGTT()
    BA->>EA: order_id confirmed
    EA->>U: "Trigger order placed for Tata Motors at ₹980"
Figure 2: The workflow from intent to execution. A lightweight model classifies intent, the strategy agent formulates the plan, the risk analyst validates it, OPA gates execution, and the broker adaptor normalizes the protocol.

Five specialized agents divide the work. (The public site groups the product into four user-facing jobs, charting, research, backtesting and deploy; in the backend the five workers are strategy, risk, backtest, performance and market scanner. Part 3 of the series explains the mapping.)

Strategy agent. Turns natural language into deterministic plans. It drafts strategies, runs backtests with VectorBT, and outputs structured plans against a strict schema. It doesn’t place orders or talk to brokers.

Market scanner. Filters the instrument universe through the market data aggregator. “Find all stocks with RSI below 30” becomes a vectorized scan across the NIFTY 500 that returns structured matches with indicator snapshots.

Risk analyst. Validates every trade against the user’s portfolio before it reaches OPA. It calculates value at risk (VaR), checks sector correlation (“You’re already long on 3 auto stocks; adding Tata Motors increases concentration risk”), and verifies real-time margin requirements against the broker’s API.

Execution agent. Converts confirmed plans into broker-ready payloads through the connector layer, and handles retries, executor reassignment, and failure recovery. It doesn’t reason about markets.

Performance analyst. Analyzes trades after the fact and writes “lessons learned” entries back into the user’s memory in pgvector, so the strategy agent adapts to the user’s preferences over time.

User memory: how the system improves#

The agents don’t operate in a vacuum. A layered memory architecture informs each interaction with the user’s trading personality:

Resource-scoped memory holds permanent facts retrieved in every session: “User is risk-averse,” “User prefers limit orders over market orders,” “User holds 1,000 shares of ITC.”

Thread-scoped memory holds ephemeral conversation context: “We just discussed Tata Steel.”

The RAG pipeline runs when the strategy agent drafts a plan. It queries pgvector for past strategies this user rejected; if the user previously rejected high-beta strategies, that constraint is injected into the prompt context. The system learns what you don’t want, not only what you ask for.

The neuro-symbolic boundary: DSL and sandbox (L3)#

Layer 3 is the core of the system: it translates the LLM’s probabilistic output into deterministic, auditable logic. It exists because of one security constraint: AI never executes code directly. Letting an LLM generate Python for execution is a vulnerability, because prompt injection could lead to malicious code execution.

The Strategy Intermediate Representation#

The LLM never emits free-form text that reaches an interpreter. It emits a JSON object that conforms to a strict schema, which we call the Strategy Intermediate Representation (SIR). Think of it as an abstract syntax tree for trading strategies rather than a config file.

The SIR uses a closed operator set: operations are enums, not free strings; indicators resolve through a registry; and every field is typed and validated. A real SIR looks like this:

JSON
{
  "strategy_id": "gen_uuid_123",
  "trigger_condition": {
    "type": "indicator_comparison",
    "left": {"type": "indicator", "name": "SMA", "period": 50},
    "operator": "CROSS_ABOVE",
    "right": {"type": "indicator", "name": "SMA", "period": 200}
  },
  "execution_logic": {
    "action": "BUY",
    "order_type": "LIMIT",
    "limit_price": "LAST_TRADED_PRICE * 1.001"
  }
}

This is far more expressive than {op: "PLACE_ORDER", qty: 50}: it encodes conditional triggers, indicator logic, and execution parameters. But every element maps to a precompiled, safe function. CROSS_ABOVE maps to a specific math function, and SMA maps to a pinned, deterministic implementation (a pandas rolling mean, with adjust=False for EMA). The LLM selects the blocks; the engine assembles them.

The SIR compiler#

Like source code, the SIR is compiled before it runs. Compilation has four steps:

  1. Schema validation with Zod in TypeScript and Pydantic in Python. The schema sets additionalProperties: false, numeric fields have type and bound constraints, operators are an enum, and symbol is checked against the instrument master rather than accepted as free text.
  2. Default resolution. Fees and slippage default to the product’s cost model (5 bps and 3 bps), session timing is normalized to UTC, and a deterministic sir_hash ensures that identical strategies always produce identical plans.
  3. Enum resolution. Indicator types and operators resolve against the indicator registry. Unsupported features fail fast with clear errors; there are no silent fallbacks.
  4. Compilation to an ExecutionPlan. The SIR compiles into a strongly typed ExecutionPlan made of Pydantic models (UniverseSpec, IndicatorPlan, RulePlan, RiskPlan, and SizingPlan) that the execution adapters consume. The same SIR compiles identically for backtesting and screening; the intent field (BACKTEST | SCREENER) selects the execution path, not the logic.

If validation fails at any step, the request is rejected, not repaired. A retry is allowed, but the retried output goes through the full compiler again. The LLM never patches its own output into the execution path.

The sandbox: isolated execution#

Even with a typed DSL, evaluating triggers (computing indicators on live data and checking cross-above conditions) requires computation. That computation runs inside a sandbox with no escape path.

The DSL interpreter runs in an E2B sandbox with Firecracker as the isolation backend, under the following constraints:

  • No network access. The sandbox can’t reach the internet, the broker, or any internal service.
  • No filesystem access beyond a read-only data volume and a write-only output directory.
  • No environment variables, so there are no leaked credentials and no configuration injection.
  • Deterministic I/O. The sandbox accepts input (market data candles from mounted Parquet files) and returns output (boolean trigger signals and performance metrics). Nothing else.

Even if a strategy is malformed, or under active prompt injection, the worst case is a wrong computation that the sandbox reports as output. It can’t compromise the host, exfiltrate data, or reach a broker API.

For backtesting, the sandbox mounts historical OHLCV data as read-only Parquet files and writes results (a summary JSON file, a trades Parquet file, and an equity-curve Parquet file) to the write-only output directory. After the sandbox exits, the orchestration layer stores the time-series data in ClickHouse and the summaries in PostgreSQL.

graph LR
    A["LLM Output\n(SIR JSON)"] --> B{"SIR Compiler\nSchema + Enums + Defaults"}
    B -->|invalid| C["Reject\n+ clear error"]
    B -->|valid| D["ExecutionPlan\n(Pydantic models)"]
    D --> E{"E2B Sandbox\nNo network · No FS · No env"}
    E --> F["Indicator Registry\nRSI · SMA · EMA"]
    F --> G["Rule Engine\ncrosses_above · comparisons"]
    G --> H{"OPA Sidecar\nRego Policies"}
    H -->|deny| I["Reject\n+ deny reasons"]
    H -->|allow| J["Broker Adaptor\nExecute"]

    class B,H amber
    class E violet
    class C,I red
    class J green
Figure 3: The path from LLM output to execution. Three independent barriers (the compiler, the sandbox, and OPA) can each reject a request, and the default is deny.

The explainability engine#

Users trust the system when it explains why it acted, in plain language rather than JSON.

The explainability engine takes trigger signals (for example, RSI=28, Volume=1.5x_avg) and reverse-parses them into a narrative using specialized prompt templates: “I entered this trade because the RSI dropped to 28, indicating an oversold condition, while volume spiked 20% above the 10-day average, suggesting institutional buying interest.”

Users see this explanation in push notifications and the trade log. It’s also stored in the audit trail next to the raw DSL, so a compliance review can trace both the machine logic and the plain-English reasoning.

The compliance firewall: OPA and the risk engine (L5)#

Layer 5 is the platform’s key differentiator. Layer 2 is designed to be creative; Layer 5 is designed to be obstructive. It’s a deterministic firewall that blocks any action that violates regulatory or safety rules.

The firewall has two components that enforce different kinds of safety:

Component Enforces Type of safety
OPA/Rego Static rule compliance: SEBI regulations, segment restrictions, ban lists, exposure limits Regulatory
Risk engine Dynamic market risk: real-time margin, VaR, sector correlation, kill switch Market risk

OPA and Rego: compliance as code#

Open Policy Agent runs as a sidecar to the execution service. Every API call that places an order must pass an OPA authorization check, and no code path bypasses it. With the sidecar pattern, the execution service can’t reach the broker without OPA’s approval.

An order flows through OPA like this:

  1. The execution service receives a DSL payload: Order(User=Ajit, Symbol=TATASTEEL, Qty=100).
  2. The execution service queries OPA: POST /v1/data/trading/allow {input: {user: Ajit, order: Order}}.
  3. OPA evaluates the Rego policies against the input.
  4. OPA returns {"result": false, "reason": "Exposure limit exceeded"}.
  5. The execution service throws an exception, and the order never reaches the broker.

The critical design choice is that OPA receives its input from trusted code, not from the LLM. Deterministic services assemble the policy input:

  • op and args come from the validated SIR: the LLM’s contribution, after compilation.
  • user.id, user.segment, and user.limits come from the identity service.
  • context.ltp (last traded price), context.market_open, and context.vix come from the market data service.

The LLM never populates trust-bearing fields. If it could, the boundary would be fake.

The following Rego policies map to specific SEBI requirements. They’re illustrative, not our production policies.

Exposure check. SEBI requires brokers to keep clients within exposure limits based on reported net worth:

Rego
package trading.compliance
default allow = false

allow {
    input.user.is_active
    exposure_check
}

exposure_check {
    projected := input.user.current_exposure + (input.order.price * input.order.quantity)
    projected <= input.user.max_exposure_limit
}

F&O segment activation. Users can’t trade derivatives until they’ve uploaded income proof and activated the segment:

Rego
deny[msg] {
    input.order.segment == "FNO"
    not input.user.segments.fno_active
    msg := "Trade Rejected: F&O segment is not activated for this user."
}

Ban list enforcement. No fresh positions are allowed in securities during an F&O ban period:

Rego
deny[msg] {
    input.order.action == "OPEN"
    data.banned_securities[input.order.symbol]
    msg := sprintf("Trade Rejected: %v is currently in F&O ban period.", [input.order.symbol])
}

The defaults matter: OPA timeouts, errors, and undefined results all resolve to deny. The system fails closed.

The risk engine: dynamic market risk#

OPA handles static regulatory rules. The risk engine handles what changes by the millisecond:

Real-time margin calculation. Before execution, the engine queries the broker’s API for the exact margin the order requires (including SPAN margin) and compares it with free cash. If the required margin exceeds free cash, it blocks the trade, avoiding broker-side rejection penalties.

Kill switch. As SEBI guidelines require, the system has a kill switch. If the risk engine detects a logic loop (for example, an algo placing more than 10 orders per second), or the user hits their daily maximum loss, the kill switch has OPA set a global allow = false policy for that user, freezing all activity immediately. This works even if the LLM, the orchestrator, and the interpreter all “want” to execute, because OPA sits in the blocking path and defaults to deny.

Portfolio correlation. The risk analyst agent feeds into this check: “You are already long on 3 auto stocks; adding Tata Motors increases sector concentration risk.” This is a soft warning rather than a hard block, and it’s shown to the user in the confirmation step.

Mapping SEBI requirements to enforcement#

āagman operates under SEBI’s algo trading framework, anchored by the circular on “Safer Participation of Retail Investors in Algorithmic Trading” (4 February 2025). After a timeline extension, the framework became mandatory for all brokers on 1 August 2025. Under it, algo providers are agents of the broker, and the broker carries the regulatory exposure.

Figure 4 shows which layer enforces each requirement:

graph TD
    subgraph "OPA/Rego (static compliance)"
        A["Permitted instruments\nper user segment"]
        B["Exposure limits\n(net-worth based)"]
        C["F&O segment activation"]
        D["Ban list enforcement"]
        E["Market hours check"]
    end

    subgraph "Risk engine (dynamic)"
        F["Real-time margin\ncalculation"]
        G["Kill switch\n(loss cap · order rate)"]
        H["Portfolio VaR\n+ correlation"]
    end

    subgraph "Rate limiter → OPA input"
        I["10 OPS threshold\nmonitoring"]
    end

    subgraph "Interpreter / order gateway"
        J["Algo ID stamped\non every order"]
    end

    subgraph "Infrastructure"
        K["Static IP whitelisting"]
        L["Per-client API key"]
        M["OAuth + 2FA"]
    end

    subgraph "Organizational"
        N["SEBI IA: INA000021951"]
        O["NSE empanelment: 40/2026"]
    end

    class A,B,C,D,E amber
    class F,G,H blue
    class I violet
    class J green
    class K,L,M gray
    class N,O red
Figure 4: SEBI compliance enforcement map. OPA handles static regulatory rules, the risk engine handles dynamic market risk, and the stateful rate limiter feeds its count into OPA as input.

Three of these mappings are worth explaining:

The 10 OPS threshold. SEBI’s framework requires algo orders that exceed 10 orders per rolling second per exchange to be registered with a strategy ID. That’s a stateful computation that needs a counter, not a policy rule. Our rate limiter tracks the count and passes it to OPA as an input field. OPA doesn’t count anything; it checks input.rate.current_ops > 10 and enforces the appropriate tagging.

An algo ID on every order. The interpreter stamps the algo ID on every order action: place, modify, and cancel. OPA adds a secondary check that denies any order whose algo ID is missing or malformed. Belt and suspenders.

An immutable audit trail. OPA decision logs, which record every evaluation’s input, the policy version applied, and the result, feed into ClickHouse. This gives a complete, time-ordered, immutable record of every compliance decision. When a broker has to answer “why did this order execute?” in a regulatory review, the answer is a ClickHouse query, not a post-hoc reconstruction.

The hard question: black-box algo classification#

SEBI’s framework distinguishes white-box algos, whose logic is transparent and replicable, from black-box algos, whose logic isn’t known to the user or can’t be replicated. Black-box providers must register as research analysts and maintain a research report for each algo.

The open question is whether an LLM parsing “buy 10 Reliance” into an order counts as a black-box algo. If the LLM, rather than the user, chooses the strategy or parameters, the logic is arguably not replicable, which raises the research-analyst registration question.

This isn’t settled. Only the circular’s definitions decide it, and a clarification process is ongoing. Our position: when the user explicitly specifies the instrument, quantity, and order type, āagman is an execution tool. When the system generates strategy parameters autonomously, for example from a screener, the classification may differ, and we’ve built the system to support per-algo registration for that case.

Backtesting and the data backbone#

Two parallel pipelines complete the system. One validates strategies before any money is at risk; the other makes every production decision auditable afterward. Both converge on ClickHouse.

Deterministic strategy validation with VectorBT#

VectorBT handles all backtesting. It runs inside the E2B sandbox and reads historical OHLCV data from Parquet files mounted read-only. We pin it to version 0.28.2, because unpinned versions can cause result drift and backtesting needs bit-for-bit reproducibility.

A backtest runs in seven steps:

  1. SIR compilation produces an ExecutionPlan with intent: BACKTEST.
  2. The Parquet path resolver deterministically generates file paths from the universe, timeframe, and date range, for example /parquet/venue=NSE/asset=EQUITY/tf=1m/date=2026-01-03/symbol=RELIANCE.parquet.
  3. The OHLCV loader reads and concatenates the daily Parquet files, sorts by timestamp, enforces the schema (open, high, low, close, volume), sets a UTC index, and applies warmup bars.
  4. The indicator registry computes indicators such as RSI, SMA, and EMA, each pinned to one implementation (for example, pandas.ewm(span=period, adjust=False) for EMA) to prevent variant drift.
  5. The rule engine evaluates conditions (crosses_above, crosses_below, and comparisons) into boolean series at bar close. There’s no eval() and no dynamic code: only whitelisted operators.
  6. The BacktestAdapter shifts entry signals by one bar, so entries fill at the next bar’s open rather than on the signal bar, which is a critical realism constraint. It then applies intrabar stop-loss and take-profit logic, with stop-loss taking priority, and runs VectorBT Portfolio.from_signals().
  7. Serialization writes a summary JSON file, a trades Parquet file, and an equity-curve Parquet file to the output directory.

Golden tests guarantee determinism: fixed Parquet fixtures with hand-verified expected values. The same SIR always produces the same metrics, and any engine change requires explicit test updates and approval. That’s how the system protects user trust.

Screening uses the same SIR but routes it to the ScreenerAdapter. The adapter processes the NIFTY 500 universe with batch-vectorized evaluation: it partitions symbols into configurable batches, pivots OHLCV data from long to wide format, and computes indicators across all columns at once, with no per-symbol loops for the math. Memory is cleared between batches, under a configurable memory limit. This processes more than 500 symbols at more than 10× the throughput of sequential evaluation.

ClickHouse as the analytics backbone#

ClickHouse is more than the audit trail; it’s the time-series backbone for the whole platform. It stores four kinds of data:

Market data. OHLCV time series for equities, commodities (MCX and NCDEX), and F&O contracts, plus mutual fund NAV history. All of it uses ReplacingMergeTree for idempotent ingestion, with ingested_at as the version column.

Backtest artifacts. Time-series results (equity curves and trade logs) from every backtest, stored with the SIR hash for reproducibility.

OPA decision logs. Every compliance evaluation (input, policy version, result, and deny reasons), which together form the regulatory audit trail.

Order lifecycle events. Every state transition of every order, with timestamps and upstream context.

A Parquet export pipeline connects ClickHouse to VectorBT. Each day, finalized bars are exported from ClickHouse to partitioned Parquet on object storage, which the backtesting sandbox mounts read-only. Live writes go to ClickHouse continuously, while the Parquet export writes only the previous day, so files are never rewritten.

graph LR
    subgraph "Data ingestion"
        A1["NSE / MCX / NCDEX\nBhavcopy"] --> CH["ClickHouse\n(ReplacingMergeTree)"]
        A2["AMFI\nMF NAV History"] --> CH
        A3["Broker WebSocket\nLive Tick Data"] --> CH
    end

    subgraph "Parquet export"
        CH --> PQ["Partitioned Parquet\n/venue/asset/tf/date/symbol"]
    end

    subgraph "Backtesting (sandbox)"
        PQ -->|read-only mount| VBT["VectorBT\nE2B Sandbox"]
        VBT --> RES["Results: JSON + Parquet"]
        RES --> CH
    end

    subgraph "Compliance audit"
        OPA["OPA Decision Logs"] --> CH
        ORD["Order Lifecycle"] --> CH
        OTEL["Mastra OTEL Traces"] --> CH
        CH --> API["Explainability API\nfor broker audits"]
    end

    class CH blue
    class VBT amber
    class PQ,API green
Figure 5: ClickHouse is the convergence point. Market data ingestion, Parquet exports for backtesting, backtest results, and compliance audit logs all flow through it.

The explainability API is what makes the system auditable for broker customers. When a broker asks why order X executed at time T, the API returns the full decision chain: the user’s original message, the strategy agent’s output, the parsed SIR, the confirmation step, the OPA evaluation (with policy version and all inputs), and the execution result. None of this is reconstructed; it’s a query over structured, immutable logs.

The connector layer: abstracting broker fragmentation (L4)#

Layer 4 deserves a brief note, because the problem it solves is invisible when it works and catastrophic when it doesn’t.

Indian brokers have no standard API. Zerodha (Kite Connect), Groww, Angel One (SmartAPI), and Upstox all use different authentication flows, order payloads, error codes, and WebSocket formats.

The connector layer implements the Strategy pattern: a generic Broker interface (placeOrder, cancelOrder, getHoldings) with concrete adaptors such as ZerodhaAdaptor and AngelAdaptor. Each adaptor handles three things:

  • Payload normalization: mapping the platform’s canonical order format to the structure the broker expects.
  • Error normalization: mapping broker-specific errors such as “Network Exception” and “Insufficient Funds” to a standard PlatformError enum. This lets the agentic layer understand why a trade failed: for insufficient funds, the agent suggests a smaller quantity; for a network exception, it retries.
  • Authentication management: storing and refreshing daily tokens and handling session expiry.

The market data aggregator keeps persistent WebSocket connections to multiple brokers and market data providers. It deduplicates feeds (if several brokers provide RELIANCE data, it picks the most timely feed), synthesizes ticks into 1-minute, 5-minute, and 15-minute OHLCV candles in in-memory buffers, and publishes completed candles to Redis Pub/Sub channels that the strategy agents subscribe to.

For brokers without public APIs, or users on legacy brokerage plans, the platform provides a Chrome extension that acts as a robotic DOM interface. The extension uses Manifest V3 with signed messaging: the backend digitally signs every execution command, and the content script verifies the signature with a built-in public key before performing a DOM click. This prevents man-in-the-browser attacks in which a compromised web page fakes a command.

What I’d do differently, and open questions#

This section covers what I’d reconsider and what remains genuinely unsettled.

Constrained decoding instead of post-hoc validation#

We currently validate the LLM’s output after generation. Constrained decoding would instead force the LLM to generate only tokens that conform to the DSL grammar during inference. That’s cleaner in theory, since malformed output becomes impossible rather than merely rejected, but it’s harder in practice with hosted model APIs where you don’t control the decoding loop. If we move to self-hosted models, it’s the first change I’d make.

The semantic cache and OPA re-evaluation#

We use a semantic cache (cosine similarity above 0.92, keyed on a market-state hash) to skip the LLM for repeated queries. That raises a design question: should a cache hit on an executable action return the cached DSL and skip OPA? It shouldn’t. Policy depends on live state such as market hours, current position size, and daily loss, so a cached decision would be stale. The cache returns the DSL, and OPA re-evaluates it against current state. This is the correct design, but it means the cache saves LLM latency, not total latency.

Scoping the “zero hallucinated tool calls” claim#

This is the claim I’m most careful with. The architecture guarantees that no malformed or non-compliant action can execute. It doesn’t guarantee that the LLM never proposes a wrong trade. The confirmation step, position limits, and OPA bounds close most of the residual risk, but they don’t eliminate it. Only the scoped version of the claim holds up in a technical review.

Where the guarantee stops#

The hostile question is: “Zero hallucinated tool calls, but what about a valid call with hallucinated arguments?”

The DSL makes invalid operators unrepresentable. It can’t make valid operators correct. If the user says “buy 10 Reliance” and the LLM emits qty: 100, symbol: RELIANCE, that passes the schema, and it passes OPA if 100 shares of Reliance is within the user’s limits.

Three mitigations apply: the confirmation step (the user always sees the parsed order), OPA bounds (position limits and notional caps), and explainability logging (the full chain is traceable). The scoped claim that survives scrutiny is: “The LLM can’t cause an action outside the allowed operator set, and can’t cause an allowed action outside policy limits. Residual risk is a wrong-but-permitted parameter, bounded by limits and caught by the confirmation step.”

A note on the “neuro-symbolic” label#

Classic neuro-symbolic AI integrates symbolic reasoning into learning or inference. This system wraps a neural component in a symbolic constraint layer. The more accurate framing is neural proposal with symbolic verification. The value is in the guarantee, not the label.

What’s next#

The architecture described here is what runs today. Next on the roadmap:

  • More execution algorithms: TWAP and VWAP are live, and iceberg and POV are next.
  • Multi-leg options support, with up to four legs placed in sync.
  • Continuous strategy evaluation in production: running the same VectorBT backtesting engine against live data to detect strategy drift in real time.

Conclusion#

The system rests on one architectural principle: the LLM has no path to the broker. Everything else (the typed SIR, the compiler, the sandbox, OPA as a sidecar, the risk engine, the confirmation step, and the audit trail) follows from taking that principle seriously and closing every gap between “the LLM emitted some text” and “money moved.”

If you’re building agentic systems in regulated domains, the approach generalizes. Separate proposers from executors at the network level. Make the execution boundary typed, policy-gated, and auditable. Accept that the LLM will sometimes be wrong, and design the system so that being wrong is bounded and visible rather than silent and unbounded.

The labels (neuro-symbolic, agentic AI, or anything else) matter less than the properties. Can you prove to a regulator that no non-compliant action executed? Can you show the full decision chain for any order? Can you guarantee that the system fails closed?

If the answer is yes, ship it. If it’s no, fix the architecture, not the prompt.

āagman is built by KotiLabs.