Beyond Rules-Based Surveillance

Trade surveillance has the same problem as insurance fraud detection: rules-based systems generate too many false positives. Compliance analysts drown in alerts for legitimate market-making activity while actual manipulation patterns slip through.

Why rules fail at scale

Modern trading happens across multiple venues, instruments, and time zones. A spoofing pattern might involve orders placed on three different venues within milliseconds. Rules that check "order cancelled within 500ms" generate thousands of false hits from legitimate HFT activity.

The Lakeflow data engineering stack

• Lakeflow Connect ingests trade data from OMS/EMS systems via managed connectors
• Spark Declarative Pipelines handle streaming trade ETL (streaming tables for real-time ingestion, streaming to Delta Lake)
• Lakeflow Jobs orchestrate batch scoring and surveillance workflows

Streaming ML plus batch inference

The surveillance pipeline:

1. Real-time ingestion: Streaming tables capture trade data in Delta Lake in real time.
2. Pattern detection: Databricks ML models detect anomalous patterns. ai_classify categorizes alerts by type (spoofing, layering, wash trading).
3. Batch screening: ai_query in SQL or PySpark processes bulk trade volumes for periodic comprehensive scans. This is critical for surveillance workloads that need to screen massive volumes cost-effectively.
4. Investigation: Agent Bricks coordinates an Investigation Agent that analyzes flagged patterns in context, cross-referencing trader communications and order history.
5. Reporting: A Reporting Agent uses ai_gen to draft regulatory filings (MiFID II, Dodd-Frank) with evidence trails.

All models on Databricks

Every model needed, whether for classification, analysis, or report generation, is available through AI Gateway on Databricks Model Serving. For high-throughput screening, pre-deployed models (e.g. databricks-meta-llama-3-3-70b-instruct) provide cost-effective inference without external API dependencies.

Unity Catalog governs compliance rules as UC functions, trade data access controls, model governance, and the full audit trail.

Results

Surveillance moves from T+1 (next-day batch) to real-time. False alerts drop by 60% because ML models are more nuanced than rules. Regulatory report drafts are generated automatically with evidence trails. AI Gateway handles rate limiting and payload logging for the complete audit trail.