A Next-Generation Agentic Observability
Platform on Databricks
Executive Summary
EagleEye is a comprehensive data observability platform powered by Aquila, a goal-driven agentic AI, and built natively on the Databricks ecosystem. At the foundation sits Databricks Lakebase (PostgreSQL), which serves as the system of record for metadata, observability intelligence, and persistent agent memory.
This architecture delivers real-time observability, AI-driven root cause analysis, governed auto-remediation, and deterministic agent behavior - all deployed securely using Databricks Apps.
The Challenge: When Spark Wasn't the Right Fit
EagleEye originally used Apache Spark for metadata management. While Spark is exceptional for large-scale data processing, it proved inefficient for transactional observability workloads.
Key limitations:
High operational costs
Underutilized compute
Latency issues
Architectural complexity
We needed a low-latency, high-concurrency, transactional foundation that integrated nativelywith Databricks.
That foundation became Lakebase.
Lakebase as the Unified System of Record
Lakebase now serves as the single source of truth for EagleEye’s entire intelligence layer — including observability data, governance metadata, and Aquila’s agent memory.
The EagleEye application stack (UI, APIs, agent services) is deployed directly on Databricks Apps, with backend services connecting to Lakebase via PostgreSQL drivers.
What EagleEye Stores in Lakebase:
Data Quality Intelligence
- DQ metrics, scores, historical trends
- Validation rules and execution outcomes
- Anomaly detection signals
Cost & Efficiency Analytics
- Cost usage tracking
- Technical debt indicators
- Resource waste analysis
Security & Governance
- Audit logs and access patterns
- Alert rules and notification configurations
- Unity Catalog aligned metadata
Operational Intelligence
- Pipeline executions, scheduler logs
- Databricks workflows health and telemetry
- SLA tracking and failure patterns
Agent Intelligence (Aquila Memory)
- Agent state & checkpoints
- Incident history and root cause outcomes
- Human approvals and action decisions
- Remediation effectiveness over time
Meet Aquila - The Intelligence Behind EagleEye
Aquila is not a chatbot.
It is a stateful, goal-driven AI agent system purpose-built for data operations.
Where traditional tools stop at alerting, Aquila:
- Correlates signals across pipelines, metadata, and lineage
- Identifies root cause with context
- Recommends or executes remediation
- Learns from outcomes over time
Aquila continuously observes pipelines, datasets, quality metrics, lineage, and system telemetry in the background — planning actions and executing them safely when approved.
How Aquila Manages Memory (Powered by Lakebase)
Aquila operates as a stateful agent, not a stateless LLM wrapper. Its intelligence is grounded in structured memory stored in Lakebase.
1. Short-Term Memory (Execution Context)
Used during active reasoning cycles:
- Current issue context
- Affected datasets and pipelines
- Active lineage slice
- Pending approvals
- Live telemetry
Persisted via checkpoints to support retry, replay, and crash recovery.
2. Episodic Memory (Operational History)
Historical intelligence such as:
- Past incidents by table or pipeline
- Previous root causes
- Actions taken and outcomes
- Recurring failure patterns
This allows Aquila to reason:“This failure matches the pattern we saw after the schema drift last month.”
3. Long-Term Knowledge Memory
Structured organizational knowledge:
- Lineage graphs
- Ownership and stewardship
- SLAs and business criticality
- Validation rule catalog
- Metadata from SQL, ADF, Databricks, and ServiceNow
This enables Aquila to reason about business impact, not just technical errors.
Lakebase as the Agent State Engine
Lakebase enables Aquila to function as a durable distributed state machine.
It stores:
- Agent sessions
- Execution checkpoints
- Action plans
- Approval states
- Event logs
This unlocks three critical guarantees:
- Determinism – replay any agent execution
- Recoverability – resume from last checkpoint after failure
- Debuggability – inspect historical reasoning steps
Because Lakebase is PostgreSQL-based, Aquila benefits from:
- ACID transactions
- Concurrency safety
- Temporal queries
- Full auditability
- Enterprise governance alignment
Efficient Memory & Context Retrieval
Instead of loading everything into prompts, Aquila dynamically queries Lakebase to retrieve only relevant slices of memory:
- Impacted datasets
- Related lineage paths
- Last N similar incidents
- Relevant validation rules
This approach:
- Keeps reasoning grounded
- Reduces token usage
- Improves performance
- Scales across thousands of assets
- Minimizes hallucination risk
Lakebase effectively becomes Aquila’s retrieval-augmented intelligence layer.
Deployment on Databricks Apps
EagleEye runs natively on Databricks Apps, strengthening both security and integration.
This provides:
- Native alignment with Unity Catalog permissions
- Workspace-level authentication and governance
- Zero external infrastructure for hosting
- Reduced latency to telemetry and metadata
- Simplified operational model
Aquila executes directly inside the Databricks environment, with proximity to:
- Databricks workflows
- Catalog metadata
- Lineage services
- Monitoring APIs
- Lakebase memory store
Why Lakebase Changed Everything
- Speed- Sub-10ms metadata lookups enable real-time UI responsiveness and fast agent reasoning.
- Concurrency- Supports many users, pipelines, and agent executions simultaneously without degradation.
- Simplicity- Eliminates Spark clusters for transactional workloads.
- Cost Efficiency- Serverless separation of storage and compute reduces total cost significantly.
- Governance Built-In- Unity Catalog integration ensures consistent security and auditability.
AI Capabilities Enabled by Aquila + Lakebase
With this architecture, EagleEye delivers:
- Continuous data quality monitoring
- Real-time observability across pipelines
- AI-driven root cause analysis
- Impact-aware prioritization
- Governed auto-remediation (reruns, rollbacks, rule patches)
- Conversational data operations
- ServiceNow incident/change creation and closure
- Full audit trails for every action
Every decision Aquila makes is grounded in structured memory and is fully traceable.
The Road Ahead
Lakebase and Aquila form the foundation for a new generation of intelligent observability.
Our roadmap includes:
Proactive issue prediction before failures occur
Self-optimizing cost and performance recommendations
Fully conversational operational workflows
Increasing autonomy with policy-based guardrails
This is not just observability.
This is agentic data operations.
