PO/AI Engineer - Agent Harness for the Purchasing Datalake
My role and scope
I independently design and develop the agent harness, from defining requirements to building quality controls and evaluating it with business users.
Context
The Purchasing Datalake holds 30,000 data assets in 1,500 tables, 300 schemas and three databases. Its production and preproduction environments connect to around 60 business applications. The agent is intended to help five groups explore this data: data modelers, developers, Business Analysts, Product Owners and clients. The harness connects the language model to analysis tools and business knowledge, with checks on the results.
Requirements and architecture
- Gather requirements from the five user groups and translate them into technical use cases.
- Design the modular architecture and present the choices to Datalake steering. The harness is separate from the language model.
- Define interactions between the orchestrator, analysis modules, SQL tools and business-knowledge sources.
Development and deployment preparation
- Develop Python and PostgreSQL tools to search and catalogue data, analyze mappings and joins, and query business knowledge.
- Automate tool and workflow selection based on the application and requested analysis.
- Develop a single entry point, specialized analysis modules and API/MCP connections for applications.
- Prepare GitOps deployment of the modules, with shared tools, connection orchestration and query serialization.
- Cache stable data and split large datasets for processing.
Quality controls and evaluation
- Define quality controls and evaluate the agent’s capabilities with 19 testers.
- Run comparative acceptance tests, cardinality checks and traceable, reproducible statistical tests.
- Compare processing versions and analyze the impact of ETL transformations.
- Check analyses against existing knowledge and reject unsupported inferences through controls in the orchestrator.
- Record negative feedback and share business knowledge to revise analyses and controls.
Deliverables
- Functional and technical harness architecture and operating documentation.
- Single entry point, analysis modules and SQL tools adapted to the Datalake.
- System for checking and sharing business knowledge.
- API connections and an automated MCP connection orchestrator.
Decision and delivered capabilities
- Modular architecture selected by Datalake steering.
- 30,000 data assets and 1,500 tables available through a catalogue for nontechnical users.
- Lineage tracing across sources, targets, transformations and impacts. Analysis of mappings, joins and data-quality rules.
- Tools for acceptance testing before production releases, with quantified reporting.
The assignment is ongoing, with deployment and application interoperability still in preparation.
Time savings in the business-user pilot
- 85.4%median processing time reduction compared with manual work in the business-user pilot.
- 62.9%mean processing time reduction per case.
- 89.6%weighted aggregate processing time reduction.
- 76.8%weighted reduction excluding the extreme case with an estimated manual duration.
Compared with manual processing in a pilot with business users. The weighted gain compares total durations and is more sensitive to long-running cases.
Consistency of time savings
- 89.5%of timed cases faster than manual processing in the business-user pilot.
- 63.2%of cases achieve time savings of at least 80%.
- 47.4%of cases achieve time savings of at least 90%.
Shares of timed pilot cases, compared with manual processing.
Output quality
- 82%of the maximum qualitative score on average in the business-user pilot.
- 57.1%of cases reach at least 80% of the maximum score on every content criterion.
- 61.9%of cases reach at least 80% of the maximum score for both accuracy and actionability.
- 35.7%of cases receive the maximum score on every content criterion.
Criteria: relevance, accuracy, completeness, comprehension and actionability. The mean aggregates all recorded ratings; it is a normalized score, not an accuracy rate.
Identified novelty
- 30.8%of completed responses to the innovation question identify an innovative case.
Pilot result calculated among responses to this question. The assessment scope differs between time, quality and innovation metrics.