A toy agent for a small warehouse
The question. When a model is asked a plain-language question about a data warehouse, which written context helps it answer correctly: table and column descriptions, a narrowed set of tables, or a short narrative of how each process moves through the tables?
What I built. A toy agent in Python: one plain-language request in, one SQL query and its result out, against a 12-table warehouse of New York taxi and congestion pricing data, using two small models running locally. Two experiments, 18 requests, 144 scored answers in the second, with a prediction for each committed before the run.
The result. In a single run on the larger model, adding process narratives raised correct answers from 8 to 10 of 18. Table selection made that model faster but less accurate, 5 of 12 against 7. Splitting the work into separate stages did not improve accuracy.
Three kinds of context
The project tests three kinds of context a model can be given.
None of these is new. A described schema is a data dictionary, and asking what one row represents is what dimensional modeling calls declaring the grain. The BIRD benchmark for turning questions into SQL reported large gains when each question came with a written hint about the data: for GPT-4, from about 35% to about 55% of test questions. This project is a small, repeatable test of each of the three.
The setup
I call it a toy agent because it is the smallest piece of one: the model has no tools and no second try. It is the query-writing step of a conversational reporting tool.
- Warehouse: 12 tables of New York taxi, ride-hail and congestion pricing data, stored in Apache Iceberg (an open table format) and queried with Spark SQL. It is the warehouse from my expectation-driven pipeline project.
- Descriptions: every table and column has a written description, stored as a comment in the catalog and exported to one markdown file.
- Models:
llama3.2:3b(about 3 billion parameters) andqwen2.5:14b(about 14 billion), run on a laptop with Ollama, a tool for running models locally. Temperature 0 and seed 0, so a rerun gives the same answers. The context was set to 16,384 tokens and the largest prompt was under 10,000, so nothing was cut off. - Requests: 18 plain-language questions, each with a reference query and a stored reference answer.
- Scoring: an answer counts only if it returns the same rows and values as the reference, with a 0.1% tolerance on decimals.
- Guardrails: generated SQL must be a single SELECT, and is limited to 120 seconds and 1,000 rows.
Who did what. I decided what to build and what to test: a toy agent on small local models, a test of process narratives, and a staged version in which each call starts with no memory of the others. I approved the plan for each experiment before any code was written. Claude sessions wrote the code and drafted the table descriptions, requests, reference queries, narratives and predictions. I reviewed the table descriptions before the first experiment, settling three open meanings and making eight corrections, and approved the other inputs as drafted. In the second experiment the session that wrote the narratives did not have the requests, and the six new requests came from a session that had seen neither the narratives nor any results.
Experiment 1: does table selection help?
A selector, one model call, read an index with one line per table and picked the tables it wanted. A second call wrote SQL from only those tables. That was compared with handing over the whole schema, and with handing over exactly the tables the reference query uses.
Experiment 1: correct answers out of 12
| Model | Whole schema | Selector's tables | Reference query's tables |
|---|---|---|---|
llama3.2:3b | 1 | 4 | 3 |
qwen2.5:14b | 7 | 5 | 5 |
The selector picked well: on the larger model it included the tables the reference query uses on 11 of 12 requests. That did not make the larger model more accurate. It scored lower, though it was faster, at 40 seconds a request instead of 107. The small model scored higher with fewer tables. With 12 requests, neither difference is strong evidence.
On the larger model the same three requests failed under all three methods, including when it was given exactly the right tables. Each depended on meaning: what "inside the zone" means for a trip, how to total a ten-minute block across vehicle classes, and what counts as outside the normal range.
Experiment 2: do process narratives help?
Four narratives, each about a page, follow one thing through the warehouse: a trip, a vehicle's entry into the toll zone, an expectation (a forecast recorded before its data arrives), and a data load. Each names the tables and columns its steps touch. Six new requests were added to the first twelve, for 18 in all.
Four methods were compared:
- Schema only, in one call.
- Schema and all four narratives, in one call.
- Staged, meaning three calls that each start with no memory of the others: one picks which narrative applies, one reads that narrative and writes a plan, and one writes SQL from the plan and the schema of the tables the plan names.
- Staged, with the right narrative supplied in place of the first call.
The staged method: what each call is given
Before the run, a prediction for each of the 144 combinations of model, method and request was committed to the repository.
Experiment 2: correct answers out of 18
Narratives added two correct answers
On the larger model they gained two and lost none. Both gains were among the original twelve requests; on the six new ones the score was 1 with narratives and 1 without. The two requests about the normal range stayed wrong under every method. Two of 18 is a hint and not a finding.
One of the two gains shows what a narrative supplies. The request:
In March 2025, which hour of the day had the most yellow taxi pickups inside the congestion zone, and how many trips was that?
The trip narrative includes this sentence:
A trip is classed only by its two ends: zone_relation is 'both_in', 'pickup_in', 'dropoff_in' or 'neither'.
The same request and model, without and with the narrative
Schema only
WHERE service = 'yellow' AND zone_relation = 'pickup_in'
9 p.m., 52,480 trips. Counts only trips that start inside the zone and end outside it.
Schema and narratives
WHERE service = 'yellow'
AND zone_relation IN
('both_in', 'pickup_in')
6 p.m., 180,134 trips. Matches the reference answer.
qwen2.5:14b. Only the filter differs; the rest of the two queries is identical.Picking the narrative was easy
The larger model picked the intended narrative on 17 of 18 requests, and supplying it changed no score.
Staging did not improve accuracy
On the larger model the staged methods were faster (about 70 seconds a request against 110) and more of their queries ran (15 and 16 of 18 against 12). They answered the same number correctly, and not the same ones: four gained, four lost.
In two of the losses the planning call changed the question, and the SQL call, which sees the plan and not the request, could not repair it. Asked how many yellow taxi trips were picked up on one day, the plan asked for trips by hour, and the SQL returned 24 rows. In the other two, the plan named a table without the column it needed, or invented tables and values. In these runs, the question changed at the handoff.
Across both experiments, the failures were about meaning more than about finding tables.
Limits
- It is small. Twelve tables fit in both models' context, so this does not test table selection at a size where it is hard.
- Each request was asked once, in one wording. The settings make a rerun repeat exactly, so the run says nothing about how answers vary with phrasing.
- The scoring is strict. One answer gave the right shares as percentages instead of fractions and was marked wrong.
- The narratives may be fitted to the first twelve requests. The session that wrote them did not have the requests, but it could read the write-up of the first experiment, which describes what went wrong. The gain appeared only on those twelve.
- The predictions were a model's. A Claude session that had seen the first experiment's results drafted them, and I approved them unchanged. They agreed with the outcome in 118 of 144 cases, but predicting failure everywhere would have scored 109. They predicted 14 of 18 correct with narratives on the larger model; it scored 10.
- The models are small. The smaller one answered correctly once in 72 attempts in the second experiment. Larger models were not tested.
- The first full run was thrown out. A fault in the code let timed-out queries hold the query engine's capacity, which made later queries time out too. The fix restarts the engine after any timeout. The invalid run is kept in the repository, labeled, beside the rerun.
Next
- Let the model retry when a query fails.
- Add a look-alike table, to make table selection hard on purpose.
- Give the selector its past correct selections as examples.
- Test a staged version where the SQL call also sees the request.