// personal project · experiment

A toy agent for a small warehouse

Bobby Palmer · November 13, 2026 · A personal project on public data

The question. When a model is asked a plain-language question about a data warehouse, which written context helps it answer correctly: table and column descriptions, a narrowed set of tables, or a short narrative of how each process moves through the tables?

What I built. A toy agent in Python: one plain-language request in, one SQL query and its result out, against a 12-table warehouse of New York taxi and congestion pricing data, using two small models running locally. Two experiments, 18 requests, 144 scored answers in the second, with a prediction for each committed before the run.

The result. In a single run on the larger model, adding process narratives raised correct answers from 8 to 10 of 18. Table selection made that model faster but less accurate, 5 of 12 against 7. Splitting the work into separate stages did not improve accuracy.

Three kinds of context

The project tests three kinds of context a model can be given.

1The described schemaTables, columns, types, and a sentence on each.
2Table selectionOnly the part of the schema that matters for this request.
3The process narrativeHow a process moves through the tables: what event creates each row, what changes it, and what a term means here.

None of these is new. A described schema is a data dictionary, and asking what one row represents is what dimensional modeling calls declaring the grain. The BIRD benchmark for turning questions into SQL reported large gains when each question came with a written hint about the data: for GPT-4, from about 35% to about 55% of test questions. This project is a small, repeatable test of each of the three.

The setup

I call it a toy agent because it is the smallest piece of one: the model has no tools and no second try. It is the query-writing step of a conversational reporting tool.

Who did what. I decided what to build and what to test: a toy agent on small local models, a test of process narratives, and a staged version in which each call starts with no memory of the others. I approved the plan for each experiment before any code was written. Claude sessions wrote the code and drafted the table descriptions, requests, reference queries, narratives and predictions. I reviewed the table descriptions before the first experiment, settling three open meanings and making eight corrections, and approved the other inputs as drafted. In the second experiment the session that wrote the narratives did not have the requests, and the six new requests came from a session that had seen neither the narratives nor any results.

Experiment 1: does table selection help?

A selector, one model call, read an index with one line per table and picked the tables it wanted. A second call wrote SQL from only those tables. That was compared with handing over the whole schema, and with handing over exactly the tables the reference query uses.

Experiment 1: correct answers out of 12

ModelWhole schemaSelector's tablesReference query's tables
llama3.2:3b143
qwen2.5:14b755

The selector picked well: on the larger model it included the tables the reference query uses on 11 of 12 requests. That did not make the larger model more accurate. It scored lower, though it was faster, at 40 seconds a request instead of 107. The small model scored higher with fewer tables. With 12 requests, neither difference is strong evidence.

On the larger model the same three requests failed under all three methods, including when it was given exactly the right tables. Each depended on meaning: what "inside the zone" means for a trip, how to total a ten-minute block across vehicle classes, and what counts as outside the normal range.

Experiment 2: do process narratives help?

Four narratives, each about a page, follow one thing through the warehouse: a trip, a vehicle's entry into the toll zone, an expectation (a forecast recorded before its data arrives), and a data load. Each names the tables and columns its steps touch. Six new requests were added to the first twelve, for 18 in all.

Four methods were compared:

The staged method: what each call is given

Given Given Given the request one line per narrative the request the chosen narrative the plan schema of the plan's tables 1. Pick a narrativenew call, no memory 2. Write a plannew call, no memory 3. Write SQLnew call, no memory name plan not given the request returns one SQL query
The plan has five parts: tables, filters, measure, grain and definitions.

Before the run, a prediction for each of the 144 combinations of model, method and request was committed to the repository.

Experiment 2: correct answers out of 18

Each bar is out of 18 requests, asked once. The full bar width is 18.

Narratives added two correct answers

On the larger model they gained two and lost none. Both gains were among the original twelve requests; on the six new ones the score was 1 with narratives and 1 without. The two requests about the normal range stayed wrong under every method. Two of 18 is a hint and not a finding.

One of the two gains shows what a narrative supplies. The request:

In March 2025, which hour of the day had the most yellow taxi pickups inside the congestion zone, and how many trips was that?

The trip narrative includes this sentence:

A trip is classed only by its two ends: zone_relation is 'both_in', 'pickup_in', 'dropoff_in' or 'neither'.

The same request and model, without and with the narrative

Schema only

WHERE service = 'yellow'
  AND zone_relation = 'pickup_in'

9 p.m., 52,480 trips. Counts only trips that start inside the zone and end outside it.

Schema and narratives

WHERE service = 'yellow'
  AND zone_relation IN
      ('both_in', 'pickup_in')

6 p.m., 180,134 trips. Matches the reference answer.

Model qwen2.5:14b. Only the filter differs; the rest of the two queries is identical.

Picking the narrative was easy

The larger model picked the intended narrative on 17 of 18 requests, and supplying it changed no score.

Staging did not improve accuracy

On the larger model the staged methods were faster (about 70 seconds a request against 110) and more of their queries ran (15 and 16 of 18 against 12). They answered the same number correctly, and not the same ones: four gained, four lost.

In two of the losses the planning call changed the question, and the SQL call, which sees the plan and not the request, could not repair it. Asked how many yellow taxi trips were picked up on one day, the plan asked for trips by hour, and the SQL returned 24 rows. In the other two, the plan named a table without the column it needed, or invented tables and values. In these runs, the question changed at the handoff.

Across both experiments, the failures were about meaning more than about finding tables.

Limits

Next