How I made AI Search “Whatever data you want to see…” for Find Me in Chicago.
In any engineering project cost is an important constraint. Arguably it is the most important constraint and one thing we know about AI inference is it can be expensive. AI requires a lot of compute, high tech manufacturing, water, electricity, networking and thus money to use effectively. To minimize costs the first thing to look are token consumption, cloud infrastructure, and engineering complexity.
AI tokens are not all priced equally. Across AI providers, the difference between the lowest cost model and the highest cost model can easily be an order of magnitude per token. This is not necessarily an apples to apples comparison. The more expensive tokens are theoretically “better” for certain workloads and could be a better value. So examine your workload and ask which tokens are best.
For Find Me in Chicago, I had already decided on Google Gemini as my AI provider and ran basic tests to determine which was most cost effective in terms of quality results. I was surprised to find in my not very thorough testing, that the lowest cost model seemed to give the highest quality results. I assume the reason was a mix of decreased Hallucination and increased Determinism or I just didn’t measure it right.
Cloud infrastructure is often cheap compared to AI tokens. Regardless of the model chosen I can execute straightforward requests on Azure or AWS or Google Cloud for a fraction of the cost of a single AI prompt. Any work I can move off my AI prompt and into my cloud infrastructure is going to save money.
The trade-off to that is moving things from the AI prompt to the cloud infrastructure is going to increase engineering complexity and engineers are not cheap either. Engineering complexity also compounds over time with increased maintenance, debugging, and operational overhead after development is complete. If you save a penny per request, how many requests will it take before you make back all the time and money you spent optimizing the request. As Donald Knuth said: “Premature optimization is the root of all evil.”
For Text-to-SoQL, the cost isn’t simply about reducing the number of tokens, or choosing the cheapest token. It is about finding the right balance between AI, cloud infrastructure and engineering effort. Towards that end, I started with the simplest thing that could possibly work. The cheapest tokens, the simplest infrastructure and the least complex engineering. In practice, this meant using the latest Google Gemini Flash Lite model available at the time, serverless functions, and following the YAGNI principle of “You Ain’t Gonna Need It.”
So to start no RAG (via a vector database or whatever), just one shot with no agentic feedback loops or query verifications. It either works to “Just tell me the data you want to see…” in one shot or it doesn’t.