The real cost of building on top of a foundation model

The API cost is the line item that appears in the budget. The engineering time, evaluation infrastructure, monitoring, failure handling, and human review layer are the costs that appear in the post-mortem.

The budget that looks right until you ship.

A founder building an AI-powered B2B product runs the numbers before launch. They estimate monthly active users, model the average number of API calls per session, look up the current token pricing for their chosen model, and calculate a monthly API cost. The number is manageable. They add a margin for error. They feel confident about the economics.

Twelve months later, they are spending three to five times what they budgeted, their gross margin is half what they projected, and none of the overspend is on API calls — the API cost is roughly what they estimated. The overspend is everywhere else: an engineer whose primary job has become iterating on prompts that keep breaking, a contractor reviewing the outputs that users flag as wrong, a monitoring system that was duct-taped together after the first major incident, and a data pipeline that cleans the inputs before they reach the model because dirty inputs produce terrible outputs.

This pattern is repeatable and predictable, and it stems from a single failure of analysis: treating the API cost as a proxy for the total cost of running an AI feature, when in reality it is the smallest and most stable component.

The actual cost structure.

To make this concrete, take an illustrative example: a document intelligence product that takes structured financial documents and extracts specific fields for downstream processing. The figures below are a worked example with round numbers, not data from a specific company — but every line is a cost that teams routinely leave out. Simple description. Complex operational reality.

API costs: Say approximately $1,200 per month at steady state. This is what the original budget models, and it stays roughly where predicted.

Prompt engineering and evaluation: One senior engineer spending around 30% of their time on an ongoing basis, iterating on prompt improvements and running evaluations. Annualised labour cost attribution: roughly $45,000. This is usually zero in the original budget, described as "normal development work" before launch.

Data cleaning and preprocessing: Documents arrive in dozens of formats, with varying quality of OCR, varying languages, and varying structures, so a preprocessing pipeline is needed to normalise inputs before model ingestion. Initial build: several weeks of engineering time. Ongoing maintenance as new document types appear: perhaps 15% of an engineer's time. Annualised: roughly $35,000. Also usually zero in the original budget.

Monitoring: Detecting when output quality degrades requires a monitoring system that samples outputs and flags anomalies for human review. Teams often start with manual spot checks and only invest after an incident — typically a prompt regression that produces subtly wrong extractions for days before anyone notices. A proper monitoring system takes weeks to build. Ongoing operation: infrastructure costs plus periodic engineering maintenance. Annualised: roughly $18,000.

Human review layer: Even at high accuracy, a percentage of outputs needs human review — documents too unusual for the model to handle reliably, high-value extractions where an error would be costly, and the output sample required for ongoing evaluation. Two part-time reviewers. Annualised: roughly $55,000.

Failure recovery: Handling API outages, rate limit errors, and unexpected model behaviour requires retry logic, fallback routing, and a way to communicate with users when things fail. This build is routinely underestimated. Ongoing incidents requiring engineering response: around eight hours per month across the team. Annualised engineering attribution: roughly $20,000.

Total: roughly $174,000 per year, against an API-based budget of roughly $14,400 per year. In this example the ratio is about twelve times, because the workflow is unusually document-heavy. Three to five times is a more common rule of thumb. Twelve times is an outlier. Two times is a floor you should assume even in the simplest cases.

Why the API cost is the wrong anchor.

The API cost dominates the pre-launch cost analysis because it is the only cost with a published rate card. OpenAI's pricing page shows exactly what a million tokens costs. Nobody publishes a rate card for "what it costs to maintain production-quality prompts for a classification task at scale." That cost is real, it is significant, and it is paid in engineering labour — which looks like normal engineering overhead until you add it up.

The other reason API cost anchoring persists is that the hidden costs are genuinely hard to estimate before building. You do not know how much preprocessing your input data will require until you see what your users actually send. You do not know how much your prompts will drift until you have experienced a model update that broke something. You do not know how much human review you will need until you know your failure rate at the quality threshold your customers will accept.

These unknowns are not excuses for not estimating. They are parameters for a range estimate. A reasonable pre-launch cost model for a production AI feature includes: the API cost (known), a prompt engineering labour multiplier of 1.5x to 2x engineering time over what a conventional feature would require (estimated), a human review budget of 2% to 5% of outputs reviewed by a person (estimated), and a monitoring and failure recovery infrastructure budget of 20% to 30% of initial feature build cost (rule of thumb). Run these numbers and the budget comes out in the right order of magnitude.

The human review layer is not optional.

The cost item that surprises teams most is the human review layer, because it feels like admitting that the AI does not work well enough to trust — which feels like a product failure.

It is not a product failure. It is a product design choice that determines whether you can operate in high-stakes domains at all. Every credible AI product operating in consequential contexts — medical, legal, financial, safety-critical — has a human review layer. The question is not whether to have one, but how to design it efficiently.

There are three functions the human review layer serves, and conflating them produces an expensive, poorly designed review operation. The first function is quality assurance: catching outputs that are wrong before they reach the customer. The second is data generation: producing the labelled examples that feed back into prompt improvement and evaluation. The third is edge case capture: identifying the classes of input the model handles poorly, which drives product decisions about what to handle differently.

A review operation that is designed only for quality assurance will be large and expensive because it reviews too much. A review operation designed around all three functions can be smaller and more valuable because reviewers are selected for insight, not just accuracy, and their time produces outputs that reduce the volume of review needed over time.

Monitoring: what "production AI monitoring" actually means.

Software monitoring is well understood. You monitor uptime, latency, error rates, and resource consumption. These metrics tell you whether the system is running. For an AI feature, the system can be running perfectly — zero errors, normal latency, full uptime — while the outputs are degrading in quality in ways that no conventional metric detects.

This is the specific failure mode that catches AI product teams off-guard. A prompt regression produces outputs that are subtly worse — less accurate, less relevant, less compliant with format requirements — without producing any observable system error. The API call succeeds. The response arrives. The output goes to the user. Nobody knows something is wrong until users start complaining, or until a reviewer happens to notice a pattern, or until an evaluation run catches the regression.

Production AI monitoring requires monitoring the outputs, not just the system. This means: automated format compliance checks on every output, statistical sampling for quality review, distribution monitoring for output characteristics that correlate with quality (response length distribution, sentiment distribution, confidence signals where available), and A/B testing infrastructure for prompt changes. Building this infrastructure takes engineering time. Not building it means your quality regressions go undetected.

How to budget correctly from the start.

The cost model that works for pre-launch planning has five line items, each estimated as a range rather than a point estimate.

API cost: model from expected usage with a 2x upper bound for usage overestimation. This is the most predictable line item and the smallest.

Prompt engineering and evaluation: budget 25% to 40% of total engineering time ongoing, not just at launch. Prompt maintenance is not a one-time cost. Model updates, input distribution shifts, and product requirement changes will all drive prompt iteration throughout the product lifecycle.

Data preprocessing: budget an initial build of two to four weeks and ongoing maintenance proportional to the diversity of your input sources. If your product accepts user-generated input in any format, budget at the high end.

Monitoring and failure recovery: budget 20% to 30% of initial feature build cost for infrastructure, plus three to five hours per month of ongoing engineering attention. If your product has more than 1,000 daily active users, this estimate increases.

Human review: estimate your failure rate at your quality threshold, apply a review ratio to that failure rate, and price in a part-time reviewer at the volume where failures per day exceed what your team can absorb. Most products will need dedicated review capacity before they reach 500 daily active users in a high-stakes domain.

• 01
  Never use the API rate card as your cost model. The API cost is 15% to 30% of total cost in a well-run AI product. It is the smallest and most predictable line item. Build a five-line cost model before launch, even if the numbers are ranges with 50% uncertainty. Anything is better than anchoring to the token price.
• 02
  Budget prompt engineering as a permanent ongoing cost. The idea that prompts are written once and maintained thereafter with minimal effort is a pre-production assumption that does not survive contact with model updates, input distribution shifts, and evolving product requirements. Budget 25% of engineering ongoing for prompt and evaluation work.
• 03
  Design the human review layer before you need it. A review operation built reactively after the first quality incident will be expensive and poorly designed. A review operation designed from the start — with clear functions, clear sampling logic, and a clear feedback path to prompt improvement — will cost less and produce more value.
• 04
  Build output monitoring before you have users to lose. A prompt regression that goes undetected for three days in a product with 5,000 users is a much more expensive incident than a prompt regression caught in monitoring before a single user sees the degraded output. Invest in output quality monitoring early.
• 05
  Treat data cleaning as product infrastructure, not a preprocessing afterthought. The quality of your model outputs is bounded by the quality of your inputs. If your product accepts varied input formats and you have not budgeted for a preprocessing layer, you are planning to ship bad outputs and wonder why the model is underperforming.
• 06
  Price your product after costing it honestly. The teams that set prices based on API cost alone and add a margin will be selling at a loss once the full cost structure is visible. Cost the product correctly first, then price for margin. If the numbers do not work at a price customers will pay, that is important information — not a reason to use a flawed cost model.

Frequently asked questions.