Three AI implementation approaches exist. Picking the wrong one is how six months of budget disappears into an evaluation loop that never converges.
For most startup products, prompting solves the problem in 2–5 engineering days. RAG is the right upgrade when answers need grounding in private data: 2–4 weeks of engineering, 1.5–3× the runtime cost. Fine-tuning is the last resort — 4–12 weeks of engineering plus data collection — appropriate only when evaluation proves the first two cannot close the gap. Pick the simplest approach your evaluation shows works.
The comparison at a glance
| Approach | Engineering time | Runtime cost | Best when |
|---|---|---|---|
| Prompting | 2–5 days | Low (pay-per-token) | General capability: writing, reasoning, classification, instruction-following |
| RAG | 2–4 weeks | 1.5–3× prompting | Product needs answers grounded in private or frequently-updated data |
| Fine-tuning | 4–12 weeks + data | Variable; lower per-token, higher fixed | Specific style, format, or domain behaviour prompting cannot reliably reach |
Every bracket up costs a multiple of the bracket before it. That is not a reason to stop at the cheap bracket — it is a reason to measure before you move.
When prompting is enough
Prompting rents the model's general capability. It is the right choice when the product needs the model to follow instructions, apply judgement, or generate prose in a style that prompt engineering can reach in a few iterations. Two to five engineering days to baseline. Hours to iterate.
The mistake is treating "it seems to work" as good enough without measuring. Before you decide prompting has failed, you need an evaluation set — 50 to 100 representative inputs with expected outputs. Build that before you build the feature. Without it, you are guessing in both directions.
When to add RAG
RAG (Retrieval-Augmented Generation) is not a smarter prompt. It is a retrieval layer that feeds the model the right documents before it answers. It is the right choice when the product's value is grounded in private data — a knowledge base, a product catalog, a support library, a document corpus — that the model does not have in its weights and that changes faster than a fine-tune would track.
Engineering cost: 2–4 weeks for a real implementation (chunking strategy, embedding, retrieval, reranking, evaluation harness). Runtime cost: vector database plus inference; roughly 1.5–3× prompting at modest scale. At 50,000 monthly queries, expect $75–$300 per month in infrastructure and inference combined.
We built RAG for LaunchProd — a Carnegie Mellon-founded creator-economy platform that needed answers grounded in a large, evolving content library. The evaluation harness we ran on every retrieval change caught two "obvious improvements" that would have quietly regressed quality. That harness was the part we would have built first. See the full architecture breakdown.
The retrieval quality matters more than the model. A well-tuned retrieval layer with a smaller model routinely beats a frontier model with poor retrieval. Spend the first two engineering days on the chunking strategy, not model selection.
When fine-tuning actually makes sense
Fine-tuning teaches the model a specific behaviour it cannot reliably produce through prompting alone — a consistent domain voice, a structured output format with low error tolerance, or a domain so narrow that the base model's general knowledge is actively wrong.
Four signals that actually justify fine-tuning:
- Prompting plus RAG have been evaluated and cannot close the gap on the specific failure mode.
- The style or format requirement is measurable — you have labelled examples of correct versus incorrect output, not just a subjective sense that the output is off.
- The domain is narrow enough that general-model knowledge is a liability, not a starting point.
- Latency requirements rule out the retrieval step RAG adds.
If those four conditions do not apply, fine-tuning is the wrong answer. In four of four cases over the past year where a founder came to us wanting fine-tuning, the right answer was RAG or better prompting — at a third of the cost and half the timeline. Fine-tuning feels like the serious AI move. A well-evaluated prompt plus a tight retrieval layer usually produces the same outcome faster.
Evaluation is the constraint you cannot skip
The order of the decision depends on what your evaluation shows, not on what sounds technically interesting. Whichever approach you pick, the evaluation harness must come before the feature. Without it, you cannot tell whether a change made the product better or worse.
On LaunchProd, building the evaluation harness first cut the number of retrieval iterations in half. The harness took two engineering weeks. The time it saved was closer to six.
You cannot improve what you cannot measure — and an AI feature that degrades silently after launch is worse than the feature it replaced.
Heuristics
- Start with prompting. Two to five days. Evaluate it. Then decide whether you need more.
- Add RAG when the product needs grounded facts. Not because RAG sounds good — because evaluation shows the model needs context you can retrieve.
- Reach for fine-tuning only after evaluation proves the first two are not enough. Four cases in the past year. Zero fine-tunes shipped.
- The harness comes before the feature. You cannot improve what you cannot measure.
For the broader question of whether your product needs AI at all, see 4 questions before adding AI to your product.
Written 2026-07-12 by Naman Barkiya.