Skip to content

Prompt to Response: How an AI Request Travels Through the Stack

Prompt-to-Response: AI Infrastructure Flow

You type a question, press Enter, and a second later words start appearing. It feels like one step. It is closer to a dozen, spread across several machines, and each one shapes how fast the answer arrives, what it costs, and whether it is correct.

Understanding that path is not just curiosity. It is the difference between guessing why a feature is slow or expensive and knowing. Almost every practical question about building with AI — why long prompts cost more, why the first word takes longer than the rest, why caching helps, why an agent is slower than a chatbot — has its answer somewhere on this path.

This article walks the full journey of one request, from prompt to response.


You
│ 1. Prompt (text, files, images)
▼
Client app ──────────────► 2. API gateway auth · rate limits · quotas · routing
│
▼
3. Input safety moderation · policy checks · PII
│
▼
4. Context assembly system prompt · history · RAG · tool definitions
│
▼
5. Tokenization text → token IDs
│
▼
6. Scheduler batching · picking a GPU replica
│
▼
7. Inference prefill (read prompt) → decode (write answer)
│ KV cache · sampling
▼
8. Tool calls? pause → run tool → resume (loop)
│
▼
9. Detokenize+stream token IDs → text → your screen
│
▼
10. Output safety filtering · formatting
│
▼
11. Logging & billing tokens counted · traces · metrics

The steps are the same whether you are chatting in an app, calling an API from your own code, or running an autonomous coding agent. What changes is how many times the loop in the middle runs.


What you send is rarely just the sentence you typed. A request is a structured message, typically:

  • A system prompt — standing instructions that set behavior (“You are a support assistant for…”).
  • The conversation so far — every earlier user and assistant turn, because the model itself remembers nothing between requests.
  • Your new message — text, and possibly images, PDFs or audio.
  • Tool definitions — descriptions of functions the model is allowed to call.
  • Parameters — the model to use, the maximum output length, temperature, and so on.

The most important fact on this whole page is hiding in the second bullet: the model is stateless. “Memory” in a chat is the application re-sending the whole conversation on every turn. That is why long conversations get slower and more expensive — each new message makes every future request bigger.


The request first lands on a gateway, the front door that every AI provider and serious in-house deployment runs. Its job is everything that has nothing to do with intelligence:

  • Authentication — is this API key or user session valid?
  • Rate limits and quotas — requests per minute, tokens per minute, monthly spend caps. This is where a 429 Too Many Requests comes from.
  • Routing — which model, which region, which deployment. Large providers run many copies of each model across data centers; the gateway picks one.
  • Request validation — malformed input is rejected here, cheaply, before it can occupy a GPU.

Nothing here is expensive, but it is a real source of failure. A well-built client retries 429 and 5xx errors with exponential backoff, and does not retry 400-class errors that will fail identically every time.


Before the prompt reaches the main model, most platforms run it past one or more safety layers:

  • Content moderation — classifiers that flag clearly harmful requests.
  • Policy checks — organization-specific rules, such as “never discuss competitor pricing”.
  • PII detection — spotting personal data that should be masked or should not leave a boundary.
  • Prompt-injection screening — catching instructions smuggled in through documents or web pages the model will read.

These are usually small, fast models or rule engines rather than the big model itself, so they add milliseconds rather than seconds. In enterprise deployments, this is also where data-residency and compliance rules are enforced.


This is where an application earns its keep, and where most of the engineering in a real AI product lives. Before inference, the final prompt is assembled from several sources:

  • The system prompt and instructions.
  • Conversation history — possibly trimmed or summarized once it grows too long for the model’s context window.
  • Retrieved knowledge (RAG) — for questions about your own documents, the application turns the question into an embedding, searches a vector database, and inserts the most relevant passages. See RAG vs Agentic RAG.
  • Tool and connector definitions — including tools exposed over MCP.
  • User or account context — preferences, permissions, the current file open in an editor.

Two consequences matter in practice. First, the answer can only be as good as the context. A wrong passage retrieved here produces a confident wrong answer later; the model has no way of knowing what it was not shown. Second, context is cost. Every passage, every tool definition and every old turn is billed as input tokens on every request.


Models do not read characters or words. They read tokens — chunks of text drawn from a fixed vocabulary, usually tens of thousands to a few hundred thousand entries. A common English word is often one token; a rare word, a long number, or code may be several.

"Prompt to response" → ["Prompt", " to", " response"] → [15496, 284, 2882]

(The IDs above are illustrative; every model family has its own tokenizer.)

Tokens are the unit of almost everything that follows:

  • Pricing is per token, with output tokens typically costing several times more than input tokens.
  • Context windows are measured in tokens.
  • Speed is measured in tokens per second.

As a rough rule of thumb, one token is about three-quarters of an English word, but non-English text and code often use more tokens for the same content — which is why the same request can cost noticeably different amounts in different languages.


A single GPU serving a large model is expensive, so no provider dedicates one to your request alone. A scheduler places your request on a model replica and batches it with other users’ requests that are in flight at the same moment.

Modern serving engines use continuous batching: rather than waiting for a whole batch to finish, they add new requests and remove finished ones at every step. This keeps the hardware busy and is a large part of why AI inference is affordable at all.

It also explains a phenomenon users notice: the same prompt can be faster at 3 a.m. than at peak hours. During heavy load, requests queue for a slot, and the time before the first token grows even though the model itself has not changed.


This is the part people usually mean by “the AI thinking”, and it has two very different phases.

The model processes all of your input tokens at once, in parallel. GPUs are extremely good at this kind of work, so even a long prompt is read quickly. The output of prefill is the first token of the answer — and the KV cache, described below.

The time this takes is the main component of time to first token (TTFT), the pause before anything appears.

The answer is then generated one token at a time. Each new token depends on every token before it, so this phase cannot be parallelized within a single request. The model runs once per output token, and each run is limited mostly by how fast the GPU can move the model’s weights through memory rather than by raw arithmetic.

This is why:

  • Output is slower than input — and priced higher.
  • Long answers take proportionally longer — a 2,000-token answer takes roughly ten times as long to generate as a 200-token one.
  • Speed is quoted as tokens per second of output.

During prefill, the model computes intermediate values — “keys” and “values” — for every token. Rather than recomputing them for each new output token, it stores them in GPU memory: the KV cache. Each decode step only has to compute values for the single newest token.

The KV cache is what makes long conversations feasible, and it is also the thing that limits them: it grows with every token of context, and GPU memory is finite.

Many providers extend this idea across requests. If many requests start with the same prefix — the same long system prompt, the same reference document, the same tool definitions — the provider can keep that prefix’s computed state and reuse it. Cached input is typically billed at a fraction of the normal rate and skips most of the prefill time.

The practical lesson for anyone building on an API: put the stable parts of a prompt first and the changing parts last. Reordering a prompt so that the system instructions and documents form a fixed prefix can cut both latency and cost substantially.

At each decode step the model does not output a word. It outputs a probability for every token in its vocabulary. A sampler then picks one:

  • Temperature sharpens or flattens the distribution. Low temperature is more deterministic and repetitive; high temperature is more varied and more error-prone.
  • Top-p / top-k restrict the choice to the most likely candidates.
  • Stop sequences and maximum length decide when to end.

This step is why the same prompt can produce different answers, and why setting temperature to zero makes output more repeatable but not perfectly so — batching and floating-point arithmetic on parallel hardware can still introduce tiny differences.

Models with extended reasoning (“thinking”) insert an extra decode phase: they generate internal reasoning tokens before writing the visible answer. Those tokens go through exactly the same decode loop, which is why reasoning improves quality on hard problems but adds latency and cost — every thinking token is generated, one at a time, and usually billed.


A plain chat request goes through inference once. A request that uses tools does not.

When the model decides it needs information or an action — search the web, query a database, read a file, run code — it emits a structured tool call instead of a final answer and stops. The application then:

  1. Executes the tool (the model never runs it directly).
  2. Appends the result to the conversation.
  3. Sends the whole thing back through steps 4–7 for another round of inference.
prompt ─► inference ─► tool call ─► run tool ─► inference ─► tool call ─► run tool ─► inference ─► answer

This loop is what turns a chatbot into an agent, and it is why agents are slower and costlier than single answers: each round re-sends the growing context and pays for another prefill and decode. Prompt caching matters most here, because consecutive rounds share an almost identical prefix. See AI Agents Overview and Spectrum of Agency.

It is also a security boundary. Content a tool returns — a web page, an email, a document — becomes part of the prompt, and anything written in it can try to steer the model. Tools that take real actions should require confirmation or strict permissions, whatever the model says.


Each generated token ID is converted back into text and, in most applications, streamed to you immediately rather than held until the answer is complete — usually over Server-Sent Events or a WebSocket.

Streaming does not make generation faster. It changes what you perceive: instead of waiting several seconds for a complete paragraph, you see the first words within the time-to-first-token and read along as the rest arrives. For any interactive interface, streaming is close to mandatory.

It does complicate the client. A streamed response can fail halfway, so applications need to handle partial output, cancellation when the user navigates away, and reconnection.


As text streams out, it can pass through further checks:

  • Output moderation — catching harmful content the model produced despite its training.
  • Data-loss prevention — blocking secrets or personal data from leaving.
  • Structured-output validation — when the application asked for JSON, checking it actually parses and matches the expected schema.
  • Citation and grounding checks — in RAG systems, verifying that claims are backed by the retrieved sources.

Then the client renders the result: Markdown into formatted text, code into highlighted blocks, citations into links.


Finally — and in parallel — the platform records what happened:

  • Token counts for input, cached input, output and reasoning, which become the bill.
  • Latency metrics: time to first token, tokens per second, total duration.
  • Traces linking each step, so a slow request can be traced to a slow tool, a long queue or an oversized prompt.
  • Quality signals: thumbs up or down, evaluation scores, error rates.

For teams building AI products, this layer is not optional. Without it, “the AI is slow today” or “the answers got worse” cannot be diagnosed. See Evaluation Frameworks.


A typical interactive request, roughly:

Stage Typical share of time What makes it worse
Network, gateway, safety Small Distant regions, extra moderation passes
Context assembly (RAG, tools) Small to large Slow vector search, slow external APIs
Queueing Zero to large Peak load, rate-limit backoff
Prefill → first token Small to moderate Very long prompts with no cache hit
Decode Usually the largest Long answers, reasoning tokens, large models
Tool round-trips Multiplies everything Many sequential tool calls

The useful mental model is: latency ≈ (time to first token + output tokens ÷ tokens per second) × number of rounds. Most optimizations attack one of those three terms.

Cost driver Why How to reduce it
Output tokens Priced highest; generated one by one Ask for concise answers; cap max tokens
Input tokens Every request re-sends the full context Trim history; retrieve fewer, better passages
Repeated prefixes Same system prompt and documents each time Prompt caching; stable prefix first
Reasoning tokens Hidden but billed Use reasoning models only where they help
Agent rounds Each round re-pays for the context Fewer, better-designed tools; parallel tool calls
Model size Bigger models cost more per token Route simple tasks to smaller, cheaper models

See LLM & AI Model Guide for choosing the right model for the job.


“The model remembers our conversation.” It does not. The application re-sends the conversation every time. Memory features are the application storing and re-injecting information.

“A longer prompt makes the answer slower to write.” Mostly it makes the start slower. Prefill is parallel and fast; it is long outputs that dominate total time.

“Temperature zero means identical answers.” More repeatable, not guaranteed identical.

“The model looked it up.” Only if a tool or retrieval step actually ran. Otherwise the answer came from patterns learned during training, which can be out of date or simply wrong.

“The AI ran the code / sent the email.” The model only asks for a tool to be called. The application decides whether to run it — which is exactly where permissions and confirmations belong.


  • A prompt passes through a gateway, safety checks, context assembly, tokenization, scheduling, GPU inference, optional tool loops, streaming, output checks and metering.
  • Prefill reads your prompt in parallel and determines time to first token; decode writes the answer one token at a time and dominates total time.
  • The KV cache and prompt caching are why long contexts and repeated prompts are affordable — keep stable content at the start of the prompt.
  • Tokens are the unit of cost, speed and context limits, and output tokens cost the most.
  • Agents repeat the core loop, which multiplies latency and cost and turns tool output into a security boundary.
  • Quality is decided largely in context assembly: the model can only reason over what it was given.