In the previous article, the local deployment was apart and talked about:
- Ollama is more suitable for fast local validation
- vLLM is more like a service-based inference infrastructure
- SGLang is more suitable for complex reasoning orchestration
max_model_len、gpu_memory_utilization、tensor_parallel_size、max_num_seqsThese parameters determine the upper limit of service, not just "can it start"
But when many teams are actually online, the problem that they encounter quickly is no longer "whether the model can run", but:
- Why is the single user experience very good, and the queue starts or even times out as soon as the concurrent is on?
- Why does GPU utilization look high, but users still feel slow?
- Why is there no error in the log, but the service is occasionally stuck?
- Why do the same model, the same graphics card, the test environment is stable, but the production environment often shakes?
- Why does the tail delay take a long time to recover after a traffic peak?
Behind these problems, in essence, it is not "the model is not strong enough", but After the reasoning service enters production, the system goal changes.
The local stage is: whether it can run, whether the effect is good or not.
The production stage is:
- Can you hold it at peak
- Delays predictable
- Can the fault be isolated?
- Can the problem be located?
- Can the model service be operated like an engineering system?
This article focuses on this issue.
I want to split the model production service into four layers:
- How to carry high concurrency: throughput, queuing, batch processing, how to balance the scale of instances
- How to monitor: Which indicators can really reflect the health of reasoning services
- How to do the current limit: when to reject the request, not drag the system to death
- How to do failure recovery: how to stop loss as soon as possible after model service jitter, OOM, and node abnormality
If you have gone from "local can run" to "provide APIs to the business for a long time", this article is basically a watershed.
1. Let’s talk about the problem first: the core contradiction in the production environment is not that the model is pushed quickly, but the request is uneven.
Many people start with the reasoning service, and they will use the speed of a single request to judge whether the system is good or not.
For example:
- The first token is delayed within 1 second
- Generate 40 tokens per second
- A certain Benchmark runs well
These are all important, but not enough.
Because the production environment is not a request to enter the system at a uniform speed, but:
- Free time
- Sometimes a batch of requests rush in in an instant
- Some requests are very short
- Some requests are very long and the output is very long
- Some requests also require structured output, tool calls, and multiple rounds of retry
So the most critical issue on the line is not:
A single request is fast.
but:
When the flow fluctuates, the system can still maintain a stable and predictable response.
This will pull several concepts directly to the front desk:
- Throughput
- First Token Delay (TTFT)
- Total time-consuming to request
- P95 / P99 tail delay
- queue length
- Instance Utilization
There are many team problems here:
The reasoning service is regarded as a "more expensive Web interface", but it is not designed according to the queuing system.
Once the model service enters the queuing system perspective, you will find that the strategy has completely changed.
2. High concurrency is not automatically solved by "larger cards", but by scheduling strategies and capacity planning
GPUs are of course helpful, but not decisive answers.
There are usually three things that really determine whether the system can carry concurrency:
1) How many tokens can be processed per unit time
The core resource of the large model service is not the number of requests themselves, but the token processing ability.
Because the cost of a request comes mainly from two paragraphs:
- prefill: Encode the input context into it
- decode: token generates output
These two segments have different resources:
- Prefill is more like a one-time calculation peak, the longer the input, the heavier
- Decode is more like continuous occupation, the longer the output, the longer the system
So don't just look at QPS in production.
It is more secure to see at the same time:
- enter token/s
- output token/s
- Average Input Length per request
- Average output length per request
Otherwise, two "same 10 QPS" services, the system pressure may not be on the order of magnitude.
2) High concurrency is essentially a "long and short request mixed run" problem
The most troublesome is not the high concurrency itself, but the different types of requests come in at the same time.
For example, at the same time on an instance:
- 200 token input, 100 token output light request
- 8K token input, 1k token output re-request
- Complex requests that require JSON structured output, fail and retry
If there is no scheduling isolation, the system will have very typical problems:
- A light request is slowed by a re-request
- Unpredictable queuing time
- Tail delay rises rapidly
- The user feels much worse than the average value
So don't just do "unified model service", but also do it online Request type hierarchy.
Common practices include:
- Short and long requests are routed separately
- Common chat and structured extraction separate instances
- Long context capability to make a special service pool
- High and low-premise tasks use different queues
Many systems are obviously computing power, but because all the requests are mixed together, the experience is finally broken.
3) The upper limit of the concurrency is not set by the head, but is reversed by pressure measurement.
Many parameter names look like "capacity switches", but they are just your choice of risk.
For example:
max_num_seqsmax_model_lengpu_memory_utilizationmax_tokens- batch upper limit
These values are a little larger, and on the surface, the example can accept more requests.
But the reality is often:
- Video memory is more tense
- Long requests are easier to drag the queue
- P99 is more ugly
- OOM is more risky
So the correct order of capacity planning is usually:
- Define the target flow model first
- Then use the real input length and output length for pressure measurement
- Observe TTFT, throughput, P95/P99, OOM
- Inversely launch single-instance security concurrent cap
- Determine the number of instances and expansion strategy
It is not to screw the parameters to the maximum first, and then hope that the system will stabilize by itself.
One of the most important engineering problems in online reasoning is to separate prefill and decode separately
A lot of system tuning is always a misunderstanding:
"The model is slow, so continue to add the machine."
But if you don't distinguish between prefill and decode, many optimizations will actually lose focus.
1) Why does prefill often become a bottleneck in long document scenarios?
When the user stuffs a large number of contexts at a time, the system must first input the entire segment into the KV Cache.
Typical features of this step are:
- Concentration of computing power consumption
- very sensitive to input length
- May cause instant congestion at peak flow
In RAG, long-document question answering, and continuations of long conversations, requests often back up during prefill, before decoding even starts.
This is why many teams will do later:
- chunk control
- context truncation
- Resistance limit
- prompt template slimming
These actions look like "prompt optimizations", but they are essentially doing System Load Control.
2) Why decode is more like continuous system usage
After entering the generation stage, each request will continue to run forward with resources.
If the output length is uncontrolled, there are two problems:
- A single request stays in the system for too long
- There are more and more requests behind the queue
So the line is generally not allowed max_tokens Unlimited release.
A more practical approach is:
- Limit output length by interface type
- Summary interface gives a shorter upper limit
- Report generation interface separates the service alone
- Agent tasks are disassembled by stage, not generated at once
Essentially, max_tokens Not only the content parameters, it is also dispatch parameters.
3) Why do many teams split the long context and ordinary Q&A into two sets of services
The reason is simple:
- Long context instances need more video memory budget
- It is easier to be slowed down by long prefill
- It will compress the concurrent carrying capacity of ordinary Q&A
If all businesses share a set of long window instances, the result is usually:
- Common request costs become higher
- throughput drop
- Deterioration of user body
So a more reasonable solution is often:
- Normal 4K / 8K service pool
- Long text 32K / 64K service pool
- High priority routing if necessary
This is more engineering than "a set of instances to conquer the world".
4. Systems that can truly withstand high concurrency usually do queue, batch and hierarchical routing
As long as the reasoning service enters production, it is almost impossible to avoid these three things.
1) The queue is not an option, but the system buffer layer
Many people have misunderstandings about the queue and feel that a queue is not enough.
Actually not.
Systems without queues are more likely to lose control when encountering burst traffic.
There are three main functions of the queue:
- Absorb burst flow
- Leave the time window for the merge request for the scheduler
- Let the system know how congested it is currently
But the queue is not the longer the better.
If the queue is infinitely stacked, it just changes the failure from "failed immediately" to "failed after a long delay".
Therefore, there are two strategies in the project:
- Maximum queue length
- maximum waiting time
When the threshold is exceeded, you should actively reject it, instead of continuing to drag.
2) The benefits of continuous batching, not just higher throughput
A large part of the value of frameworks such as vLLM lies in the ability to continuously merge requests into batches.
The benefits of doing this are:
- Improve GPU utilization
- Reduce idling
- More cost-effective than single-request serial in concurrent scenarios
But continuous batching also comes at a price:
- Scheduling is more complex
- Long and short requests are more likely to affect each other
- Some extreme requests can damage the overall tail delay
Therefore, many mature systems will not only rely on the default behavior of the framework, but will do it at the upper layer:
- Detach the queue by request type
- Detect the queue by priority
- Separate pool for structured tasks
3) The grading route is more realistic than the "unified portal unified processing"
If your business is already complicated, it is recommended to do grading routing as soon as possible.
For example, at least:
- general dialogue
- Long Document Questions and Answers
- structured extraction
- Agent / tool call
- Offline Batch Tasks
The resource characteristics of these tasks, delay goals, and failure costs are different.
Put them all to the same layer of instances, and save trouble in the short term. In the long run, it will definitely make trouble troubleshooting and capacity planning more and more painful.
5. Monitoring is not to add a few more Dashboards, but to be able to answer "Why is the system slow, where is it slower, and whether it will be slower"
Many teams say that they have done monitoring, but in fact they just drew a few graphics card usage curves.
This is far from enough.
The core of monitoring in the production environment is not "looking a lot of data", but can support judgment and action.
It should be divided into at least four layers.
1) Resource layer monitoring: whether the machine has a problem with the hardware
At least this layer depends on:
- GPU utilization
- GPU video memory usage
- GPU temperature and power consumption
- CPU utilization
- memory usage
- disk io
- Network swallow
Why is this layer still important?
Because a lot of "models are slower", it's not a problem with the model itself at all, but:
- Containers are robbed of resources by other processes
- Disk jitter causes load exception
- Upstream timeout caused by node network problem
- Performance degradation caused by driver or CUDA-related exceptions
2) Service layer monitoring: whether the interface has the ability to provide health
This layer of recommendations focuses on:
- QPS / number of requests
- Success rate
- 4xx / 5xx scale
- Average response time
- P95 / P99 response time
- queue length
- Request timeout
- Active rejection number
Here is a common mistake:
The average delay is very beautiful, which does not mean that the service is really good.
Because what users are most likely to perceive is slow requests, not averages.
Therefore, the tail delay must be watched for a long time.
3) Inference layer monitoring: whether the model service is stuck in prefill or decode
This is the key layer that is different from ordinary APIs with large model services.
It is recommended to at least remove:
- TTFT
- Number of tokens per second
- Average input token length
- Average output token length
- prefill time-consuming
- decode time-consuming
- Batch size distribution
- Number of active sequences per instance
- OOM times
With these indicators, you know:
- Is the input is too long and prefill is slowed down
- Or the output is too long, resulting in decode occupying resources
- It's the batch, it's too small to eat full cards
- Or the batch is too big, but it causes jitter
4) Business layer monitoring: whether the system is fast or not, does not mean that the business is really available
In many cases, the model service itself is not linked, but the business is no longer available.
For example:
- The success rate of tool calls has decreased
- JSON structured output failure rate rises
- The truncation rate of the answer has risen
- A certain type of task retry has skyrocketed
These problems are not visible only by looking at the underlying GPU indicators.
Therefore, it is best to add a layer of business observation to the production system:
- Count success rates by task type
- Statistical quality fluctuation by model version
- Record typical failure samples
- Canary requests for critical links
This is the way to really find that "the system is still alive, but it is no longer easy to use".
6. Current limiting is not conservative, but to protect the system and protect the user experience
Many teams initially excluded the current limit, thinking that the current limit is losing business.
But without limiting the flow, the loss is usually greater.
Because once the reasoning service enters the overload state, the common result is not "everyone is slower", but:
- Queue swells rapidly
- Tail delay out of control
- Increase over time
- Upstream retry to amplify traffic
- Node starts OOM or jitter
- The final whole service is not available
So the flow limit is essentially:
Replace the overall availability with partial rejection.
1) The most basic current limiting: the upper limit of the instance level concurrency
Each instance should have a clear safe bearer boundary.
After exceeding, do not continue blindly.
Otherwise, the system just delays the failure.
Common control points include:
- Maximum active requests
- Maximum queuing requests
- Maximum Token Budget
- Maximum Input Length for Single Request
- Single request maximum output length
2) More practical current limiting: layer by tenant, by business, by priority
If the system serves multiple business parties, unified current limiting is usually not enough.
A more reasonable approach is layering:
- Core business priorities are higher
- Internal test traffic priority is lower
- Offline batch processing do not grab online traffic
- Key customers or key tenants reserved quotas
In this way, even if the system enters a state of tension, it will not be possible for all businesses to fall together.
3) The current limit should be designed together with timeout, fuse, and retry
If there is only a flow limit and no matching timeout and retry strategy, there will also be problems.
Typical bad cases are:
- Model service is slow
- Retry immediately after upstream timeout
- Continue to receive more duplicate requests downstream
- The whole system avalanche
So a complete set of protection mechanisms usually include:
- Define Timeout Threshold
- There is an upper limit index back off retry
- Failed to return quickly
- Suspicious examples of fuses
- Downgrade to a small model or cache answer if necessary
The current limiting is never an isolated switch, but a part of the whole set of protection chains.
7. The key to failure recovery is not "never make mistakes", but "don't spread after mistakes"
After the large model service enters production, the failure will almost certainly occur.
What really matters is:
- Can you find out quickly
- Can you limit the scope of influence?
- Can it be automatically restored to an acceptable state?
1) The most common types of failures
The most common large-model reasoning failures in production, usually include:
- Depletion of video memory leads to OOM
- An instance of a long request is dragged to death
- The inference process is stuck but does not exit directly
- Model weights fail to load or recover very slowly after rebooting
- Node network abnormality causes upstream misjudgment instances to survive
- After a certain version goes online, the tail delay is greatly increased
These failures are rarely as simple as "completely hang up".
What's more troublesome is the semi-failure state:
- Can still take the request
- But obviously slower
- Error rate starts to rise
- The upstream continues to drive traffic
This kind of state is the easiest to drag the entire system to the whole piece.
2) Health checks should be able to distinguish between "alive" and "serviceable"
The health check of many systems is just:
- Is the process there?
- The port is not working
This is only enough to judge "the machine is not dead".
However, the reasoning service needs to judge more:
- Is it possible to return the result normally?
- Is the average time-consuming soaring?
- Whether it has been continuous OOM
- Is the queue too long
So a more reliable health check is usually combined with:
- Lightweight probe request
- Recent window error rate
- Last window end delay
- Current queue backlog
Only in this way can the flow scheduling layer remove the truly unhealthy instances.
3) Recovery strategies need to be layered, not all rebooted
Of course, restarting is useful, but not the only way.
More mature recovery links are generally layered:
- Request level recovery: Timeout, cancel, retry, downgrade
- Instance level recovery: Pick stream, clear queue, restart process
- Node level recovery: Migrate traffic, rebuild containers, switch nodes
- System level recovery: Roll back version, turn off high-cost features, enable degradation model
If all problems can only be restarted by manual SSH, it means that the system is not ready to enter the long-term operation stage.
4) The downgrade mechanism is often more important than "hard carry"
When the system is at high pressure, the downgrade is more valuable than sticking to its full ability.
Common downgrade methods include:
- Change the long context request to the slow queue
- Temporarily reduce
max_tokens - Turn off high-cost structured mode
- From large model to small model
- Return cache or templated results for non-critical functions
The goal of the downgrade is not to make the experience perfect, but to make the system still "available" during abnormal times.
This is very critical in production.
8. How to understand several key parameters during production deployment
The following are the most common parameters on the line, and then explain it from the production perspective.
1)max_model_len
Meaning:
- The maximum context length that the instance allows processing
Production impact:
- Decide on the KV cache Budget
- Affect single instance concurrency
- Directly affect the drag and drop of long requests to the system
Suggestion:
- Do not default to all instances with super long context
- Long text requirements are usually more reasonable to build pools separately
2)max_num_seqs
Meaning:
- Sequence upper limit for instance active processing
Production impact:
- Determine the system concurrent bearer method
- affects Batch behavior, video memory pressure and tail delay
Suggestion:
- Use stress tests to push back, don't pursue the bigger the number, the better
3)gpu_memory_utilization
Meaning:
- GPU video memory ratio that can be used by the inference framework
Production impact:
- Direct impact on OOM risk and crest-tolerant space
Suggestion:
- Production values are generally more conservative than experimental values
- Leave safe borders for sporadic peaks
4)max_tokens
Meaning:
- Maximum build length for single request
Production impact:
- Determines the duration of decode
- Impacts overall throughput and queue recovery speed
Suggestion:
- Settings by Interface, not giving all interfaces the same value
5) Request timeout parameter
Meaning:
- How long can a request be allowed and executed in the system at most
Production impact:
- Decide how long the system will drag the system
- Directly affect the upstream retry behavior
Suggestion:
- Distinguish queuing timeouts and execution timeouts
- After the timeout, there must be a clear return code and downgrade strategy
9. A set of production structures that are closer to reality, what does it usually look like
If you abstract a relatively healthy large-model production system, the common structure will be:
- API gateway layer: authentication, routing, quota, current limiting
- Scheduling layer: queue, priority, instance selection, circuit breaker
- Inference Layer: vLLM/SGLang/Other Inference Services
- Monitoring and log layers: metrics, logs, traces, alarms
- Model governance layer: version management, grayscale release, rollback
In this architecture, the reasoning engine is only the middle layer.
It is not just the model framework itself that really determines the availability of production, but whether it is supplemented at the front and rear ends:
- Is there any entry protection in the front?
- Are there any observations and recovery?
If there is only a reasoning engine, there is no queue, current limiting, monitoring and rollback, the system can easily stay in the "demo service that can run" stage.
10. The three most practical production suggestions
If you are going to officially use the model service for the business, I think the most worth doing are these three things.
Suggestion 1: First dismantle the pool according to the request type, instead of first pursuing a universal pool
General question and answer, long document, structured output, and Agent scheduling, do not mix into one pot.
Once the pool is dismantled, many delays and capacity issues become easier to explain.
Suggestion 2: First look at the tail delay and rejection rate, and then talk about the limit GPU utilization rate
A system with high GPU utilization but poor P99 is usually not "more efficient", but "more brittle".
Production priority should be:
- Stability
- predictability
- recoverability
- And then the limit
Recommendation 3: Be sure to do a stress test with real flow characteristics
Don't just measure the short prompt.
At least cover:
- short input short output
- Long input and short output
- Long input long output
- Peak concurrent increase
- Upstream retry stacking
- Structured output and tool call scenarios
Only in this way, monitoring, current limiting and recovery strategies are not on paper.
11. Final summary: The large model service in the production environment is essentially a queuing system plus a fragile high-cost computing core
Many people do large-model deployment, the first half is concerned with the model, and the second half is the real difficulty is the system.
Because as long as you start providing services to the outside world, the core problem becomes:
- How to absorb the flow fluctuation
- How to isolate long and short requests
- What indicators can really show the health of the system
- When to refuse a request to protect the overall availability
- How to quickly recover after a node problem
So the core conclusion of this article can be condensed into one sentence:
Putting a model on a GPU is only the start of serving it in production. You need to be able to observe and control the inference service and restore it after a failure.
If you do this well, the model service will really go from "can run" to "operation".
Next, continue to advance, this main line will naturally enter the other two problems that are closer to the application layer:
- Why do agents and tool calls redefine the upper-level application schema
- Why does RAG become the default route for many businesses to supplement knowledge blind spots?
At that time, the focus was no longer just "how to stabilize the service", but "how to connect the model capability into the real business".