
Who this is for
- Back-end engineers who are ready to launch the model function
- Technical leaders who need to improve stability
What problem does this article solve?
A single API call failure will directly affect the user process, and there is a lack of observable and downgrade policies.
Key concepts
- time out
- exponential withdrawal
- flow limit
- blow
- Demote
Walkthrough
- Encapsulate uniformly call functions
- Set timeout and maximum retries
- Reason for record failure
- Returns a mock prompt if no Key or fails
Python code example
The companion demo is generated in:
cd examples/llm-guides/004_llm_api_error_retry
python demo.py
The example follows two principles:
- From
OPENAI_API_KEY, load the key at runtime to avoid hard-coding it. - Without an API key, the example runs in mock mode so you can follow the workflow.
The essential call looks like this:
import os
api_key = os.getenv("OPENAI_API_KEY")
if not api_key:
print("未配置 OPENAI_API_KEY,进入 mock 模式。")
else:
# 真实项目中在这里初始化 SDK 客户端并发起请求。
print("已读取 OPENAI_API_KEY,可以调用模型。")
Common pitfalls
- Publishing after a single successful demo run leaves out error handling, logging and human review.
- Including API keys, real file paths, account details or business data in published articles or code.
- Readers can struggle to reproduce an example if it is unclear which parts are mocked and which run in production.
- Citing an API or framework feature without checking its documentation can mislead readers when versions change.
Summary
- LLM Guide 04 covers retries, rate limiting, and graceful degradation after API failures. Without visibility into failures or a fallback strategy, a single failed call can interrupt the user’s workflow.
- Make the workflow reproducible, reversible and easy to review. One good generation tells you little about how it will hold up in use.
- Before publishing, check current documentation for model capabilities, API parameters and third-party frameworks.
CSDN tags
Large model, API, Python, Stability, Backend
WeChat article summary
The guide shows how to handle LLM API failures in the example code so that it more closely reflects production conditions.
Alternative title for Zhihu
- Should I directly report an error when the large model API fails?
- How to add retry and downgrade to LLM calls?
- What stability measures should be made to make up for the large model before it goes online?
References and further reading
- Check the OpenAI documentation for your model, SDK and account permissions.
- Python documentation on environment variables, virtual environments and exception handling.
- Consult the documentation for the tools used in the example, such as LangChain, LlamaIndex, Haystack or the vector database.