Cover illustration

Date:2026-07-06
Who this is for: Graduate students, end-side AI/inference system engineers, technical readers who pay attention to the productization of small models
search date:2026-07-06

Abstract

2025 to 2026, Small Language Models (SLM) and end-side AI The discussion has obviously shifted from "can you put a large model into a mobile phone" to "how to design a set of local intelligent systems that can be managed, update and evaluated". There are three simultaneous changes behind this: First, the family of Models such as Gemma, Qwen, Llama, Phi, and SmolLM continues to release 0.5b to 4b Level of dense model or sparse activation model, emphasizing distillation, multilingual, long context and equipment side adaptation; second, INT8/INT4, Weight-only Quantization, KV cache Optimization, 1-bit/low bit research and dedicated Runtime expose bottlenecks other than "parameters"; third, Apple Foundation Models, ExecuTorch, ONNX Runtime Frameworks such as GenAI, Llama.cpp, MLC/WebLLM, MLX, etc., let local inference go from research demo to the application stack.

The core judgment of this article is that SLM is not a cheap alternative to a large model, but another system form. Its value comes not only from low cost, but also from low latency, offline availability, privacy boundaries, personalization, real-time edge control, and cloud-based strategy. The real competition point is not just the score of a small model on the open list, but the model, data distillation, quantitative format, runtime kernel, hardware backend, update mechanism, permission isolation and evaluation of whether playback can form a stable closed loop.

Contents

  1. Research background: Why the small model has become the main line again
  2. Nearly a year's roadmap: from Compact LLM to the local Agent
  3. Representative paper and system group interpretation
  4. Comparing the methods
  5. Technical trends
  6. Implications for deployment
  7. Limitations and open questions
  8. Questions to watch
  9. Summary
  10. References

1. Research background: Why the small model has become the main line again

In the past two years, the leap in model capacity has been mainly driven by larger parameters, larger data and longer training. But when LLM enters the real product, simply enlarging the model does not solve all the problems. Mobile keyboard completion, conference abstract, photo retrieval, offline translation, wearable assistant, car interaction, browser AI, enterprise intranet tools, industrial edge equipment and other scenarios have hard constraints on delay, power consumption, privacy, network availability and unit cost. A 70B or 400B level model, even if it is more capable, cannot be called in real time in all interactions.

The chances of SLM come from this constraint. It splits the "stronger" into "strong enough, fast enough, cheap enough, private enough, and controllable." For many tasks, users do not need general research-level reasoning, but only need to stably complete a few high-frequency actions: extract schedules, summarize notifications, generate short replies, rewrite tone, explain pages, execute local shortcut commands, retrieve in device files, and convert voice into structured operations. If the small model can be started or streamed back at the 100-300 ms level, and the original private data is not uploaded, the product value will be very direct.

Another change is that data distillation and post-training make small models no longer just "training models that are not big enough". The small-size version, SmolLM, and MobileLLM of the Phi series, Gemma/Qwen/Llama, etc. have all explained that the small model capabilities are increasingly dependent on high-quality data, teachers models, synthetic instructions, inference trajectories and task formulas. In other words, the core problem of SLM is not "there are few parameters, so the ability is poor", but "what knowledge and behavior are most worthy of being pushed in when the parameter budget is limited".

At the same time, the end-side AI has amplified the inference system problem. Cloud services can dilute costs with batch processing, tensor parallel, PagedAttention, continuous batching and large GPU memory; Throttling, battery, application sandbox, model download packages, incremental updates, different operating system APIs, and privacy compliance. The study context of SLM must therefore look at the model structure, compression, runtime and product strategy at the same time.

2. Roadmap for the past year: from Compact LLM to local Agent

2023-2024: The small model has changed from "parameter compression" to "data recipe". The Phi-2/Phi-3, MobileLLM, TinyLlama, Distinyllama, DistilGPT work emphasizes high-quality data, architecture selection and training budget. Meta positioned the 1B/3B text model as an Edge/Mobile scenario in the official release of Llama 3.2; the title of Microsoft's Phi-3 technical report emphasizes "locally on yours" phone”. The key word at this stage is: the small model can be available on local tasks, but the behavioral ability must be obtained through strict data selection and post-training.

2024-2025: End-side runtime and quantization become infrastructure. Projects such as llama.cpp/GGUF, MLC/WebLLM, ExecuTorch, ONNX Runtime GenAI, Apple MLX, etc. Distribute models, low-bit privileges, Kernel, KV cache And the hardware backend is made into a tool chain that developers can use. The research side continues to advance compressed routes such as SmoothQuant, AWQ, GPTQ, QLoRA, KIV, and 1-bit LLM. The focus of this stage is not "can run once", but "can run stably, measurable and upgradeable on different equipment".

2025: The small model family began to be released simultaneously with the large model family. Qwen3 official blog lists the DENSE/MoE series from 0.6B to 235B-A22B, emphasizing the Thinking/Non-Thinking mode and multi-language; Gemma 3 released 1b/4b/12b/27b, where 1b is clearly a lightweight text model; the Gemma 3n document is further mobile-oriented, mentioning MatFormer, Per-Layer Embeddings and 2B/4B Effective parameter scale. SmolLM3 shows the positioning of the 3B-level open source model on long context, multi-language and reasoning.

2026: The end-side model starts to move from "local Q&A" to the "local Agent infrastructure". The Google AI for Developers page has listed new generation Models such as Gemma 4 1b/4b/12b/27b and Gemma 4n E2B/E4B; Apple Foundation Models framework exposes local model capabilities to application Developers; browsers, local IDEs, privacy-sensitive enterprise applications and consumer electronic devices start to use small Models as Intent Router, Local Planner, privacy filters, draft generators, and preprocessors before cloud calls. Many of the product availability and regional limitations are changing very quickly here, and should be verified again with official documentation before release.

SLM and end-side AI deployment stacks

3. Representative paper and system group interpretation

3.1 Small Model Family: Ability from Parameters, Data and Distillation Co-Design

The Phi series is the classic entry for understanding SLM. The Phi-3 technical report believes that through high-quality network data and synthetic data training, the small-size model can obtain strong language ability under the constraints of mobile devices to deploy. Phi-4-mini-reasoning narrows the goal further to mathematical reasoning, emphasizing the distillation of high-quality mathematical reasoning trajectories from Phi-4, and for educational and mobile scenarios. The engineering inspiration here is: small models should usually not pursue "everything is not bad", but should make data formulas around the distribution of target tasks.

The 1b/3b model of Llama 3.2 represents another path: the large model family maintains the simultaneous maintenance of small models as edges and mobile device members. Meta officially released that the 1B/3B model supports local Summarization, Instruction Following and overwriting class tasks, and cooperates with mobile hardware partners for optimization. Its significance is that the small model is no longer a by-product of the community secondary tailoring, but the formal deployment goal in the design of the model family.

Qwen3 shows wider dimension gradients. The official blog lists Dense models such as 0.6B, 1.7B, 4B, 8B, 14B, 32B, and MoE models such as 30B-A3B and 235B-A22B. For end-side AI, the 0.6B-4B interval is particularly important, because they allow products to route between "local fast processing" and "high-quality processing in the cloud". Qwen3 also emphasizes the switching of Thinking Mode and Non-Thinking Mode, which is critical to the edge system: not all requests are worth the calculation of at inference time.

The official documentation of the Gemma 3/Gemma 3N/Gemma 4 places the "device side" in a more conspicuous position. The Gemma 3 release page says that the 1b model can run in low-resource environments; the Gemma 3n document is clearly oriented to Mobile/Edge, using MatFormer and Per-Layer Embeddings, allowing The E2B/E4B model is close to the valid parameters of 2B/4B in memory usage; the Gemma 4 document lists 1B to 27B model cards and 4N versions as of 2026-07-06. It's not a single benchmark that deserves attention here, but Google puts the small model, mobile terminal and multimodal capabilities on the same product line for maintenance.

SmolLM3 is a 3B-level model worth watching the open source community in 2025. Hugging Face's official blog describes it as Smol, Multilingual, Long-Context Reasoner, and provides open weights. It reminds us that the 3B-level model is not only toy Demo, but if the data, training recipes and reviews are serious enough, it can become one of the default candidates for local developers, browsers and lightweight agents.

3.2 Architecture and training: small models are not uniformly reduced large models

The contribution of MobileLLM lies in the systematic study of how the Sub-Billion parameter model is optimized for on-device use cases. The paper emphasizes that the direct scaling large model architecture is not necessarily suitable for small parameter intervals, and it is necessary to rebalance the Depth, Width, Attention, Embedding and FFN configurations. For engineering readers, this means that the "same 1b parameter" is not equivalent; the architecture allocation significantly affects latency, memory and accuracy .

MatFormer is another representative idea: embed multiple granular sub-models in the same trained model, so that different calculation paths can be selected according to the device ability or request difficulty. The Gemma 3n documentation mentions the use of the MatFormer architecture, which shows that the resizable model structure begins to enter the official end-side model. The future end-side system may no longer just download a fixed model, but a model family that can be dynamically cropped by power consumption/delay/quality.

MoE is also inspiring for small models, but be cautious. The MoE version of Qwen3 and many cutting-edge models use sparse activation to improve parameter efficiency; on the end side, MoE can make "the total parameters are large, and the activation per token is less" is possible. But it also introduces Expert Routing, weight paging, storage package, and cold start issues. For mobile phones or embedded devices, low Active Parameters does not automatically mean good experience, and weight loading and memory bandwidth are often more deadly.

Small models increasingly rely on teacher models and synthetic training data. Phi, the DeepSeek-R1 distilled models, and smaller versions of Qwen and Gemma illustrate how a large teacher can transfer reasoning traces, tool-use formats, conversational style, and domain knowledge to a student. The student can also inherit the teacher’s mistakes, refusal boundaries, language preferences, and characteristic hallucinations. Evaluate the small model itself on the tasks it will perform on the device. The teacher’s reported capabilities cannot establish the student’s performance.

3.3 Quantization and compression: the real bottleneck is often the memory bandwidth

The first thing you encounter with end-side deployment is memory. A 3B parameter model even uses FP16 to weigh memory about 6GB, and does not include KV cache, runtime buffer, tokenizer, application itself and operating system pressure. INT8/INT4 Weight-Only Quantization can significantly reduce the bandwidth of the package body and memory, and is the basic tool of the end-side LLM.

GPTQ, AWQ, SmoothQuant, QLoRA and other routes promote low-bit deployment from the perspective of post-training quantization, Activation-aware weight protection, activation smoothing and low-rank adaptation. For the product team, the problem is not "which quantitative paper is the best", but: whether there is a corresponding kernel in the backend of the target device, whether the quantization format can be supported by runtime, and under the long context KV cache Whether the memory is exploded, whether the low bits destroy the structured output, function calls and security policies.

KV cache is another often underestimated end bottleneck. Weight weighting only compresses model parameters, long dialogue, multi-round Agent, RAG and local file retrieval will quickly increase the KV cache. KIVI and other work focus on KV cache quantization, indicating that memory management and weight quantization are equally important in long context and streaming interactions. If you need to keep multiple rounds, you must design digests, truncations, cache multiplexing and cloud bottoming, rather than infinite heap contexts.

1-bit/very low bit LLM is a very active direction in 2024-2026. The BitNet B1.58 paper proposes to press the LLM weight to three-valued/1.58 bit that the goal is to significantly reduce memory and energy consumption; Microsoft's BitNet.cpp project tries to give 1-bit LLM provides an efficient CPU inference framework. It should be noted that this type of route is still evolving rapidly, and the benefits of different models, tasks and hardware need to be measured, and the ideal throughput in the paper cannot be directly regarded as a product commitment.

3.4 Runtime: A large part of the "model capability" of the end-side AI is in runtime

The influence of llama.cpp and GGUF is to turn on-premises LLM inference into something that the majority of developers can operate. It supports multiple quantization formats and CPU/GPU backends, and community model distribution also forms an ecosystem around GGUF. For graduate students and engineering teams, it is a common baseline to quickly verify "whether a model is available on this machine".

ExecuTorch represents a systematic investment in the Edge/On-Device Inference on the PyTorch ecosystem. It faces mobile and edge devices and integrates model export, graph optimization, delegate/backend and runtime. For the enterprise team, the value of this type of official runtime lies in long-term maintenance, equipment coverage and connection with the training framework, not just a single performance.

ONNX Runtime GenAI provides another cross-platform path, emphasizing the use of ONNX Runtime to run the Generative AI model and support multi-hardware backends. It is suitable for teams that already have ONNX/Windows/Azure/Enterprise Deployment Ecology. MLC LLM/WebLLM brings the browser and WebGPU into the end-side AI vision, making it possible to "private inference without the need to install local applications". Apple MLX and Core ML/Apple Foundation Models are more biased to the Apple device stack.

The reality here is that there is no uniform runtime that will win all scenarios. Browsers, iOS, Android, macOS, Windows, embedded Linux, car systems, and NPU SDK have different constraints. A more secure approach in engineering is to define the model intermediate format, quantization strategy, evaluation set and fallback contract, and then select Runtime according to the target platform.

3.5 Product form: The local model is often the front layer of the cloud model

The end-side AI doesn't mean "all things are done offline". The more realistic product form is layering: the local small model is responsible for low-risk, high-frequency, privacy-sensitive, low-latency tasks; the cloud-model model is responsible for complex inference, cross-document synthesis, long-chain tool execution and high-risk output; use policy router in the middle Decide whether to upload, what summary to upload, and whether to need user confirmation.

Apple Foundation Models Framework is the representative of this trend. The official documentation opens the on-device Foundation model capabilities to the application, so that developers can call the model in the device to complete tasks such as text generation and structured output. The Apple research page also introduces Apple Intelligence's Server and On-Device Foundation Language Models, emphasizing the adaptation of device constraints. Due to the rapid changes in the availability of regions, system versions and APIs, the specific calling capabilities need to be based on the latest developer documentation.

Local Agents also need permission governance. A local model that can read clipboards, emails, albums, files and calendars is not naturally safe because "no cloud uploads". It requires minimal permissions, user-visible tool calls, data retention policies, revocable memory, sensitive information filtering, and Prompt Injection protection. The privacy value of the end-side AI comes from the system design, not the model location itself.

4. Method comparison table

Route Representative work/system When to use it Advantage Risks and Attention Points
0.5B-4B dense SLM Phi、Qwen3 0.6B/1.7B/4B、Gemma 1B/4B、SmolLM3 Local summary, rewriting, classification, light weight Q&A Small body, low latency, easy deployment Complex reasoning and limited upper limit of long context
Small size MoE/sparse activation Qwen3 MoE, end-side variable calculation idea Cloud-edge collaboration, variable computing power equipment High parameter efficiency, ability/cost adjustable Weight loading, routing and memory management are complex
Distillation Small Model PI-4-mini-reasoning, DeepSeek-R1 distill class route Mathematics, Code, Formatting Tasks, Vertical Fields Ability to push the teacher behavior to a small model Inherit Teacher Errors and Prejudice
INT8/INT4 quantization GPTQ, AWQ, SmoothQuant, GGUF quantization Most end-side LLM deployments Reduce memory and bandwidth, ecological maturity Structured output and safety boundaries need to be retested
KV cache Quantization/Management KIVI, long context runtime optimization Multiple rounds of dialogue, local RAG, Agent Relieve long context memory pressure May affect long-range consistency
1-bit/very low bit BitNet b1.58、BitNet.cpp CPU/Low power consumption exploration, future hardware collaboration High potential energy consumption and bandwidth benefits Ecological and model coverage still needs to be measured
Browser/local runtime WebLLM、MLC LLM、llama.cpp、ExecuTorch、ONNX Runtime GenAI、MLX Cross-platform local reasoning Reduce cloud dependencies and facilitate privacy scenarios Fragmentation at the back end, the performance is very different
Cloud edge collaboration router Apple/Google/Enterprise Private Architecture Product-level AI Assistant Privacy, quality and cost Policy, permissions, and log governance are complex

5. Key technology trends

Trend 1: Small model family. The leading model provider no longer only publishes one maximum model, but maintains multiple points such as 1B, 3B, 7B, 30B, MoE, and end-side multimodal at the same time. In the future, the product team will be more like choosing a database or search engine: look at the ecology, license, runtime, update frequency and stability, rather than just the rankings.

Trend 2: The end-side AI moves towards the "task-specific cerebellum". Intent Classifier, privacy filter, summary model, embedding model, visual encoder, speech model, and cloud router may exist on a device at the same time. They do not necessarily share the same large model, but consist of multiple small models Pipeline.

Trend 3: Quantization format becomes a fact interface. GGUF, ONNX, Core ML, ExecuTorch Export, MLX Checkpoint and other formats will affect whether the model can be circulated. If the research paper only gives FP16 weights, but there is no mainstream flow version and end-side evaluation, there is still a distance from the real deployment.

Trend 4: NPU/CPU/GPU back-end difference determines the experience. The device side is not an abstract GPU cluster. Certain NPUs are sensitive to specific OPs, Shapes, Precision and Batch Size; CPUs may be more stable but power consumption; GPUs may compete for resources with UI rendering. The model design must be viewed with Kernel/Backend.

Trend 5: Local memory and personalization will become the main selling point of SLM. Small models may not be able to answer the world's knowledge problems, but it can understand user habits, files, agendas, and context locally. This capability must be designed with a deletable, viewable, exportable, and isolated memory system.

Trend 6: The evaluation has shifted from a general list to device playback. A model is a few points higher on MMLLU or GSM8K, which may not necessarily be able to output JSON with low latency, low power consumption and stable output on a certain mobile phone. The end-side AI needs more fixed equipment, fixed quantization, fixed runtime, fixed temperature and real task log playback.

6. Inspiration of the project landing

First, start with the task definition, not from the model size. First, the tasks are divided into four categories: local must be done, local priority, cloud priority and prohibition of automation. For example, notification summary, short text rewriting, privacy filtering and simple intent routing are suitable for local; cross-document complex inference, medical legal advice and high-risk decisions are more suitable for cloud or manual confirmation.

Second, build a separate evaluation harness for on-device use. Draw test inputs from de-identified samples of real device logs and common UI states. Include offline use, poor connectivity, long conversations, low batteries, and thermal limits. At a minimum, measure time to first token, tokens per second, peak memory, energy use, crash rate, the fraction of structured outputs that are valid, refusal rate, and the fraction of requests that fall back to the cloud.

Third, don't just evaluate the FP16 model. If the final product is using the INT4/GGUF/Core ML/ONNX version, it is necessary to evaluate the security, format, tool calls, long context and multi-language performance on this version. Quantization may not significantly affect the average score, but will destroy the boundary case.

Fourth, take the model package and update mechanism as a product problem. Mobile apps need to consider first download, incremental update, weak network recovery, version rollback, user storage space and regional compliance. For enterprise intranet equipment, model signature, supply chain scan and offline distribution should also be considered.

Fifth, design a clear Cloud Fallback Contract. The local model should know when it is "uncertain" "need more context" "must ask for consent upload". What uploaded to the cloud is not necessarily the original data, it can be a abstract, feature, candidate draft or a user-confirmed fragment.

Sixth, put permissions and audits into the Agent tool layer. The local model requires visible permission boundaries when calling calendars, files, albums, clipboards, browser history, and messages. Prompt Injection attacks also exist locally, especially when models read web pages, PDFs, mail and shared documents.

Seventh, match the small model with a "narrow and hard" tool. SLM may not be as good as a large model in free generation, but it can be very reliable on vertical tasks if it is matched with templates, schema, retrieval, rule validation, lightweight sequencing and limited tools. The focus of the project is to constrain the output space, rather than let the small model chat naked.

7. Limitations and Disputes

Small models are not automatically more secure. The local operation reduces the risk of data transmission, but it may also make the error output more difficult to be monitored by the server. Security policies, red team samples, sensitive information detection and permission control are still necessary.

The end-side benchmark is difficult to compare horizontally. Models, quantization, runtime, device temperature, system load, prompt length, and sample parameters can all affect the results. Speed numbers in papers or blogs are often not directly migrated to your product.

Compression changes model behavior. Low-bit quantization, pruning, distillation, and KV cache compression may only lose small on average indicators, but abnormalities occur on structured output, rare languages, and long-tailed security requests. Task-level regression must be done before release.

The boundary between the small model and the cloud model leads to privacy disputes. Users may think that "AI is running locally" means that all data is not out of the device, but the product may actually call the cloud at complex requests. Interface, privacy policy and technical architecture need to be consistent.

Open source weights do not equal commercially available deployments. License, training data sources, model terms of use, patents, and third-party runtime licenses are all verified. This article is only a technical inventory and does not constitute legal advice.

8. What's next worth noting

  1. Will the end-side multimodal small model become the default entry? The small text model is already available, and the next step is local vision, voice, screen comprehension and multimodal tool calls.
  2. Can the variable calculation model mature? MatFormer, EARLY EXIT, SPECULATIVE DECODING and DYNAMIC Routing all point to "Different computing power by task difficulty".
  3. How to personalize the small model for a long time? Local LoRA, Adapter, Memory Store, user feedback and privacy deletion mechanisms may become new infrastructure.
  4. Will the in-browser AI change the Web application architecture? The WebGPU/WebNN and WebLLM class projects make the lightweight AI capabilities eliminating the need for server-side inference, but also bring about model download and device differences.
  5. How to prevent Prompt Injection on the end-side Agent? When the local model reads mail, web pages and files and can call system tools, the security boundaries are more complex than normal chat.
  6. Can the 1-bit/very low-bit route cooperate with the hardware? If the model, Kernel and chip are optimized together, new breakpoints may appear in the energy consumption curve of the SLM.

9. Summary

The value of Small Language Models and On-device AI is not to prove that "Small Models can also defeat large Models", but to put AI capabilities into the complete system between the cloud model and the complete system between local devices. The Small model is responsible for fast, private, low-cost, offline high-frequency tasks; the large model is responsible for complex inference and high-risk tasks; runtime, quantization, hardware backend and strategy Router decide whether the system can work stably.

For researchers, it is worth noting that small model architecture, distillation data, low-bit representation, KV cache management and device-level evaluation. For the engineering team, the most pragmatic path is to start with a high-frequency, low-risk, and verifiable local task, choose a small model, a quantitative format, a runtime, a real device evaluation set, and then make up for the cloud Fallback and authority governance. Only when these components are verified together can the small model really change from "minified LLM" to a deliverable end-side intelligent infrastructure.

References

The search date is 2026-07-06. Model availability, official documentation version, license, benchmark ranking, hardware support matrix and API area limitations change rapidly, and it is recommended to manually verify before release.

  1. Microsoft Research. “Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone.” arXiv, 2024. https://arxiv.org/abs/2404.14219
  2. Microsoft. “Phi-4-Mini-Reasoning: Exploring the Limits of Small Reasoning Language Models in Math.” arXiv, 2025. https://arxiv.org/abs/2504.21233
  3. Meta AI. “Llama 3.2: Revolutionizing edge AI and vision with open, customizable models.” Official blog, 2024-09-25. https://ai.meta.com/blog/llama-3-2-connect-2024-vision-edge-mobile-devices/
  4. An Yang et al. “Qwen3 Technical Report.” arXiv, 2025. https://arxiv.org/abs/2505.09388
  5. Qwen Team. “Qwen3: Think Deeper, Act Faster.” Official blog, 2025-04-28. https://qwenlm.github.io/blog/qwen3/
  6. Google. “Gemma 3: Google’s most capable open AI model.” Official blog, 2025-03-12. https://blog.google/technology/developers/gemma-3/
  7. Google AI for Developers. “Gemma 3n model overview.” Official documentation, accessed 2026-07-06. https://ai.google.dev/gemma/docs/gemma-3n
  8. Google AI for Developers. “Gemma 4 model card.” Official documentation, accessed 2026-07-06. https://ai.google.dev/gemma/docs/core/model_card_4
  9. Hugging Face. “SmolLM3: smol, multilingual, long-context reasoner.” Official blog, 2025. https://huggingface.co/blog/smollm3
  10. Jianwei Yang et al. “MobileLLM: Optimizing Sub-billion Parameter Language Models for On-Device Use Cases.” arXiv, 2024. https://arxiv.org/abs/2402.14905
  11. Han Lab et al. “AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration.” arXiv, 2023. https://arxiv.org/abs/2306.00978
  12. Elias Frantar et al. “GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.” arXiv, 2022. https://arxiv.org/abs/2210.17323
  13. Guangxuan Xiao et al. “SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models.” arXiv, 2022. https://arxiv.org/abs/2211.10438
  14. Tim Dettmers et al. “QLoRA: Efficient Finetuning of Quantized LLMs.” arXiv, 2023. https://arxiv.org/abs/2305.14314
  15. Zirui Liu et al. “KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache.” arXiv, 2024. https://arxiv.org/abs/2402.02750
  16. Shuming Ma et al. “The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits.” arXiv, 2024. https://arxiv.org/abs/2402.17764
  17. Microsoft Research. “BitNet.cpp: Efficient Edge Inference for Ternary LLMs.” GitHub repository, accessed 2026-07-06. https://github.com/microsoft/BitNet
  18. ggml-org. “llama.cpp.” GitHub repository, accessed 2026-07-06. https://github.com/ggml-org/llama.cpp
  19. PyTorch. “ExecuTorch documentation.” Official documentation, accessed 2026-07-06. https://pytorch.org/executorch/stable/
  20. Microsoft. “ONNX Runtime GenAI.” Official documentation, accessed 2026-07-06. https://onnxruntime.ai/docs/genai/
  21. MLC AI. “MLC LLM documentation.” Official documentation, accessed 2026-07-06. https://llm.mlc.ai/docs/
  22. Apple. “Apple Intelligence Foundation Language Models.” Machine Learning Research, accessed 2026-07-06. https://machinelearning.apple.com/research/apple-intelligence-foundation-language-models
  23. Apple Developer Documentation. “Foundation Models framework.” Official documentation, accessed 2026-07-06. https://developer.apple.com/documentation/foundationmodels