Local Language Models are back on top
How open-weight models are bringing capable, general-purpose AI into highly regulated environments.
Thanks to the global AI arms race, open-weight LLMs that can run locally have nearly caught up with closed, cloud-only LLMs on many tasks. The largest state-of-the-art models still require substantial hardware infrastructure. However, broader advances in training, model architecture, and model compression have made smaller models increasingly capable.
This opens up LLM adoption to organizations that cannot send sensitive customer data to cloud providers. They can now run capable models on premises, keeping sensitive data private, given they have suitable infrastructure. However, many organizations still lack the infrastructure and expertise needed to deploy them.
The right hardware depends primarily on the level of model capability required for a given use case. More complex models are generally more capable, but they also require more expensive hardware and generate responses more slowly. Using a high-end model for a simple, time-critical task can therefore be slow and unnecessarily costly.
To provide an overview we grouped existing models into 4 Tiers and provide a representative task for each tier.
| Tier | Representative task | Example models, ordered byIntelligence IndexThe Artificial Analysis Intelligence Index: one score combining nine independent evaluations of agentic work, coding, scientific reasoning and general knowledge. Higher is more capable.Source: artificialanalysis.ai | Rough hardware investment cost before tax |
|---|---|---|---|
1Intent Understanding and RoutingWhere the tier lands against the best model available that year, measured on the Artificial Analysis Intelligence Index. Progress is fast and these models differ fundamentally in structure and in how they reach a score, so some tasks compare far better than others. Treat it as an estimate. | Understand the intent of an incoming request and route it to the correct process, workflow or department. | Ministral 3 3B6Qwen3 4B8*Gemma 4 E2B9Ministral 3 8B9Gemma 4 E4B12 | Approx. CHF1,500 to CHF3,000 |
2Information ProcessingWhere the tier lands against the best model available that year, measured on the Artificial Analysis Intelligence Index. Progress is fast and these models differ fundamentally in structure and in how they reach a score, so some tasks compare far better than others. Treat it as an estimate. | Summarize documents, answer questions based on retrieved information, normalize content and extract clearly stated information from relatively clean inputs. | Qwen3.5 9B21Gemma 4 12B22Gemma 4 26B A4B26 | Approx. CHF3,000 to CHF5,000 |
3General reliable Intelligence & Complex Document handlingWhere the tier lands against the best model available that year, measured on the Artificial Analysis Intelligence Index. Progress is fast and these models differ fundamentally in structure and in how they reach a score, so some tasks compare far better than others. Treat it as an estimate. | Reliably extract and validate information from heterogeneous document collections, interpret images and their contents, compare information across documents, identify inconsistencies and prepare structured assessments. | Gemma 4 31B29Qwen3.5 35B A3B29*Qwen3.5 27B34*Qwen3.5 397B A17B34 | Approx. CHF6,000 to CHF100,000+ |
4Frontier Agentic Intelligence and CodingWhere the tier lands against the best model available that year, measured on the Artificial Analysis Intelligence Index. Progress is fast and these models differ fundamentally in structure and in how they reach a score, so some tasks compare far better than others. Treat it as an estimate. | Execute long-horizon workflows across documents, tools and internal systems, conduct complex research, navigate large codebases and autonomously develop or debug software. | GLM-5.251Kimi K357 | Approx. CHF100,000 to CHF1.5 million+ |
How we got here
Until recently, companies had to choose between capable AI and private AI. Yet even this choice is relatively new. For many years, highly capable general-purpose AI did not exist at all. Every new business AI use case effectively meant building or fine-tuning a dedicated system.
A company that wanted to classify incoming requests, extract information from documents, or detect a particular type of claim typically had to build or fine-tune a dedicated model using its own labelled data. These systems could run locally and privately, but they were narrow, expensive to develop, and costly to adapt whenever the task changed.
General-purpose language models did simply not exist. Existing systems were relatively small and mostly local, ranging from simple n-gram models and recurrent neural networks to early Transformers such as GPT-2. They could complete simple sentences, but the longer a passage became, the more obscure and contextually incoherent the generation was.
The feats were impressive, but not yet as useful in practice. As it turned out, simply increasing training data, model size, and compute made the outputs more coherent, context-aware, and useful. Large Language Models, or LLMs, were born: models hundreds of gigabytes in size, whose most capable versions could only be operated on massive data-centre infrastructure.
These models were general-purpose, meaning that businesses no longer needed to train a dedicated model for every new task. Companies could use one general-purpose model through prompting, examples, retrieval, and context. A single LLM could classify requests, summarise documents, extract information, draft responses, and answer questions without being retrained for each individual workflow.
While this drastically reduced the cost of deploying AI applications, it also raised the question of whether it is acceptable to send customer data to large cloud providers such as Google, AWS, or Azure. For some businesses it is. For others, such as those in the public sector or the pharmaceutical industry, it is simply not an option.
Companies subject to strict data privacy requirements were thus excluded from the new possibilities that came with increasingly capable AI models. An example use case would be the automated checking of applications or requests submitted to a public authority for completeness & correctness, or the review of sensitive pharmaceutical documents such as batch records, deviations, and CAPA reports.
What made it possible
Two techniques have played an especially important role in this development: distillation and quantisation. Distillation transfers capabilities from a larger teacher model into a smaller student model. Quantisation reduces the numerical precision used to represent an existing model, lowering its memory requirements and often increasing inference speed. Together, they have made extremely capable local LLMs possible on affordable hardware.
