Glossary term
Glossary term
Infrastructure and Serving
Running model inference on private infrastructure rather than public cloud.
Llama 3.1 405B is deployed on-prem by a European bank on a 16-GPU H100 cluster using vLLM, ensuring customer financial data never leaves the bank's data centre - required by BaFin data-sovereignty rules.
Mistral AI's Mistral 7B and Mixtral 8x7B are deployed on-prem by healthcare providers using Ollama, enabling clinical-documentation agents to run on hospital servers without PHI leaving the facility.
NVIDIA NIM microservices are deployed on-prem by a defence contractor running Llama 3.1 70B on a DGX H100 system, meeting government data-sovereignty requirements while serving 200 concurrent analyst sessions.