Tag archive
Compare vLLM, SGLang, and TensorRT-LLM for self-hosted LLM serving. Learn why traffic shape dictates engine choice, benchmark costs, and avoid engine lock-in.
In 2025, DeepSeek said its internal inference engine was impractical to deploy publicly without significant modifications. On October 20, 2026, it released DeepInfer: a model-specific inference engine that opens much of the runtime while leaving key cluster-management components out.
We evaluate GroqCloud for latency-sensitive LLM agents, including a runnable streaming benchmark harness, July 2026 token-pricing snapshots, rate-limit and service-tier constraints, model-routing constraints, and an offline SSE failure test.
We examine NVIDIA Dynamo's Kubernetes prefill/decode architecture, build a release-pinned vLLM deployment workflow, and identify the validation gates required before publishing performance or cost claims.
Mercury 2.5 is clocking ~780 tok/s on Artificial Analysis. At that speed, decode stops being your bottleneck. Here’s what actually changes in production: batching, streaming UX, context packing, cost controls, and evals that don’t lie.
拆解网易有道翻译的 AI 工程栈：大模型推理、低延迟架构与亿级流量下的成本优化 ...
Migrating LLM inference on SageMaker HyperPod to NVMe weights cache and image cache: the steps, the risks, and what changes in autoscaling.
FuriosaAI's Warboy NPU is engineered for efficient inference of ultra-compact LLMs, enabling high-speed AI on edge devices, a crucial development for
TurboFieldfare sets a ~2 GB memory budget first and then rebuilds the Gemma 4 26B-A4B runtime, its per-layer expert cache, and even the inst
kimi-k3-in-c runs a 2.78-trillion-parameter Kimi K3 on one CPU in 8.24 GB of RAM, with byte-identical output from 8 GB up to 224 GB, so memo
Groq retired llama-3.3-70b-versatile on 16 August 2026 and points production teams at a...
ollama run llama3 gets a model answering questions in about thirty seconds, on a laptop, with zero...