#llminfrastructure

LLM Gateway Compared: LiteLLM vs Portkey vs OpenRouter vs Cloudflare vs Vercel TL;DR: An...

An AI video pipeline's cost-tracing tool had recorded $0 for every day of its operational life. A vendor worth 36% of total spend reported $0 on every call. The audit that found both, and the dozen smaller leaks hiding next to them.

We installed Inspect AI and attempted an offline scorer contract check, then developed a regression-task walkthrough around its task, solver, scorer, and agent interfaces. The framework is compelling, but we have not validated log reproducibility, grading stability, or model-upgrade cost accounting.

Why the LiteLLM MCP authentication bypass (CVE-2026-59822) matters beyond the gateway itself, and how to reduce the exposure it creates.

Claude's cache needs a 512–4,096 token prefix depending on model — miss it and nothing errors, nothing caches. Here's how to verify and fix

What Is vLLM: The Fast Inference Engine for Large Language Models TL;DR: vLLM is an...

Finance will hand you a per-token comparison. Local inference at some fraction of a cent per thousand...

The Kuleshov Group's guide to building a diffusion language model quietly changes the economics of running AI. Here's what parallel decoding means for a small team paying in USD.

Local models can be a strong fit for agentic systems, but mostly in narrower workflows where privacy, latency, and cost matter more than peak reasoning. This piece looks at where they help, where they fail first, and why a hybrid setup usually wins.

Exact-match LLM caching only works if two equivalent requests fingerprint to the same key. The seven normalisation pitfalls that break naive

The architectural decisions that separate controlled spend from compounding surprises AI and...

Real first-party data: Anthropic prompt caching cut Citare's AI bill 25-35% on parsing-heavy workloads. What works, what doesn't, what burned me $20.