Red Hat sees inference as AI’s next battleground — with Kubernetes at the core

Red Hat is pushing Kubernetes inference into the mainstream by contributing llm-d to the CNCF, as enterprises race to run AI ...
SDxCentral

Nvidia, hyperscaler-backed open standard for AI inference torch passed to Linux Foundation

An open standard for AI inference backed by Google Cloud, IBM, Red Hat, Nvidia and more was given to the Linux Foundation for ...
VentureBeat

How attention offloading reduces the costs of LLM inference at scale

Join our daily and weekly newsletters for the latest updates and exclusive content on industry-leading AI coverage. Learn More Rearranging the computations and hardware used to serve large language ...
NextBigFuture

Defeating Nondeterminism in LLM Inference by Thinking Machines

A research article by Horace He and the Thinking Machines Lab (X-OpenAI CTO Mira Murati founded) addresses a long-standing issue in large language models (LLMs). Even with greedy decoding bu setting ...
InfoQ

NVIDIA Dynamo Addresses Multi-Node LLM Inference Challenges

Serving Large Language Models (LLMs) at scale is complex. Modern LLMs now exceed the memory and compute capacity of a single GPU or even a single multi-GPU node. As a result, inference workloads for ...