Tag
28 articles
Together AI open-sources OSCAR, an attention-aware 2-bit KV cache quantization system that significantly reduces memory usage and improves decoding speed for long-context LLMs.
This article explains how to build a complete Langfuse observability and evaluation pipeline for LLM development, covering tracing, prompt management, scoring, and experimentation.
Poetiq's new meta-system automatically builds a model-agnostic inference harness that improves performance across multiple LLMs without fine-tuning.
A new tutorial explores how to build a cost-aware LLM routing system using NadirClaw, which classifies prompts locally and switches between models like Gemini for optimal performance and cost efficiency.
A new tutorial demonstrates how Memori, an agent-native memory infrastructure, can be implemented to build persistent and context-aware LLM applications in multi-user and multi-session environments.
Learn to implement sparse matrix operations using CUDA kernels to achieve 20.5% inference and 21.9% training speedup in LLMs, following the TwELL approach by Sakana AI and NVIDIA.
Learn how to set up and use TokenSpeed, an open-source LLM inference engine optimized for agentic workloads, with step-by-step instructions for beginners.
A new analysis explores top 10 KV cache compression techniques for LLM inference, focusing on eviction, quantization, and low-rank methods to reduce memory overhead.
Learn how to build and manage LLM workflows using tools like Promptflow, Prompty, and OpenAI to make your AI projects more reliable and traceable.
Learn how to implement kvcached for dynamic KV-cache management in LLM serving, including setting up Qwen2.5 models with an OpenAI-compatible API and simulating bursty inference workloads.
Learn how PrfaaS (Pre-fill and Decode as a Service) rethinks how large language models are served across datacenters to make AI faster and more efficient.
Google AI introduces Auto-Diagnose, an LLM-powered tool that automates the diagnosis of integration test failures, significantly reducing debugging time for developers.
