Titus Winters’ Post

Prompt caching is a new approach making everything faster and cheaper. https://lnkd.in/gAt98cpS

Recently, I came across a research paper on the Sieve eviction algorithm, and I'm thrilled to share my key takeaways with you: 🔍 Sieve: Simple, Efficient, and Scalable The Sieve algorithm introduces a novel approach to cache eviction, supporting both lazy promotion and quick demotion. ⚙️ Efficiency and Scalability Sieve requires fewer operations for cache eviction and retention, making it highly efficient. Its simplicity also reduces the need for locking, resulting in superior scalability, especially in high-throughput scenarios. 🌐 Web Cache Optimization Sieve is tailor-made for web caches, offering superior performance compared to state-of-the-art algorithms. However, it may not be suitable for OS caches due to its limitations in evicting range scanned items. 📈 Testing and Validation Also I got to know that there are datasets and simulators to rigorously test cache performance and efficiency. If you're interested in cache optimization or data management, I highly recommend exploring Sieve and its implications further! Link to the research paper here: https://lnkd.in/gYk_Eyji Link to the website and the code here: https://lnkd.in/gmK85CCe #CacheOptimization #SieveAlgorithm #Innovation #ResearchDiscovery #LinkedInLearning

Prompt caching with Claude .. reduction in cost as well as latency

Tech Rant. Caching the LLM response for all document chunks that are similar is dumb, First off, the LLM response if it is not repeatable by the LLM on consistent basis is a random response. It may be a random correct response, but it is still a random response. The only way to know you have a non-random response is if it repeats for the same meaning you put in, and by meaning I don't mean using vectors as a representation because that is always going to be vague and imprecise. Caching LLM output based on vector similarity of input does two things: 1) Gives you the same random output that could be correct or wrong, and is certainly not optimal... because if it was optimal the LLM will produce it consistently without the need to cache. 2) Gives you the illusion of Stable Reasoning.. For the same actual human-understood meaning and logic you give it, if the LLM gives you back the same meaning and logic each time (after reasoning about the input) then you know you have the optimal output for the logical constraints you've programmed into it. Also, RAGGers say Chunking is Hard. No, being stupid is hard. Why would you expect to solve the comprehension problem with cave man tools like chunking?? End Rant.

Weekend Workshop #6 - Claude's Prompt Caching I had the opportunity to try Anthropic Claude's Prompt Caching today. It's glorious, it's fast, and it would save a lot of money for my clients going forward. My use case today is generating a video script for a client. From a very long context, I have to extract 3 pieces of information, then use them, alongside some other inputs, to get the final script. I have 15,782 tokens for my input and 1,374 tokens for my output. Using Claude 3.5 Sonnet's pricing, that would put me at $0.068 to generate this script using the old method (which is still incredible low for something that I consider to be very valuable). Using Prompt Caching, my pricing would look like this: Base input tokens: 1,260 Cache writes (for writing to the cache the 1st time): 3,618 Cache hits (for "referencing" the cache all subsequent times): 10,854 Output tokens: 1,374 That puts me at $0.041 to generate the script. This price is only 60% of the previous method, and allows me to scale better if I need to hit the cache more times during its window. What's better is if you have an app that has a cache that is constantly hit, the cache's lifetime is refreshed each time the cached content is used. Otherwise, it has a lifetime of about 5 minutes. Per Anthropic's article, it also hugely reduces the latency when you use caching, up to 79% for chatting with a book (100,000 token cached). This will have great value for use cases like large document processing, talking to books, PDFs, papers, podcast transcripts, or coding assistants. You can read more about it here: https://lnkd.in/gY_VXjry

Curious about how your computer loads websites lightning-fast? That's caching in action! 🌟 It's like a super-speedy memory storage system that stores frequently accessed info. So, when you revisit a site, your computer grabs it from the cache instead of fetching it again. It's not just your computer; caching is everywhere—from your browser to your phone, making everything faster and smoother. Next time you marvel at quick loading times, thank caching! 💡💻 Read more on our blog to dive deeper into the world of caching! https://lnkd.in/dt5regai #Tech #Caching #Efficiency

This week has been buzzing with discussions about 'Context Caching' supported in Gemini 1.5 Pro and Gemini 1.5 Flash. After the introduction of Flash attention to reduce cost and latency in processing long contexts on Gemini Flash, Google has started supporting context caching on two of their state-of-the-art models: Gemini 1.5 Pro and Gemini 1.5 Flash. I believe that developing a Retrieval-Augmented Generation (RAG) application, along with context caching and incorporating all the necessary guardrails and guidelines for long system prompts, will be very promising. This approach will contribute to creating a safer, faster, and more cost-efficient LLM application. Official docs from Google: https://lnkd.in/dCQyCjkJ sudhanshu kumar Krish Naik SUNNY BHAVEEN CHANDRA Sunny Savita Boktiar Ahmed Bappy

I've just published a new article on Medium: "Next.js 14 Caching: An In-Depth Examination". This article explores the caching mechanisms and strategies employed by Next.js, delving deep into the automatically generated files, and their roles in enhancing the application's general performance.

✍️ I wrote a post about caching. - What - Use Cases - Two popular write strategies - Eviction Policies It's more of an introduction than an in-depth post. https://lnkd.in/e3KXgM24