Sai Kumar Arava

We just released a blog on whether the use of multimodal feature transfer can detect deception (phenomena like sarcasm, irony, and condescension) in conversation. The short answer... Yes. This particular write-up profiles a subset of our work at Symbl.ai on multimodal large language models, and is research led by one of our summer interns, Benjamin R. (from Prof. Larry Heck's AVA Lab at Georgia Institute of Technology). We have a paper currently under submission on this work, and we can't wait to share the full details with all of you soon. Blog Link: https://lnkd.in/g6m_wFDi #multimodal #featuretransfer #LLMs #deception #conversation #sarcasm #irony #language #AI #ML #NLP #LargeLanguageModels #SummerInternship #ResearchPapers #ACL #Llama2 #SymblDotAI

16

In LLMs, long input sequences presents a significant challenge in detecting hallucinations using smaller, cheaper models. One of the key innovations (that blows past LLM-as-judge techniques) introduces a novel windowing mechanism specifically designed to address this issue. Consider this -- conventionally, the standard procedure for models with limited context lengths includes Segmentation, Localized Prediction and Aggregation. This traditional approach proves suboptimal for hallucination detection. e.g. a scenario where the language model generates a statement, "X", which is substantiated by context, "Y". If segments are created such that "X" and "Y" reside in separate chunks, the model remains oblivious to the mutual reinforcement between "X" and "Y", thereby impairing accurate hallucination detection. The approach we take circumvents the limitation by deploying a more refined windowing strategy (RAG context and the model-generated responses getting divided into small overlapping windows). This allows a smaller, auxiliary model to evaluate every possible pair of possible context and response windows. This shift in the windowing methodology offers a more robust framework for hallucination detection, advancing the frontier of reliability in RAG deployments. If you're interested in learning more, read our research here: https://lnkd.in/gFAqr7wB #llmhallucinations #llms #generativeai #llmevaluation #llmevals #llmbenchmarking

38

2 Comments

Optillm: An optimizing inference proxy for LLMs Optillm is an optimizing inference proxy that has over a dozen techniques that aim to improve the accuracy of the responses using test-time compute. Over the last couple of months we have set several SOTA results using smaller and less capable models like gpt-4o-mini. Recently, we have added support for plugins that enable capabilities like memory, privacy and code execution to optillm. Plugins are just python scripts that you can also write yourself, optillm would then load them at start from the directory. You can now also combine the plugins and techniques using & and | operators. E.g. We recently evaluated the new FRAMES benchmark from Google. Using a combination of plugins and techniques (we used readurls&memory-gpt-4o-mini) we were able to get 65.7% accuracy on the benchmark which is very close to what Google reported in their paper with Gemini Flash 1.5 (66.5) which has a context length that is almost 10 times that of gpt-4o-mini. Do check out Optillm at https://lnkd.in/gN6_kNky

21

1 Comment

Learn a development pattern to systematically improve the accuracy and reliability of LLM applications in our new short course, Improving Accuracy of LLM Applications, built in partnership with Lamini and Meta, and taught by Lamini’s CEO, Sharon Zhou, PhD, and Meta’s Senior Director of Partner Engineering, Amit Sangani. (Disclosure: I am an investor in Lamini.) The path to tuning an LLM application can be complex. In this course, you'll learn a systematic sequence of steps for improving accuracy by reducing hallucinations:  - Create an evaluation dataset to measure model accuracy - Add prompt engineering and self-reflection - Fine-tune your model including "memory-tuning" which is a new method of embedding facts in an LLM Using the Llama 3-8B parameter model, you will: - Build a text-to-SQL agent with a custom schema and simulate situations where it hallucinates - Understand the difference between instruction fine-tuning, which gives pre-trained LLMs instructions to follow, and memory fine-tuning - See how Performance-Efficient Fine-tuning (PEFT) techniques like Low-Rank Adaptation (LoRA) reduce training time by 100x and Mixture of Memory Experts (MoME) reduces it even further I appreciate Meta releasing the Llama's family of open models -- this course gives an example of the unique type of work that developers can do with such models. Please sign up here: https://lnkd.in/gyxHwPkp

3,934

72 Comments

Good detailed overview of Retrieval-Augmented Generation (RAG), one of leading techniques to improve accuracy of LLMs. When developing a production-grade LLM-based system, it's essential to understand the advantages and limitations of different techniques (RAG, prompt engineering, fine-tuning etc), across training and serving/inference phases, to improve LLMs for specific applications (see my previous post https://lnkd.in/giuJdJJt) A balanced approach to LLM applications that takes into account accuracy, latency, and cost is crucial for successful productization at scale. #artificialintelligence #largelanguagemodels #GenerativeAI #airevolution

10

Kudos to Kshitiz Parashar for his pioneering work on Agentic Chunking! His latest primer has quickly become the must-read guide for AI Application Engineers looking to optimize LLMs for smarter chunking, better retrieval, and sharper accuracy. This Speaks volumes about the calibre of the team which I am super proud of.  Even a lean, dedicated group can now harness the power of LLMs to design state-of-the-art architectures—proof that AI is leveling the playing field like never before. Ready to dive in? Explore the full article here: https://lnkd.in/eJ8cem3x #aiagents #chunking

37

1 Comment

Hallucinations in LLMs are inevitable, no seriously! Josh Fritzsche and I have worked hard at Voze to find a way to tune, prompt and even plead 🙏 with LLMs to prevent them from the occasional 🍄 hallucination. It seemed like the better we have done the harder the edge cases have been to solve and eventually we realized that they tend to frequently happen when LLMs predict or make anticipations on "nothing". I remember at one point nearly going insane with frustration because when LLMs had a hallucination, they were EPIC! 😱 Then we pushed through it and benefited for the effort! How did we solve this common problem? We didn't (Not completely), we got better at identifying the issues as they happened. Once we identified in the data we were able to start looking for patterns. The same catalysts that cause it to happen once if left unchanged caused it over and over again. Our solution is astoundingly simple. Once you know that it's likely to happen..... let it! Accept it will 🍄, but also be preemptive in not focusing on it and use alternative data sources, generators or providers. Change the conditions that cause it but continue to monitor it as it happens. As Dan Caffee is fond of saying to me on a regular bases. "Dusty we need the system to be adaptive and provide mechanisms to learn from it's past." Well he is right! All we needed was data and the people that contribute to it's quality. Huge props 👏 to strong support from Kathy and Janelles audit teams. We were armed with a knowledge and a desire to learn from past mistakes. Armed with confidence that we could correct these problems we could focus on the data to prevent their impact going forward. When doing millions of LLM (Generative AI) calls regularly Josh and I have learned that hallucinations will happen. We also know exactly how we will handle and prevent it going forward! Failure is not doing anything and accepting defeat.

4

2 Comments

📊 The Power of High-Quality, Diverse, Real-World Data Fine-Tuning LLMs 🌍 When it comes to training Large Language Models (LLMs), the mantra isn’t just “more data”—it’s better data. While having more data can be beneficial, it’s the quality, diversity, and real-world relevance of the data that truly drive impactful results. Here’s why: 1️⃣ Higher Quality: clean, well-curated data improves model accuracy and reduces noise, making the model more reliable and stable. Poor-quality data often leads to inconsistent responses and increased hallucinations. Unbiased labeling consistency is key. 2️⃣ Diversity: it ensures the model can generalize across different contexts and user scenarios. This makes the model more robust, safe and inclusive, which is especially important in NLP for avoiding bias and ensuring representation. 3️⃣ Real-World Datasets: training on real datasets helps models perform better in real-world applications, as it learns from actual patterns and usage scenarios, making the model more practical and effective. For LLMs, there’s a general playbook to prepare data effectively: (1) Collect Instruction-Response Pairs: gather examples of how tasks should be answered, covering various domains and complexity levels. (2) Concatenate Pairs: organize the data into structured sequences, allowing the model to learn from end-to-end interactions. (3) Tokenize: process the data into tokens, using padding and/ or truncation as needed to maintain uniform length. I particularly love 🤗 transformers library's AutoTokenizer to identify which tokenizer to use on every model. (4) Split into Train/Dev/Test Sets: partition your data into training, validation, and testing sets to ensure proper model evaluation and tuning. Ultimately, data quality prevails over quantity, specially as we live in the LLMs era when models have been pre-trained with almost all the public data available in the 🌎 . A well-prepared dataset tailored to your application will make all the difference in the performance and reliability of your model. #AI #MachineLearning #DataQuality #LLM #NLP #ModelTraining #DataPreparation

5

🔥 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐯𝐞 𝐀𝐈 𝐚𝐧𝐝 𝐋𝐋𝐌 𝐢𝐧𝐭𝐞𝐫𝐯𝐢𝐞𝐰 - 𝐓𝐨𝐩 𝐪𝐮𝐞𝐬𝐭𝐢𝐨𝐧𝐬 🔥 🤖 I've conducted a mock interview with my student, Bheesetti Anand, focused on Generative AI and LLM (Large Language Models) interview questions. This session is designed to benefit other students preparing for similar interviews, helping them secure their dream roles in the field of Generative AI. 📔 Here’s why watching the video can help you make a 𝐬𝐮𝐜𝐜𝐞𝐬𝐬𝐟𝐮𝐥 𝐜𝐚𝐫𝐞𝐞𝐫 𝐬𝐰𝐢𝐭𝐜𝐡 𝐭𝐨 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐯𝐞 𝐀𝐈: ✅ 𝐂𝐨𝐦𝐩𝐫𝐞𝐡𝐞𝐧𝐬𝐢𝐯𝐞 𝐂𝐨𝐯𝐞𝐫𝐚𝐠𝐞: Detailed exploration of top interview questions related to Generative AI and Large Language Models. ✅ 𝐏𝐫𝐚𝐜𝐭𝐢𝐜𝐚𝐥 𝐈𝐧𝐬𝐢𝐠𝐡𝐭𝐬: Real-world examples and practical applications to understand complex concepts better. ✅ 𝐄𝐱𝐩𝐞𝐫𝐭 𝐆𝐮𝐢𝐝𝐚𝐧𝐜𝐞: Tips and strategies to approach and answer interview questions effectively. ✅ 𝐂𝐚𝐫𝐞𝐞𝐫 𝐏𝐫𝐞𝐩𝐚𝐫𝐚𝐭𝐢𝐨𝐧: Essential knowledge and skills to prepare for interviews and stand out to potential employers. 📽 𝐢𝐧𝐭𝐞𝐫𝐯𝐢𝐞𝐰 𝐥𝐢𝐧𝐤: https://lnkd.in/g_nkP7fs #generativeai #llm #interview #jobpreparation #mockinterview

14

2 Comments

Many of the companies you'll hear from at #RaySummit have gone through a 5-10 year AI infrastructure journey (even longer in some cases). They've managed the migration from classical ML to deep learning, then from deep learning to generative AI, and they are gearing up for the next generation of advances. The amount of collective wisdom and hard-earned lessons is astounding. I'm always blown away by how technical and how information dense the content is. https://lnkd.in/gMVdCgiK

308

8 Comments

Batch processing for LLM inference activity: "One sign of that is the rise of batch processing. Today, popular consumer AI products like ChatGPT and Perplexity provide users with near instantaneous responses, otherwise known as real-time inference. However, businesses don't always need immediate responses and are often willing to wait hours, days, or even weeks for responses, as long as they don't have to shell out as much money. Founders of cloud and inference providers have told me there's growing pressure from business customers for this kind of flexibility, otherwise known as batch processing." -- Nathaniel Whittemore on "The AI Daily Brief" (with much deserved shout-out to his inspo from Stephanie Palazzolo at The Information) I fall into this bucket as we have a lot of backfilling to do as we're building out the underlying data in our service, and some of that could just as well be done in the wee hours of the night. The new stuff flowing in I want processed ASAFP, but these backfills—I'd mos def take more cost-efficient delayed batch processing. There's also use cases like needing (near) real-time inference for the first inference pass for timeliness / contemporaneousness value, but then a second pass slow-time inference for wider classification exercises over an ever-growing data corpus. I could see us mixing Sonnet & Haiku models in both cases. Just a heads up to Anthropic Mike Krieger as you contemplate this, along with Amazon Web Services (AWS) Jeff Barr Mark Roy Swami Sivasubramanian on the Bedrock side of things...and I don't mean the "batch" mode that doesn't seem to have any real value other than helping AWS manage its own rate limit challenges. (DeepCast is using Bedrock.)

11

2 Comments

Building generative AI systems is NOT about choosing the right model—it’s about balancing technical depth with real-world impact. 💫 In this episode of Vanishing Gradients, I sit down with Ravin Kumar, Senior Research Data Scientist at Google Labs, to discuss: 👉 His journey from SpaceX to Sweetgreen to Google Labs, and what he’s learned about building AI systems along the way. ⚙️ How to define meaningful evaluations and avoid jumping too quickly into model selection. 📈 Real-world applications of generative AI, from helping businesses like bakeries to scaling systems at Google. Here’s a key moment from our conversation, where Ravin explains why starting with evaluations is the principled way to build AI products: Want to hear more? Catch the full episode here (or on your app of choice): https://lnkd.in/gYdV8DcE

9

New short course: Safe and Reliable AI via Guardrails! Learn to create production-ready, reliable LLM applications with guardrails in this new course, built in collaboration with Guardrails AI and taught by its CEO and co-founder, Shreya Rajpal. I see many companies worry about the reliability of LLM-based systems -- will they hallucinate a catastrophically bad response? -- which slows down investing in building them and transitioning prototypes to deployment. That LLMs generate probabilistic outputs has made them particularly hard to deploy in highly regulated industries or in safety-critical environments. Fortunately, there are good guardrail tools that give a significant new layer of control and reliability/safety. They act as a protective framework that can prevent your application from revealing incorrect, irrelevant, or confidential information, and they are an important part of what it takes to actually get prototypes to deployment. This course will walk you through common failure modes of LLM-powered applications (like hallucinations or revealing personally identifiable information). It will show you how to build guardrails from scratch to mitigate them. You’ll also learn how to access a variety of pre-built guardrails on the GuardrailsAI hub that are ready to integrate into your projects. You'll implement these guardrails in the context of a RAG-powered customer service chatbot for a small pizzeria. Specifically, you'll: - Explore common failure modes like hallucinations, going off-topic, revealing sensitive information, or responses that can harm the pizzeria's reputation. - Learn to mitigate these failure modes with input and output guards that check inputs and/or outputs - Create a guardrail to prevent the chatbot from discussing sensitive topics, such as a confidential project at the pizza shop - Detect hallucinations by ensuring responses are grounded in trusted documents - Add a Personal Identifiable Information (PII) guardrail to detect and redact sensitive information in user prompts and in LLM outputs - Set up a guardrail to limit the chatbot’s responses to topics relevant to the pizza shop, keeping interactions on-topic - Configure a guardrail that prevents your chatbot from mentioning any competitors using a name detection pipeline consisting of conditional logic that routes to an exact match or a threshold check with named entity recognition Guardrails are an important part of the practical building and deployment of LLM-based applications today. This course will show you how to make your applications more reliable and more ready for real-world deployment. Please sign up here: https://lnkd.in/gRU6qhzN

3,514

109 Comments

Code Comparison: RAG vs Long Context LLMs RAG has traditionally been the go to technique for augmenting Language Models with external data. RAG came by when LLMs had small context window and could only fit ~4k tokens. But now we have LLMs with 1MM context window. So how does RAG stand against that? In my blog below, we explore that with code - Comparing RAG with Gemini 1.5 Pro by feeding tough examples to both. We will evaluate both approaches on quality of answers, cost and latency. Plus we will discuss other differences between the approaches. Additionally, we discuss what the future holds here and cover some techniques that combine the strengths of both RAG and Long context LLMs. Read more below: https://lnkd.in/g74j8uVq #LLMs #RAG #Gemini #GenerativeAI

34

Good article outlining the approach and limitations of Retrieval Augmented Generation RAG) in improving accuracy of LLMs. While RAG is undoubtedly useful technique (for example, in cases of very recent information that doesn't exist in LLM), it is not a silver bullet: - It can dramatically increase costs, as it requires LLMs to (at least temporarily) store retrieved documents in memory, so the model can refer back to it - LLMs tend to “ignore” some retrieved documents, due to “lost in the middle” phenomenon (https://lnkd.in/gMdKqJ_p), or due to opting instead to rely on LLM’s ‘parametric memory’ — i.e., on the knowledge that’s stored in its parameters as a result of training on massive data from the web. Productizing LLMs, especially for enterprise use cases, requires deep understanding of the use case, and considering multiple techniques such as domain specific LLMs, to find optimal balance between model accuracy, costs and other important aspects #GenerativeAI #largelanguagemodels #artificialintelligence #airevolution

88

10 Comments

I remember when this all started. Faris - invited me to catch up at our favorite spot around Blok M. I had no idea he’d just wrapped up a procedure at Pondok Indah Hospital! (He really knows how to add drama to our catch-ups.) We’d been on separate paths and projects since "the winding down project", but there he was, reminding me of our dream to ride this AI wave together again. It was a plan we’d hatched with Sahal Zain & Gurnoor Dhillon when the world of AI was still mostly theoretical and jargon-heavy. Somehow, the conversation continued with Gurnoor Dhillon while he was waiting for his meeting with someone who would eventually become one of CalvinBall’s key early adopters. It felt like everything was coming together, like a series of seemingly random dots connecting to form a bigger picture. Seeing how far we’ve come as a team at CalvinBall Technologies is truly exciting. What started as theoretical discussions has now evolved into practical Gen AI solutions that are solving everyday problems and hold endless possibilities to shape the industry. 👨‍🚀 👩‍🚀 Here’s to pushing boundaries and achieving new milestones together! Gurnoor Dhillon, Sandeep Ramesh, Faris -, Sahal Zain, Eva Zuliya, Ratu Nadiah Khairunnisa 🚀 #GenAI #AIInnovation #TechJourney #ArtificialIntelligence #TechTransformation #CalvinballTech #Calvinballers

55

2 Comments

Look how strong #Phi-3.5-MoE is! On a completely unseen (the dataset didn't ever exist during the training time), low resource multilingual, and a new healthcare domain dataset, it strongly holds the ground against all those larger or private models. A new paper from MSR India. An extensive evaluation of various language models for multilingual healthcare scenarios. https://lnkd.in/gNSX3gj8

42

1 Comment