Todd Lipcon

AI memory is fundamental for conversational AI and AI agents. It is critical for building personalized experiences and improving reasoning. Check out Letta’s course to understand how to use AI memory to enhance the customization and quality of your AI app.

15

Everyone knows Moore’s law, the idea that CPU performance doubles roughly every two years. But remarkably, driven by GPU performance and AI model breakthroughs, we’ve seen a very different law emerge with respect to the rate at which AI model performance improves — in many cases improving by orders of magnitude in the past decade. This introduces massive implications in a world of AI Agents. For the past five plus decades, in any domain that computers touch, we’ve generally seen massive efficiency gains that bring more tech to more people. Yet the same has not been possible for essentially any non-digital services. Historically, anytime you want to solve non-computerized business problems (i.e. most knowledge work), we have seen the cost to solve a problem — say from healthcare services to legal work — tend to only go up over time. Due to a mix of inflation, specialization in certain fields, regulation, decreased competition in markets, or other added costs, we’ve seen the price of most services generally monotonically increase over time. This also puts a burden on driving economic activity in general, especially when creating new ideas from scratch, or starting and running a new business. Yet, in a world of AI Agents, anyone — from a small business or large enterprise — has access to automated work that can let them address problems with variable capacity. When you decrease the cost of entry for getting knowledge work done, you dramatically increase the use cases for that work. That new sales program can be tested instantly instead of waiting weeks or months or never happening at all; you can test a broader surface area of your product for bugs or security issues; legal work and reviews becomes affordable on the seemingly insignificant transactions that otherwise produce risk; or you can launch a new marketing campaign in markets you didn’t otherwise serve. Even in a world of AI costs remaining stable this is a super compelling proposition, but in a world of constantly improving technology the implications are vast. Just as Moore’s law steadily lowered the cost of software, increased its sophistication, and made it more ubiquitous, we can imagine a world where knowledge work regularly gets more affordable, sophisticated, and ubiquitous at a constant rate. This means AI Agents present the ability for any unit of work to continuously drop overtime, enabling us to solve greater and greater problems for a wider set of people and businesses. Of course, a large portion of these efficiency gains will quickly get “eaten up” by solving increasingly more sophisticated problems, but those problems will be solved at the cost of a previously simpler problem. The implications of this are enormously interesting, and it’s going to create incredible opportunities all around.

151

13 Comments

Come check out this talk if you want to learn how we built distributed queries for Iceberg + Python!

4

Today, I found a fascinating way to think about “Tokens” in LLMs. It came from a conversation between Jensen Huang and Patrick Collison at a recent Stripe event. When asked about the future of compute capacity in five years, Jensen smartly dodged that question a bit, but then gave an excellent explanation and analogy about why tokens are the new forces that will power the next decades of humanity. He explained that we're now producing something unprecedented (and at scale): floating point numbers that possess “value”, which we now call tokens. These tokens are valuable because they encapsulate “intelligence”. People are now taking these tokens and transforming them into English, French, images, videos, chemicals, proteins, robotic movements, etc. And many are working hard to expand the range of concepts and ideas we can create with these tokens. He then goes on to made a compelling comparison between tokens and electricity. In the past industrial revolution, we successfully found a way to convert “atoms” into “electrons” (by boiling water to power electricity turbines). And now, we've discovered a way to convert "electrons" into "tokens" (by using energy to power data centers that train and run LLMs). When electricity was first introduced, few people understood its value. Today, paying for kilowatts is routine. The same will happen with tokens. Right now, only early adopters and builders are paying tokens. Soon, everyone will be paying for tokens on a daily basis to supercharge productivity and power new products and services. Many new industries will be born from and built on top of tokens. When I first heard about this way of thinking, I got goosebumps. Perhaps it’s because Jensen has a good storytelling skill that helps him sell what he builds, but I have to admit this way of thinking gave me a brief surge of pride. I'm proud of humanity's collective effort and creativity, which have taken us from living under rocks to building machines that can "think." That's nothing short but a miracle.

18

5 Comments

I like this idea of "technical taste". It gets into where software engineering is as much creative art as it is science. Four great takeaways (IMHO) for software devs in this era in particular: 1. Aspiring engineers should cultivate a sense of curiosity, experiment with different tools and technologies, and embrace a mindset of continuous learning. 2. AI has the potential to streamline processes and enhance productivity for engineers, but it may also lead to disruptions in traditional software development workflows. 3. Developing technical taste and judgment is essential for making informed decisions about which technologies and approaches to pursue. 4. Collaboration and open-mindedness are key to leveraging the full potential of AI and staying ahead of technological advancements. I also really liked these two quotes from Sam Schillace: 1. "The right time to do something is when you have that feeling in the pit of your stomach that's like, 'oh, this is a great idea and it's going to suck to build because nothing's ready yet.'" 2. "Technical taste is like, 'how well have you consolidated that set of experiences and heuristics into judgment that you can apply accurately when you see new things?'" 3. "It may be the case that very small teams can do very large projects, or like we were talking about before, it may be the case we're just going to get really ambitious about what we try to do with the same size of teams, which is kind of where I would put my money." Great job on continuing to put out great episodes, Brett Berson and team First Round Capital! #ai #engineering #softwaredevelopment

5

Some takeaways and opportunities for new AI product tooling from my time at KubeCon last week: 🖥️ CPU Inference and On-Prem Deployments: While we're still in the early innings of GPU Inference optimization, there are opportunities worth building towards in the CPU inference space as well. Optimizing CPU inference on-premises is critical for reducing cloud reliance, improving data privacy, and cutting costs. Companies aim to streamline local compute efficiency by enabling faster, more resilient operations with reduced network dependency. 🎮 Texture Generation and Modeling Behavior in AI Systems: Advanced AI techniques, such as texture and mixed-generative models, are proving useful in applications requiring nuanced behavioral analysis and are particularly valuable in industries like gaming and simulation, where creating realistic environments and interactions is crucial. However, training these AI models often requires vast datasets and considerable processing power. Infrastructure teams must ensure that these models can operate at scale, often requiring hybrid solutions that span both on-prem and cloud environments. 👀 Observability and Real-Time Monitoring on multi-format data: AI-powered observability tools can aggregate and analyze systems across data from kernel, applications, and infrastructure layers and do so in real-time, improving troubleshooting and lowering MTTR (mean time to response). However, balancing insights with rising data storage costs remains a key challenge for enterprises. 💻 Modernizing Codebases with AI Transformation Tools: AI tools are helping modernize legacy codebases, reducing technical debt and easing maintenance. While adoption challenges persist, these tools are evolving to make code updates faster and more efficient. Solutions like Moderne, and Codemod are enabling engineers to translate outdated code into modern, efficient formats with minimal manual intervention. 💡 AI-Driven Cost Management and Efficiency: Infrastructure costs are continuing to skyrocket as companies scale up, particularly in terms of data storage, compute, and network usage. AI offers promising opportunities to optimize these expenses through predictive cost management, dynamic resource allocation, and automated scaling. By implementing tools that forecast usage trends and recommend cost-saving measures, such as with CloudZero organizations can gain a clearer picture of their resource consumption. This proactive stance on cost management allows for more efficient resource utilization, ensuring that infrastructure spending aligns with actual demand. AI is no longer a nice-to-have in the infrastructure software space; it’s quickly becoming a competitive necessity. If you're a founder building in any of these areas or themes, I'd love to chat.

47

3 Comments

Alphabet committed another $5 billion recently to Waymo's self-driving cars. A renewed vow of confidence. Per my post yesterday, fast growing additional compute and better AI architectures are changing self driving cars rapidly. Notably, some traditional car manufacturers pulled out investments in full self-driving cars. It didn't think the tech was going to be ready for decades to come, if ever. I think it may be a mistake given AI and compute grow exponentially, not linearly. As an example, it may take 20-40 years to achieve 1% of progress, then just 5-10 years for the remaining 99% (instead of 1,000s of years per our intuition) link in comments #AI #Transformers #waymo

2

1 Comment

Zoom has been using the power of LLMs to enable novel platform features like AI Companion. Excited to see that #DBRX will enable enterprises like Zoom to cost-effectively build, train, and serve their own custom LLMs! Learn more about DBRX here👇

25

where’s OpenAI ‘s sama? oh look. he’s in elon’s shadow. makes sense. he’s shorter. Lol. the island of cannibals that is silicon valley has a new co-chair as mayor of sf. unelected. uninfluential in washington. has weasled his way into a transition team for sf. I wonder is watching elon w trump he found his trump locally in sf. sama is a mirror. he was for pg. he was also for many others. including elon. I guess this is a way to perhaps shore up political points locally in sf. to protect himself as best as he can. washington is a country wide, global game. kinda reminds me of another rebranding. im just jenny from the block. https://lnkd.in/e_jNXXRD if ai companies are to flourish OpenAI Google Microsoft Amazon Apple will require that startups are not somehow impeded. this is a game of davids and goliaths. where the slingshot appears innocuous. that is llms, transformers, access to privileged data. this out of no-where announcement demonstrates that this is a defense tactic by sama. he speaks fondly of sf and has done nothing until now. so his disingenuous nature finds yet another loop hole to worm into. we would be more supportive of his efforts except he’s created an industry for himself through a con game approach. why is he ceo of OpenAI when he’s there 6-7 hours a week. forgive me. isn’t a ceo supposed to be singularly focussed. his game is getting funding that invests in companies that he’s already invested in and as it’s a non for profit there are no conflicts of interest applied. wow. he uses other peoples money to invest in what he’s already invested in. sounds like y combinator showed him how to play the sv players. the next trick will be to do something no one has ever done. take a multi-billy non for profit and flip to for profit. as a non for profit there is no reporting required. there are tax incentives. a moral narrative to spin. then. oooops. how do we monetize this. we believe he will not succeed. to do so would cannibalize capitalism as it’s structured. it also would create a new pathway for universities which are by definition research centres. creating this precedent would alter their business. given that elon has beef with sama. and elon has never been more on top of his game. well. we find local political maneuvering is a way of squirming. OpenAI investors chiefly Microsoft are about to be thrown under the bus. unwittingly. recall it’s the talent at OpenAI that makes it special. the a team has left. the b team is in place. 2025 will not be good for OpenAI elon will have his revenge. 44 millys invested. for what. for sama to throw his weight around. he’s a lightweight. just look at him. lol. source: https://lnkd.in/e9WtQ48k < . >

2

NVIDIA's Jensen said "ChatGPT democratized computing, Llama2 democratized generative AI" ...and now Llama3 is the next level 🔥 It's fascinating to watch Meta's strategic moves. With Llama3, they prioritized training on a massive 15T token dataset to pack all into a lean 70B param model, instead of building a massive model. This allows Llama 3 to match trillion+ parameter models like GPT-4, but at 1/10th the compute, storage and inference costs! 💰 This technique was published by Google DeepMind a few years ago https://lnkd.in/gx7VU8aA Meta is not a dark horse in AI anymore. They might be the top dog. https://lnkd.in/gRRJYuxt #llama2 #llama3 #llm #chatgpt #gpt4 #ai #ml

16

5 Comments

One week to go for AI Engineer World Fair in SF 🤓 Come check out my talk with Trey Doig (CTO, Echo AI) in the Evals track covering: * How Echo AI transformed their business with LLMs * Why accuracy is critical not just for the initial win but also to scale efficiently * How Log10 enabled Echo AI to improve LLM accuracy without needing to build the infra in-house See you there! #AIEWF #LLMOps

24

If we're only focusing on today's AI costs, we're missing the bigger picture. NVIDIA's Blackwell platform is set to break barriers, potentially cutting inference costs by 25x starting in early 2025. I'm super excited — tons of things to try and can't sleep! LLM + ViT + RAG + LangChain + Tr-VAD combined can be used to build a lot of 0 to 1 products. Where most of the founders are solving for the 1 to n. The future is closer than we think! #predicting

3

Please media types writing about the cost of AI: recognize there is a difference between huge GPU/TPU clusters needed to train foundation models vs. inference costs to run most apps on top of foundation models. The two are not synonymous, so you can dial back some of the doom and gloom about AI startups and your commingling of the two different AI business types.

11

Last week, Zach and I headed to Seattle for 2 days of our executive offsite. The goal? To set our vision for 2025. We focused on: → Solidifying our vision, → And allocating resources across our 3 product lines. We answered critical questions like → How can we get our customers to find even more value? → How can we be the rising tide that lifts all boats for our target industries - moving & insurance → How and when can we scale Yembo beyond our first two verticals I think this was our most productive offsite yet! Looking forward to executing on these priorities in 2025 - making ideas into code and into customer life improvements!

53

9 Comments

Managing production software, and especially "root causing" issues, whether at the infra or app layer, continues to be a growing challenge despite the success of companies like Datadog. I have looked at applying deep learning to this problem since 2017 when we worked with Gurashish Brar & Mohit Gupta to incubate Opas.ai. The team was exceptional, and they even hired a great SDR (me 😜 ). We learnt the hard way that it was too early and #AI was just not ready. A lot has changed since in terms of data infrastructure (check out Conviva's Operational Data Platform), causal inference models and LLMs. Jaya Gupta and I are excited to take another whack at this market, which we call #AgentSRE to distinguish from the outdated #AIOps framing. If you are building an #AgentSRE company, ping us. https://lnkd.in/gvBgfN7k

45

6 Comments

~40% of Frec’s customers are engineers working in tech, many of whom expect frequent and large capital gains. They also prefer to self-manage their money, which makes Frec an ideal platform. Meet Evan: an engineering leader who upgraded from investing in ETFs to direct indexing. Direct indexing appealed to Evan because it offers comparable returns to his ETFs while saving him money on advisory and ETF fees. “ETFs are true and tested, and giving folks a different way of investing in the market while saving more money is really appealing.” He’s used many other robo-advisors and other fintech tools, but “Frec seems to maximize tax loss harvesting opportunities better than any other product that I’ve used.” He also likes Frec's transparent fee structure.  “I feel like with a lot of other companies, that’s muddled in with the cost of investments,” he told us, “and you don’t really know how much you’re paying or how much you’re not saving as a result. I think Frec does a really good job of laying all of that out and giving you a true understanding of the cost of these investment products.” Read more about his Frec journey here ⬇

45

2 Comments

I've been saying this for 2 years now and we are finally here. Deep learning models went through this pruning phase as well. If I recall correctly, they were able to do up to 70% with hardly any loss. SOOOO, what that means is, inference is going to be even cheaper. Open source on-prem models are going to be even more prevalent. And, while pruning has nothing to do with this, it does usually follow, people will start experimenting with new techniques and architecture. Now is the time to keep your eyes peeled. This is where the cool shit starts happening. I give it 6 months before we start hearing about a flurry of new techniques. They are probably already working on the right now, to be honest. Some quiet hidey hole with researchers seeing if they can tease out new designs and new techniques or resurrect old ideas to strengthen the model or give it new capabilities. Maybe it will come from a big company, but they are too focused on products right now, so I don't know how that will fit into the calculus of how things used to work. Definitely keep your ear to ground for the universities or random private research. Updates come from unusual places. Anything that is being worked on right now will start getting finished up here mid next year. Maybe even sooner. My guess, is either completely new architecture but still transformer, or graph based enhancements, mixture of experts improvements, or something using older but solid techniques like q learning, k-means, etc. I suspect a huge reevaluation of ethics training as well. Because that is a cluster fuck that only makes the model worse and encourages it to be non-compliant with instructions. Anyway, long story short, the king is dying, long live the next one.

10

2 Comments

One of my favorite people is Bob Van Nortwick. He started & built AWS for Startups in 2010. Let me share a story about Bob. It’s 2012. By this time I’ve known Bob for a few years - he was the AWS liaison to our portfolio at a16z. Credits, technical support, meeting Werner Vogels, you name it, Bob got it done. He built this team to over a thousand people by the time he left ten years later. This is a HUGE role. Startups are a large part of AWS, which is the growth engine of Amazon. You’d expected Bob to have self-portraits adorn his home, start off meetings with grandiose introduction, & relentlessly name drop founders & investors. Not Bob. He’s hilarious & humble, a combo of dad jokes with f-words sprinkled in. He listens to you with complete attention. Even when I call Bob, I can HEAR his focus. He sees me even when he cannot see me. Most of all, Bob was (& is) RESPONSIVE. Anytime I’d email him, he’d reply in five minutes or less. If I explicitly wrote “this can wait” he’d STILL reply in five minutes or less. Back to my story. It’s about 11pm ET on a Tuesday in NYC. I’m in my hotel room winding down. I should sleep but jet lag. Then I see the email no one wants.   A founder emails me about how his entire infrastructure melted down an hour earlier. I’m not technical but I understand adjectives. This is disaster. And, he’s launching his new ipad app this Thursday.   Less than 48 hours.   Which will be featured on the App Store. To millions of people. I immediately email Bob but I’m sure we’ll have to delay the launch and therefore lose the feature. As usual, he replies in five minutes, asks to be connected directly with the founder. I email the founder the next day. No reply. Not a good sign, but understandable. I email Bob. No reply. Shit. What I didn’t know was that Bob & his team were meeting every hour, pulling all nighters, do everything possible. They probably needed a week; they did it less than 48 hours. On Thursday, the founder emails “WE’RE ON THE STORE!!” The launch is a success! Whenever I share this story about Bob, he smiles & says “We’re just doing our job.” Amazingly, everything I just wrote is not the best part about Bob. He’s a fantastic dad with three grown & accomplished children. But of course he’d be a great father! He’s humble, hard working. Even though we’ve never worked formally together, when you work with Bob, you feel like you share an email address with him.   Why did I write this?   Do I owe Bob a poker debt? Is this a sponsored post by AWS? No, I just caught up with him, reminiscing about old times, & he has the same warmth, same deep attention, same dad humor. Our industry has wild & extreme personalities.   I’ve worked with some of them. It’s fun but exhausting. BUT there are very decent people here too.   They don’t get headlines. They don’t want them. But their stories,  their achievements, their character,  deserve sharing. If you know Bob, please tease & embarrass the shit out of him. 😁

432

47 Comments