Ryan Metz

📣 Machine learning developers: Check out this very technical webinar by our very own Milos Puzovic where he details how to get the most out of Llama on Arm-based AWS Graviton instances.. #onarm #ml Ed Miller

37

Organizations have insufficient technical capabilities for the speed and scale of their ML model implementations. My colleague Sumit Agarwal just published a very helpful research with a capability view and sample vendors for defining a modular, flexible and scalable MLOps solution. https://lnkd.in/gnB_T9bq #MLOps #LLMOps #AI

149

17 Comments

Come see this talk, which I view as the culmination of several years of discussions and co-teaching with George Fairbanks. It's a bit of an overview, a bit of an explainer, a bit of a how-to, but mostly a sneaky way to make some important vocabulary from design become load-bearing in how you think about software testing.

86

1 Comment

We just released our first pretrained draft model that we use to accelerate Llama 2 7B and CodeLlama 2 7B on Ryzen AI. #AMD #AI #ML

173

1 Comment

One way to think about the impact of Generative AI and all AI at scale is a fundamental shift the datacenter recipe. The ingredient ratios are radically different than the legacy datacenter recipe. That then has knock on effects that further impact design. Pretty fascinating once you onboard the implications of that new recipe

64

6 Comments

Checkout our latest report on how just eight SN40L chips run 1 Trillion parameter model at blazing fast speed https://lnkd.in/grwdFVAi. It was great to collaborate with some of the most talented folks in this area. Here is the link to the demo operating on similar principles https://fast.snova.ai/

28

We are not slowing down on Gen AI, come see why...and how it changes things, the nature of work. https://lnkd.in/gW3ettg3

2

🤗 Transformers now supports speculative decoding for ANY model - bringing ~2x acceleration to 140k+ LLMs on the Hub, without accuracy loss 🤯 Introducing Universal Assisted Generation -- a new method from Intel Labs and Hugging Face that expands assisted generation to support target / assistant model pairs across ANY model family 🔥 > Update to the latest 🤗 Transformers version (see first comment for pipeline usage below) > Choose a target model from one of 140k+ LLMs or Mixture of Experts models from the 🤗 Hub > Select a lightweight assistant model such as Qwen2.5-0.5B or Vicuna-68m to generate draft tokens > Accelerate target model inference by ~2x 🔥 > Tokenizer mismatches are identified and corrected behind the scenes, with minimal overhead Check out the blog for more details: https://lnkd.in/dYwgMgWV Grateful for the collab with Oren Pereg, Moshe Berchansky, Jonathan Mamou, João Gante, Lewis Tunstall, Nadav T. and Moshe Wasserblat

357

15 Comments

Nice summary Gautam. But knowing what it takes and overcoming the hubris and actually following the obvious path are two very different stories. Don't get me wrong, there are a number of very capable engineers at Intel, way too many to list here, well, Amber Huffman really deserves mentioning. In a way, the layoff may be a good thing for Intel for the simple reason that some of the egos may get the axe, one can only hope that it is not the innovators instead. The company needs to re-define themselves. Signal integrity, built-in, firmware-based design verification tools used to be the stronghold of Intel's architecture and designs until "Tok" with its cost reductions turned the same features into the Achilles heel. But we can all muse, words are cheap. The next 6 months are going to be crucial for getting the right visionaries in charge and of course that may come at the expense of short term profits but the last thing Intel needs now are some knee jerk reactions trying to regain lost territories that are already a thing of the past.

2

1 Comment

Watch this video to understand how LLMs store facts. The entire series is excellent to get an intuitive understanding. It all boils down to basic linear algebra. About 18 minutes into the video, it touches on sparse auto encoders and why LLMs generalize so well as the dimensionality increases

4

1. IMO, agent patterns are one of the most interesting ways to extend LLMs. memary is based on leveraging a graph database (neo4j) for long-term memory (to get around the limited memory inherent in even the largest context windows in LLMs) and augmenting it with other tools like web search and Computer Vision. https://lnkd.in/gfFcBqJx Looks interesting. 2. More about how consolidation effects the entire healthcare industry: https://lnkd.in/gXJKqNJB The Change hack/ransomware attack not only affected claims, service, refills, etc. but now CMS & NCQA are delaying the entire HEDIS reporting process (used for Medicare/Medicaid quality ratings) due to the impact. 3. NARA bans ChatGPT use agency-wide. https://lnkd.in/gW8T94JU Federal Government agencies see this as a risk, including their information leaking and being used to train models (which may then output the wrong information or show PII/PHI other proprietary information). 4. Oh great, now I can't even be angry anymore :) https://lnkd.in/gcrN9mjx

1

1 Comment

It is good to see vendors racing to introduce the next-gen HBM memories at a 2-2.5-year cadence. I hear rumors about production-ready HBM4 parts in 2026… With the growing demand for memory bandwidth from GPUs and AI accelerators, these announcements are encouraging for the AI community. However, the downside of this increased demand for the highest bandwidth HBMs is that, with limited production capacity, the life cycles of previous-generation parts are shrinking. The HBMs typically have a life cycle of 4-5 years. If customers need older-generation parts for longer, they must purchase them in bulk (last-time buys) and stock them up. But the shelf life of standalone HBM parts is only around 2 years! After that, they would have to be assembled in the package with the core dies to extend the shelf life by another few years. These shorter life cycles are hurting other non-AI customers, such as networking system vendors. Many switching silicon built with HBM2/2E parts could be used for products spanning more than 10+ years, as not every customer needs the highest bandwidth switches. Since each generation of HBM is not pin-comparable with previous ones, at EOL, they are either forced to re-spin their core dies to interface with next-gen HBM parts or shorten the life cycles for the systems they release. I hope the JEDEC community considers interoperability between different generations of HBM. This would help all applications using HBM. I agree that there are challenges; it is hard to double the bandwidth with the same set of interfaces and pinouts. Should they consider moving to other standard die-to-die interfaces like UCIE that guarantee one generation of interoperability? Any thoughts? #sharadayeluriblogs #routers #ethernetswitch #junipernetworks #highperformancecomputing #gpus

109

11 Comments

AI Agent Systems! As we discussed with the Compound AI Systems blog in Feb, we're seeing this in practice. Moving from Monolithic to Modular design is a natural engineering progression..it means we can control quality, latency, etc for each app in Databricks! https://lnkd.in/gFacmiwR

279

1 Comment

If you think compute scaling will plateau, think again. We are in the baby states of inference time compute. VLLMs coupled with computer tools, reward learning and RL can leverage orders of magnitude more inference time compute than what we have seen with foundation models. Here’s a concrete training architecture (replace \pi with a large pre-trained vLLM and add it to the reward learner too, \R is the space of all computer tools)

21

1 Comment

Some great commentary about the industry consolidation in generative AI. Worth reading.

2

1 Comment