Bespoke Labs

We're #hiring a new Founding Software Engineer in Mountain View, California. Apply today or share this post with your network.

Bespoke Labs is excited to contribute to Evalchemy, an open-source platform for LLM evaluation. The problem: Running popular Evals for an LLM like MMLU, MTBench, WildBench, RepoBench, IFEval, AlpacaEval requires you to download and install more than 10 repos, each with different dependencies and issues. This is, as you might expect, an actual nightmare. Many LM benchmarks are not optimized for performance and cost, and can take dozens of hours to compute. Evalchemy can run the full battery of benchmarks 3x faster compared to previous repos, due to parallelism optimizations in our implementation. It also allows easy installation and a consistent platform to run benchmarks and keep track in a leaderboard. We also support adding your own custom benchmarks and leaderboards. https://lnkd.in/gZQG9ZTY We hope that the open source community will help us develop this library into a convenient evaluation tool for AI engineers. Please tell us about your favorite benchmarks or features and we can add them!

Very happy from the news that our paper  "Which questions should I answer? Salience Prediction of Inquisitive Questions" received an outstanding paper award in EMNLP 2024.  Congratulations to Yating, Ritika and the whole team.   #EMNLP2024 The paper is available online: https://lnkd.in/gNQBenZ6

AI monoliths vs Unix Philosophy:  The case for Small Specialized Models. The current thinking in AI is that AGI is coming, and that one gigantic model will be able to reason and solve business problems ranging from customer support to product development. Currently, agents are basically big system prompts on the same gigantic model. Through prompt engineering, AI builders are trying to plan and execute complex multi-step processes. This is not working very well.  This monolith view of AI is in sharp contrast to how we teach engineers to build systems. When multiple people have to build complex systems, they should build specialized modular components. This makes systems reliable and helps large teams of people coordinate with specs that are easy to explain, engineer and evaluate. Monolithic gigantic AI systems are also extremely wasteful in terms of energy and cost: using GPT4o as a summarizer, fact checker, or user intent detector, reminds me of the first days of the big data wave, when people where spinning Hadoop clusters to process 1GB of data.  Instead, I would like to make the case for Small Specialized Models following the Unix philosophy guidelines:  1. Write programs that do one thing and do it well. 2. Write programs to work together. 3. Write programs to handle text streams, because that is a universal interface. Now replace programs with AI models. I believe that the best way to engineer AI systems will be to use post-training to specialize Llama small models into narrow focused jobs. 'Programming' these small specialized models will be done by creating post-training datasets. These datasets will be created by transforming internal data by prompting big foundation models and then distilling them through post-training. This is similar to the "Textbooks is all you need", but for narrow jobs like summarization, legal QA, and so on, as opposed to building general-purpose small models. Several papers have shown that it is possible to create post-training datasets by prompting big models and creating small specialized models that are faster and also outperform their big teachers in narrow tasks. Creating small specialized models is currently hard. Evaluation, post-training data curation and fine-tuning are tricky, and better tools are needed. Still, its good to go back to UNIX philosophy to inform our future architectures. 

Bespoke Labs is excited to support the Datacomp community and related open-source AI efforts with curation tools, datasets and compute.

Quite the speaker list for the Metadata &AI Summit.

Stellar panel: Our Chief Scientist Alex Dimakis will be presenting on the struggles of moving AI from research to production at a panel at #MetadataAISummit2024 with Joe Hellerstein Teresa Tung Deepak Agarwal hosted by Shirshanka Das

This and the Nobel Prize for Hinton have made our day :)

Our small verifier model is now integrated with Ollama. Now we can all check our hallucinations locally

Bespoke-minicheck-7B is better than GPT-o1 on grounded fact checking (and small enough to run on a macbook).