Upcoming Events

We want to start off the newsletter by thanking everyone for all of the positive feedback we’ve received since the official launch of the community newsletter! We are continuing to expand our digital presence and are super excited to announce the first virtual GenAI Collective event on October 26th where we will be moderating a panel with OctoML around using LLMs to build domain-aware applications! Onwards, builders!

Bridging the Silicon Gap: The Rise of CPUs in Generative AI

In the last 12-18 months, we have seen the demand for GPUs rise precipitously given the widespread adoption of generative AI (Nvidia revenues grew 123% from 2023Q4-2023Q2). This unprecedented demand has led to an enterprise arms race predicated on who can obtain the largest number of GPUs the fastest. For example, Inflection AI raised $1.3B in June led by industry leaders including Microsoft, Reid Hoffman, and Nvidia, which they publicized would be allocated to building a 22,000 unit Nvidia H100 cluster. However, the capital requirements and current GPU supply make this move virtually impossible for most new entrants and incumbents.

Large semiconductor competitors and the open-source community have quickly responded, and the landscape of hardware for Generative AI training and inference is undergoing a seismic shift. CPUs are gradually inching into a domain predominantly ruled by GPUs thanks to open-source innovations such as GPT4All who have developed an ecosystem to train and deploy powerful and customized large language models that run locally on consumer grade CPUs. This development is necessary and fueled by GPU shortages, technological advancements, the relentless pursuit of cost-efficiency and energy conservation. Sharing some interesting developments below:

  1. Hardware Advancements:

    1. Newer CPUs like Intel’s 4th Gen Xeon Scalable Sapphire Rapids chips with Advanced Matrix Extensions (AMX) units are specifically optimized for AI workloads by leveraging a new set of two-dimensional registers called tiles. Intel claims significant efficiency and cost advantages over competitive hardware for AI inference.

    2. In June 2023, AMD unveiled its Instinct MI300A that leverages a unique chiplet architecture (combining multiple smaller CPUs and GPUs) to boost performance and AI workload flexibility.

  2. Startups Attacking the Problem:

    1. Our friends at OctoML have built a number of accelerated models that are optimized for a wide range of hardware choices. Their vision is to put ML optimization in the hands of the developers by democratizing acceleration techniques and providing hardware recommendations.

    2. Early-stage startup Deeplite has developed a platform for optimizing AI models to run efficiently on edge devices, including CPUs. Smaller and more performant edge models have the opportunity to transform what’s possible within the IoT ecosystem.

  3. Market Dynamics:

    1. According to Mckinsey’s 2022 Report, the semiconductor industry is expected to surpass $1 trillion by 2030, and many industry reports believe this timeline has been significantly accelerated by the recent advancements and adoption of LLMs across industry verticals.

    2. Generative AI continues to be a must-have for enterprises across all sectors, necessitating hardware that can keep pace with evolving demands​. New AI accelerators and ASICs are projected to experience the highest amount of growth over the next 2 years.

Semiconductor product split in 2021 and 2025

As the generative AI landscape develops, the hardware and software ecosystem is adapting with it. Innovations in CPU technology and developer tools are not just a testament to the silicon industry and open-source community’s agility, but a precursor to more accessible and sustainable Generative AI ecosystem.

As always, we are committed to elevating unique perspectives from the community. If you have any topics or insights you would like to share, hit up Eric on the community Slack or reach out via email at [email protected] for a feature in our next newsletter.

Events Recap

October has definitely started off with a bang after another successful Brainstorm Brew. We appreciated the flexibility and hope everyone had as much fun as we did during the intimate and lively discussions at Martuni’s around all things Gen AI from founder adventures to open-source developments. We also appreciate all attendees and hackers who joined us for the Ted AI For Good Hackathon. The event was at max capacity, and it’s always exciting to see how many builders have a shared passion for using AI to make a societal impact!

About Eric Fett

Eric joined The GenAI Collective in early September to lead the development of the newsletter. He is currently an investor at NGP Capital where he focuses on Series A/B investments across enterprise AI, cybersecurity, and industrial technology. He’s passionate about working with early-stage visionaries on their quest to create a better future. When not working, you can find him on a soccer field or at a sushi bar! 🍣

Keep Reading