E31: From Hype to ROI: How Dwarak Rajagopal of Snowflake Scales Enterprise AI

CHAPTER LIST

1. Career Journey & Common Threads - Dwarak's path from AMD processors to Google AI to Snowflake
2. Self-Driving Cars & Physical World AI - Unique challenges at Uber and grounding AI in reality
3. Humans in the Loop - The role of human oversight in AI systems and autonomous vehicles
4. Snowflake's AI Strategy - How data infrastructure differs from others’ AI approach
5. Text-to-SQL: Bearish to Believer - Why context and query logs make the difference
6. Enterprise AI Adoption - Real-world use cases and ROI returns
7. Arctic Models & Open Source Philosophy - Cortex products and Snowflake's model strategy
8. Multimodal Data & AI SQL - Supporting images, audio, and text natively
9. Performance & Efficiency Optimization - Cost reduction and speed improvements at scale
10. AI Tools Transform Engineering - How Cursor and AI coding tools changed Snowflake's workflow
11. The Code Generation Bottleneck - Downstream effects on testing, CI/CD, and code review
12. Time Capsule: What's Primitive Now? - LLMs, alignment, and agentic systems for the future

TRANSCRIPT

Pete: I'm Pete Soderling and welcome to the Zero Prime podcast, where we explore the early stories of top startups through the experiences of their engineer founders. So I'm here today with Dwarak Rajagopal, who's the VP and head of engineering at Snowflake. Dwarak, welcome to the Zero Prime podcast.

Dwarak: Great to be here, Pete. It was great to have Snowflake at Data Council this year, and I was struck by the degree to which AI was infusing so much of Snowflake's products in ways that I hadn't really anticipated. So I'm really excited for the conversation, and I think we're going to have the opportunity to dive into some of that today with you.

Pete: So before we kick off that, I wanted to sort of understand a little bit about your background because you've gone from optimizing AMD processors to making AI run faster at Google and now Snowflake. And I'm just curious, looking back, do you see a common thread in your career connecting all these different dots?

Dwarak: That's a fantastic question, Pete. It's a journey that I've actually talked quite a bit about. My career began right at the genesis of a new era working on compilers for the first x64 processors at Intel. It was like a pivotal time in computing, really laying the groundwork for how we think about 64-bit architectures. I then moved on to x86-64 architecture at AMD.

One of the profound learning experiences there was understanding how innovating within existing systems and doing incremental work is often actually sometimes more disruptive than doing something from scratch, which is why x86-64 actually won compared to x64 in the way it got adopted by users and developers. But my focus was always empowering developers within the space when I started working on compilers. Around 2009, I transitioned towards working on GPUs at Apple.

That timing was quite serendipitous, as it coincided with a massive shift in computing both from the era of mobile, as well as the era of GPUs. Around 2012, when AlexNet demonstrated groundbreaking scalability and performance on GPUs for machine learning, it became clear that GPUs were the future of accelerating AI. My work naturally evolved into building robust systems for machine learning around that time.

Then came a fascinating challenge for me going into self-driving cars at Uber. This wasn't just about ML, but about real-time ML processing at unprecedented scale. Safety is super critical in a highly dynamic environment. It taught me the absolute necessity of reliable, high-performance systems for real-world applications. Then following that, I worked on PyTorch aiming to speed up faster iteration cycles for researchers and ML developers. After that, I worked at Google on huge-scale Gemini training across TPUs, pushing the boundaries on device ML and also leveraging JAX, which is another high-performance ML framework.

Now I'm leading the AI engineering and research efforts, entirely focused on getting AI to all enterprises. But if you think about looking back, the overarching thread that connects all of these chapters is the pursuit of optimization and efficiency across the entire computational stack, always with the goal of empowering developers and bringing tangible, practical impact. That's what I've been thinking about.

Pete: Very cool. It seems like that Uber experience was one that stands out to me because you really got into the world of atoms and not just the world of bits. I'm wondering what were the unique challenges about working in AI at Uber and just talk to us a little bit about that experience, because that's one that I think is a little bit unique in terms of all the other things that you've done.

Dwarak: Yeah, absolutely. I think self-driving was a fascinating domain because it forces AI to be grounded in the physical world. Mistakes have real consequences. It taught me the importance of system reliability and reasoning with uncertainty, which translates directly to some of the problems that we encounter with enterprise AI as well, because it's important to not just be intelligent, but also be predictable, cost-effective, and trustworthy. And some of the challenges at Uber involved not just making the right decision, but making the right decision within a certain short period of time, which is super critical.

And to think about it, even though it was an entirely different domain that I worked in, a lot of learnings from that actually apply to the current world where latency, safety, and the criticality of being able to do reliable systems that affect real-world use cases is something I think self-driving started in some sense in the 2016, 2017 timeframe. But we see AI being applied to all of these different areas now, even though self-driving isn't at full scale for all users yet, but it's quite there.

Pete: I've often thought of self-driving as being a really interesting illustration of the varying degrees to which a human can stay in the loop. And if you think about Tesla autopilot, there's still a human in the cockpit who can obviously take over the controls. And that's really tempting security there around keeping a human in the loop and allowing the human to take over the process. In the case of full self-driving, I imagine there are still agents or folks on the other end watching and can step in in certain cases. Just curious if you have any comments on that idea of like, how far can the human be removed from the loop and can the human be completely removed from the loop or not?

Dwarak: I don't think you can remove humans from the loop because the purpose is transporting humans from one place to another in the self-driving car. But in general, I would say one of the key things to think about is: instead of AI replacing humans, this is about how AI works with humans, both in terms of enabling them to do more, helping them to do more, but also knowing when to reach out for clarification to humans. And this applies not just to the self-driving aspects of things, but also to other areas and other domains, which is pretty clear at this point. And so it's not just about aligning AI to do what we want, but also making AI work closely with humans, which is definitely not going away anytime soon.

Pete: Yeah, and I think these touch points are more interesting, more nuanced. It's not black and white, it's gray. And all these human-computer interaction aspects - the panacea of these fully automated autonomous agents or humanoid robots or things that are kind of designed in our fantasy to take humans out of the loop, I think are not actually great representations of the way the technology is actually evolving. And it would make sense that for a long time, there would be humans in the loop with all these systems in some capacity. And we just have to figure out where those edges are and where those surface areas are. And really as technologists and founders and engineers, we have to anticipate that there will be humans in the loop and make them first-class citizens and figure out what the interaction between the AI and the humans should be, because that's going to be critical for a long time.

Dwarak: I absolutely fully agree with that. I think of AI actually scaling humans rather than replacing humans. So instead of an agent driving one car, they can actually now drive ten cars remotely depending on how that works. So it's more like scaling ourselves much more.

Pete: Yeah, it makes sense. Talk to us about Snowflake and what opportunity you see there, because perhaps you have a different approach. I mean, you're a piece of data infrastructure, or at least at the beginning, that was what most people knew you as. How does your approach and your strategy around AI differ from some of the other companies that you've worked at, like Google and Meta, for instance?

Dwarak: For sure. One of the reasons why I joined Snowflake was I felt like this was an incredible, exciting opportunity because it felt like a natural evolution for my career, aligning perfectly with my passion for practical and impactful AI. But what truly excited me was that Snowflake's AI Data Cloud was the ultimate foundation for enterprise AI.

At Google and Meta, we were building cutting-edge AI, but often challenges lay in how enterprises could actually leverage that AI effectively with their own data, which was often siloed, messy, and not AI-ready. Snowflake's approach is fundamentally different because we start with data. Our customers already have critical business data in Snowflake, and it's already AI-ready.

This means we are not just building models. We are building an end-to-end ML platform, an AI platform that brings AI directly to where the data lives, eliminating complex data movement and governance hurdles. At Google and Meta, a lot of focus was on building very large general-purpose foundation models. While we certainly leverage and contribute to that, Snowflake's differentiation lies in our execution-aligned AI research and product development focused on enterprise-specific problems. We are building AI that really works in real-world cases, whether it's optimizing inference cost and latency or developing AI applications like Cortex Analyst and Cortex Search for enabling more accurate text-to-SQL, the recent model that we announced with the Arctic Text-to-SQL R1 model. It's basically all about delivering real-world ROI for businesses. That's one of the critical differentiations for us.

Pete: It's funny that you mentioned text-to-SQL because I've been famously bearish on this category. We see a lot of companies at Data Council pop up doing various flavors of text-to-SQL. What's Snowflake's differentiation there? I mean, obviously you trained your own model, so it gives you a leg up over just people consuming other folks' models. But do you think that the panacea of text-to-SQL is solvable? Can we build reliable applications that have enough guardrails and have enough accuracy and are perfect enough to use in real-world situations?

Dwarak: Yeah, absolutely. I think the key element here is for text-to-SQL, you need to understand, for example, what is the semantic information for the table schema? One of the big advantages that Snowflake has is that we have all of the query logs for a particular user. So we understand what each of these queries they have made in the past. What are the business metrics that they have gotten? And that is one of the key elements to differentiate between just emitting text-to-SQL code without context - the key element is the context. And I think the more context you have, the more information you have, the better overall experience you can have with the model.

And that's one key element for us. In addition to the query logs, we are also able to ingest dashboards or existing dashboards that users use in BI tools. So that gives a lot more context and much richer context for us to actually understand and tune the model specifically for our users and real-world applications that companies are actually using at this point.

Pete: And how does that work specifically? Because obviously, I assume you're not training a text-to-SQL model per user, you're still taking that sort of data exhaust from their query log and you're feeding it into the prompt somehow to make the responses more correct for them, I assume.

Dwarak: Yeah, there are multiple aspects to it. One is the prompt itself - the semantic model that is sent with the prompt is designed based on the query log that we learned from. So there's actually a model to generate that. And then the other aspect is the reasoning elements where we do a lot of post-training on top of the models to make them much more suitable for text-to-SQL reasoning. And that makes a big difference in terms of results, especially for real-world applications.

Pete: Very cool. I think this is the exception that proves the rule, proves that hopefully I was right in giving founders good advice. Anyone not adjacent to being the database company themselves, in my opinion, should not have been spending time building text-to-SQL systems because in the best case it was just going to be a feature on top of any other data management system. And I think you've proven that by controlling all the context and knowing the query logs and all these other advantages that you have, you have the ability to build - as Snowflake or the DBMS vendor, if you will - something that's much more robust in the real world and differentiated, using aspects of data that other third-party libraries or query tools just wouldn't have access to.

So tell me a little bit more about the other enterprise use cases. Obviously AI in the enterprise is going to be a really big deal. Are enterprises adopting AI yet? Or where are we in the evolution curve in your opinion? And that's a little bit more of a high-level question. And then I think the follow-up to that is more specific examples of where you're seeing enterprises use Snowflake-type AI features at the moment.

Dwarak: Yeah, we actually did recent research and one of the research findings we found was that enterprises were able to see about 41 cents on the dollar returns approximately. And that actually fully aligns with what we're seeing on the ground with our customers. The key, again, is how do we make it very practical and high-impact use cases. We're seeing massive acceleration in the adoption of agent AI with enterprises. These agents are becoming essential to the workforce, enhancing their productivity across various teams. And one of the common threads is how users are actually unlocking their insights from both unstructured data and structured data together.

For example, take a company like TS Imagine. They're using Cortex AI to build AI agents like Taya. This bot actually handles customer support services 24/7, speeding up resolutions and letting human agents tackle the trickier stuff. It's apparently a game changer for them in terms of the feedback that we've gotten. Another example is Whoop. They are using Snowflake Intelligence to create an AI chat app. This means everyone, not just the tech folks, can easily tap into their data. Their analytics team can now focus on predicting the future instead of just pulling reports. Same with Bayer. They're also letting their internal teams ask enterprise data questions in plain English, getting real answers and visuals back.

There are also a couple more examples like North America, which is crushing it by using Cortex AI to analyze tons of customer reviews. What used to take months for them to build pipelines to understand these insights now takes less than a week. That means all these enterprises can focus on building what they build best, like building the best products. So some of the examples that we see, even though these might not be glamorous AI breakthroughs that you read about in sci-fi, these are where companies are really seeing value and measurable returns. So I found it pretty exciting and awesome actually.

Pete: And what's the difference between your Cortex products and I know you have some open source things as well and the Arctic inference and I think you even trained your own text model outside of the text-to-SQL. There's another text model as well that you offer. Is that correct?

Dwarak: So we have embedding models that we train for our Cortex Search use case that's powered by that. Overall, I guess our philosophy for open source is how do we have open innovation drive the industry forward. Projects like Arctic Inference is one of our fundamental building blocks of our infrastructure. It automatically decides when to tune for throughput at the same time optimizing for latency in a very automated manner. Projects like that, Arctic Text-to-SQL are all prime examples where we are pushing open source critical innovations, addressing real challenges. And what we've seen from the embedding models, for example, we see a lot of user adoption externally - more than 12 million downloads a month these days for the embedding models, just powering Cortex Search.

So in terms of when customers want to fine-tune open source models on Snowflake, they fully own those fine-tuned models. So it makes it much easier for folks to understand and also improve things for themselves, but also give that feedback to the Snowflake infrastructure as well. And so what is Arctic LLM specifically? Arctic LLM is our series of large language models. So the text-to-SQL is a model that we built on top of open source models where we post-train it and tune it for structured data processing and insights. We have Arctic Embed, which is more focused on embedding.

Pete: And talk to me a little bit about the multimodal data strategy at Snowflake. How are you thinking about multimodal data and that sort of moving across your system? Are you supporting a new file format or is it a collection of different formats that you're supporting? I mean, obviously, you've supported some different file formats in the case of tabular and some other or Iceberg rather and some other extensions. But I'm just curious how you're thinking about multimodal data and are there new file formats in the works or what's your approach to that?

Dwarak: Yeah, I think our primary approach is making sure that we follow all of the standards and making it easier for our users to actually use whatever file format that they have and also supporting the open file formats like Iceberg. But in addition to that also, how do we make this natively accessible via SQL? Things that we did recently is a project called AI SQL where our customers can actually use normal SQL queries to operate on unstructured data, both generation but also classification, transcribing, translation, very natively within the SQL and they can do it for images, audio, as well as text. So basically supporting all of these different formats.

We also have our own model called Arctic TILT that powers the document processing as well. So you can actually have documents automatically parsed in a zero-shot manner. And we've seen a lot of enterprises using this extensively as well. And there are going to be more and more things coming up in the next few months where it's going to make it much more seamless where customers don't need to care about whether it's an image file or a text file, all of this will just seamlessly work.

Pete: Got it. I know one of the things that's really critical with infusing AI into all these different product lines is just performance and making it as efficient as possible. I'm just curious to hear your thoughts on how your team handles these various optimizations. And do you have KPIs that you use to track and make the AI features more efficient? And how does that actually get reflected in the engineering organization as a whole?

Dwarak: Yeah, absolutely. I think efficiency is table stakes for us. If the inferences are too slow or too expensive, we see nobody uses that in production. So I honestly think this is going to be the next frontier overall. For a while there, it felt like the only game in town was making the models bigger and bigger. But that comes with a hefty price tag both in computing power and takes a long time to actually warm up and be able to operate at scale. If you want AI to really take off in businesses and deliver consistent value, it's got to be affordable and work reliably.

So one of the key elements for us is from a cost-saving perspective, running those models can be expensive. And one of the innovations we did with Arctic Inference uses cool techniques like shifted parallelism, where we can actually reduce the cost by more than 75% in many cases. This actually opens doors and we see this immediately when we make things faster, we see more usage happening. And enterprises find it much easier to start using it. That's one example.

In terms of speed and user experience, we also see that faster AI means you can have a better experience for anyone using it. If you're chatting with an AI or a data agent, you can't have delays. So how can we make inference faster? We actually recently did an optimization where we were able to make it four times faster. And that immediately had a big difference in terms of our users using Snowflake Intelligence, which is much more responsive now. So that's the second part in terms of speed and latency being critical.

The third part is more about the scalability itself, where you're talking about enterprise scale, you need to handle tons of data without things slowing down, optimizing systems to handle batch processing. So all of this is super important. One of the critical elements for our Arctic Inference strategy itself.

Pete: I want to shift gears a little bit and ask you how AI is impacting the work of your engineering team itself. How has AI tooling, coding tooling, generated code, et cetera, swept through Snowflake and what sort of advice and insights can you give us as an engineering manager on how to effectively harness the power of AI in your engineering team?

Dwarak: Absolutely. I think this is something we've been following and seeing huge differences even in the last six, seven months, where one of the things we did was enable our engineers to have access to all of these coding tools like Cursor, GitHub Copilot, Claude. It's amazing to see how people are now able to work much faster. We also have metrics that we track internally, how much it is actually helping. It's sometimes shocking to see how folks have started using it. Even our founder, Benoit, actually loves writing code now using Cursor. You have an example where he didn't work on Go before and now he's shipping a lot of changes. It makes it much more accessible for folks to use it, not just on the engineering side, but also on the product management side. Folks are able to quickly build a prototype. Sometimes it's actually faster to do that than to write a PRD. In some sense, it's much easier to understand because it's like ground truth when you see something demo versus a doc. And so that's something we are seeing - zero to one work going at super high speed. I don't think it's going to stop. I still think we are still early in this iteration.

Pete: Definitely. I'm curious. Have you seen any pressure on the downstream aspects - testing, CI/CD, quality, security? Because one of our theses as an investor is that this fire hose of auto-generated code is ultimately going to put downward pressure on all these other things, including all the way from testing to integration to deployment to maintenance. And the code gen tools themselves are fascinating. But our question is, what are the knock-on effects on all the rest of SDLC that need to be beefed up to support this onslaught of auto-gen code? And of course, at Snowflake, you have a thousand-plus person engineering org and you have lots of checks and balances in a formal process. And you can probably allow a bunch of auto-gen code at the onset without completely destroying stuff downstream. But there are going to be startups that pop up and they'll have far less SDLC process. And they'll be wanting to put stuff faster and faster into production. I'm just curious, how do you see this ending? And where will the choke points be that are necessary to beef up?

Dwarak: That's a great question, because now you are increasing the funnel in terms of the amount of code that can be generated. And so definitely even within Snowflake, we are seeing some bottlenecks. So we see code reviews, in one aspect, where now there is a lot more code to be reviewed by folks. And that becomes a bottleneck. You move from writing the code to reviewing the code as a bottleneck. And we're building in-house AI tooling for helping on the code review side as well. But again, I'm pretty sure once we do that, we already see issues with the CI/CD. And there are some unsolved problems here. I don't think this is solved yet. There are going to be challenges. And I'm honestly looking forward to seeing what the industry comes up with. And a lot of startups are working on exciting things. And I think we need to reinvent the overall software development lifecycle with AI. And I would say writing code is the start of that. And there's going to be a downstream effect across the different parts of the cycle.

Pete: Yeah, absolutely. Just before we let you go, I'm curious to ask you one final question, which is a fun one. If you could put one piece of today's AI technology into a time capsule and send it off into space, what would it be and why - basically to ask what do you think someone in the future would find most fascinating or perhaps primitive about our current approach to AI?

Dwarak: Yeah, that's a great question. If I had to pick one piece of AI technology, I would say it would be a combination of putting an LLM model that's been through state-of-the-art post-training alignment and working inside an agentic system. I would also have tossed in real-world raw company data that it learned from and the tools that it used. Why that specific combo? Because I would say LLMs, even if you go back after 50 years or 60 years, are going to be a pretty big deal from a historic perspective and the magic of post-training and alignment where we are at this point, especially with RLHF with DPO. And then the agentic AI where we are just taking baby steps towards that at this point in my opinion. And then all of this data that needs to be there to fuel this and I'm guessing the future generations are really going to get a kick out of how basic we were in terms of all this data stuff and how in the future, all of this is magically solved. Like there's no hallucination or some wilder architecture that's going to solve it. Conversely, they could also be fascinated by how we managed to achieve so much with probably unsophisticated techniques. But yeah, it's going to be super interesting. It's fun to think about and you and Snowflake are definitely in the forefront of a lot of this.

Pete: Thanks for spending time with us and it's great to hear your thoughts and look forward to hearing more about all the AI progress around the Snowflake business going forward. I appreciate this. That's been a fantastic conversation. Thank you so much. Thanks for joining us for another episode of the Zero Prime podcast. I hope you enjoyed my chat with Dwarak Rajagopal from Snowflake. Don't forget to hit the like button and subscribe in your favorite podcast player to get more episodes from Zero Prime.