Does $5,000 in AI spend equal $5,000 in value? At Zapier, it's my job to find out. Much of it is automated already: - Every Zapier employee gets an AI spend limit - Hit 75% and get a DM: tell us what you're working on - If it checks out, more budget is added automatically No tickets, no meetings. Lots of our heaviest AI users internally don't even hit this limit, especially since we started educating more on model routing, reasoning levels, etc. Zapiens know when to use the smart model, and when to use the fast/cheap one. The harder cases are the ones the system flags instead of auto-approving, and that's where I come in. Certain roles will need much higher spend, and we've planned ahead for that. Here's the case that still doesn't have a clean answer: did the $5,000 an engineer spent on code generation last week produce $5,000 of net new functionality, in customer satisfaction, adoption, engagement? Good luck answering that directly (I've tried) Even Zapier's not there yet.
I wonder if the harder question isn’t whether $5,000 of AI spend produced $5,000 of measurable output. We don’t evaluate human salaries that way. We invest in people because they make the organization more capable than the alternative. AI still needs to demonstrate value, but perhaps the measure is whether it increases proficiency, efficiency, or decision quality enough to improve downstream business outcomes; not whether every dollar maps directly to a dollar of output. The harder question: Should AI be measured like labor, like software, or like organizational infrastructure?
Tracking spend is easy, but measuring ROI breaks down when your stack is fragmented. Once you consolidate all that app data into a single layer to see what teams are actually producing, the picture gets way clearer.
This seems pretty solid, Ryan. I know it's still an open question, but it's something I'm curious myself: Have you considered evaluations of ROI over time horizons beyond a month? Does an average make sense or something along those lines? I ask since hitting the "human in the loop" threshold at 75% spend sounds like a perfect solution, but I'm curious on if it goes into something like an "idea queue" for Projects, or a "outcomes queue" for Operations. No pressure, I think we're all trying to figure out how to make sense of this issue, opposed to pressing for a solution ^^"
This reminds me of a podcast I listened to recently discussing the AI productivity “J-curve”. They compared it to the early internet where organizations often saw a temporary decline in productivity before completely rethinking how work got done. Makes me wonder whether the harder question isn’t about your short term spend but rather “where are you on the curve?” Part of the bet seems to be that Zapier can hit the growth stride quicker than your competition. Curious if you have a sense of where Zapier is in the adoption lifecycle? How do you think about balancing short term ROI with the possibility that some AI investments only become obviously valuable after the org has had time to adapt?
I love this concept. Part of this includes if the engineer was able to finish it faster, better, or otherwise more profitable for the business. I’ve never thought of tracking generated code for the value generated!
This is such an important part of AI enablement. A lot of organizations are overspending simply because employees haven’t been taught which model to use and when. You don’t need the most powerful model to respond to an email. I love that Zapier is investing in that employee education.
Really appreciate you transparently sharing your frontier at the same time as sharing the practices other companies will want to emulate.
Auto-DM at 75% is actually so clean
Maybe AI spend shouldn't be measured like software spend. It's more of like payroll spend. We don't judge employees by hours we will judge them by outcomes. If five engineers, with the right AI tools, can deliver what used to require 20 people then $5k per person is an investment in leverage. The ROI is in outcomes, not in usage.