Don't Fall in Love With Your AI Model

Last week I started using the most powerful AI model the public had ever been given access to. It blew my mind with what it could do, so I started building with it immediately. Three days after it launched, it vanished. The government stepped in, and just like that, nobody could access it anymore.

I was going to make a whole video about that model and how impressive it was. That video does not seem all that relevant any longer, since you cannot use the model. So instead, here is something I consider far more useful: my best advice for surviving the constant chaos of AI model releases as a SaaS bootstrapper.

Here is the one-line version. Do not fall in love with any particular model. The ground keeps shifting under us, so your edge is a process, not a favorite tool. The rest of this is how to live it.

What actually happened

The details matter, so let me catch you up. The US government issued an export control order barring foreign nationals from accessing the lab's two most powerful models — the public one and its restricted sibling — citing national security. To comply, the company could not cleanly separate who was who, so it shut both models off for everyone. As of late that Friday, no one had access — not paying customers, reportedly not even some of their own staff. Every other model was unaffected and still running normally. The company says it believes this is a misunderstanding and is working to restore access. Maybe it comes back tomorrow. Maybe it does not.

I am not telling you this as news. I am telling you this as a pattern. The single best tool in your kit can disappear overnight for reasons that have nothing to do with you, your code, or your business. If your entire build depends on one model, you just learned how fragile that can be.

The bigger squeeze

Here is the part most builders are not connecting. The big AI labs are racing toward going public. The two largest both filed to IPO within a week of each other, each chasing a valuation around a trillion dollars, expected later this year.

Think about what that means. A company about to go public has to show a believable path to profitability. Right now these labs are losing enormous amounts of money, with profitability targets likely years out. So they are testing their business models in real time.

The signal was already there. The most powerful model was locked to a tiny set of vetted enterprise partners. The public model was included with a subscription for a short window, then was about to flip to metered pricing — you pay for your usage every time, which works out far more expensive than the flat subscriptions we have been used to. Gatekept and more expensive. That is the direction. As a bootstrapper leaning on these tools to build, prepare for that inevitability now.

Rule number one: stay loyal to your process

Do not fall in love with the model — at least not that kind of model. Stay loyal to your process instead. Let me prove it with my own journey over the past several months, mistakes and all.

I started building with what was the most powerful model at the time. Then I switched to a coding tool I clicked with quickly, and that is the one I fell in love with. A newer version of my old model came out; I tested it, the jump was not big enough, so I stayed. Fair enough. Then an even better one came out, and I stayed again — this time out of attachment, not judgment. That was my mistake. I was not following my own process.

Then the model we lost that weekend dropped. It blew my mind and I started using it immediately. Then we all lost access, so I had to switch back. At that point I finally did what I should have done the entire time: a clean head-to-head between my available options, landing on the best one still available for now.

That is a lot of switching in a short window, and that is the actual job. The lesson in one line: do not get attached, and keep testing. Then let results, performance, and budget together make the call. The moment you marry one tool, you lose your real edge — the freedom to always grab the best model that fits your budget.

The process

The good news is that the process is simple. Test the latest models against each other. Find what works best for you, where "best" means performance and budget together, not just raw capability. Rework your process around that model and keep building on it until a new one comes out. Then test again. If the new one is truly better and it fits your budget, switch. If not, keep building. Repeat that forever. You are basically periodically A/B testing new models.

It sounds easy, and it is a little harder to maintain in practice for one very human reason. When you are deep in building, switching tools feels completely counterintuitive. You have momentum, things are working, and stopping to test a new model feels like a distraction — so you skip it, like I did. Get comfortable with that discomfort, because the rate of change is only going to accelerate. The founders who keep testing are the ones who keep getting the best output for the lowest cost.

The SWE-bench shortcut

My process raises an obvious question: how do you even know whether a new model release is worth testing? There are something like two dozen benchmarks and variables these companies use to compare models, and reading through all of them every release is exhausting.

So here is the simpler filter I use, and it has worked well so far. Focus on one number: the SWE-bench score. It measures how well a model resolves real software engineering tasks — real issues, in real codebases. That is exactly what we do as SaaS builders, which is why, in my experience, it has been a remarkably consistent predictor.

The rule of thumb: if a new model beats the one you are currently using on SWE-bench, it tends to perform better for building bootstrapped SaaS products. That becomes your first gate. New model out, glance at its SWE-bench score against your current model. If it wins, run it through a real test. If it does not beat your current model, you can usually keep right on building without breaking your workflow. It is a fast first filter, not the final word — you still confirm with your own test and your own budget — but it saves you from chasing every shiny release.

The refactor experiment

A model passed the SWE-bench filter and you want to test it for real. Here is the specific test to steal from me, because it does double duty. Point the new model at your actual product and ask it to review your entire codebase. What should be fixed, optimized, or refactored, and why?

I did exactly this with the model we just lost access to, before it went dark. I had it audit my projects, and it came back with a genuinely useful prioritized list — everything from security enhancements to performance improvements I had not gotten around to or had missed. Bonus: if the new model does a good enough job and extended use fits your budget, have it implement the refactor too.

See the real value here. You are killing two birds with one stone. You are stress-testing the new model on work that actually matters to you, and you are improving your real codebase at the same time by following a best practice. A new release is the perfect natural trigger to do the refactoring you have been putting off anyway. You turn a boring but valuable exercise into something more exciting by testing the shiny new toy on it.

Know your real cost per task

What if the new model is expensive — so expensive that you can only use it selectively? This is exactly why you run the experiment, because it gives you the one number that makes the decision easier.

While I tested that model as part of my existing subscription, the usage command had been updated to show what each session would have cost once it flipped to metered billing. So I was running two experiments at once: how good the model is, and how much it would cost to do a specific job. I pointed it at two real codebases — Clockless, which you have heard me talk about, and a trading bot I built for fun that decides which stocks to buy and sell. A full comprehensive refactor of each took a couple of hours and would have cost somewhere between $60 and $80.

That might sound steep next to a $100-a-month subscription. Here is the reframe. Your real alternative is hiring a senior developer to do that same audit and cleanup by hand. That takes a week or more, not a couple of hours, and no senior dev is doing it for $80 — their rate alone is probably more than that per hour. The real cost runs into the thousands.

Once you know what a job costs — say $80 for a refactor — you know exactly how to use an expensive model strategically for the high-value jobs that are worth it. Instead of running a pricey metered model all day, you use your tried-and-true workhorse models for the larger build phases, where they are far more cost-effective.

The mindset that ties it together

Let me zoom out. For the most part, the days of humans writing the bulk of code are over. These days the AI is typically better, faster, and cheaper across the board, and we as solo founders need exactly those advantages to compete. That is not something to resist. That is your leverage, so lean into it.

That does not let us off the hook on good engineering hygiene. You still want clean, well-maintained code. The trick is that new model releases are the perfect time to handle that — you refactor and test the new model at the same time. This is a prediction I made about a year ago that still holds: worrying about the code quality of your current model matters less and less, because you can pretty much count on the next model to refactor it and fix those problems. So do not over-worry about today's code. Just carve out time to refactor with each new release when it lands.

Your playbook to follow

Stay model agnostic. Do not fall in love with a single tool.
Run the loop. Test the latest. Adopt it only if it is better and fits your budget. Build, then test again when the next one drops.
Use the benchmark as your shortcut. If a new model beats your current one on SWE-bench, it is likely worth testing.
Use every new release to refactor. Have the new model audit your codebase, and let it implement the fixes if that also fits the budget.
Learn the real cost per task so you can use expensive models more strategically going forward.

Do that, and it really will not matter what these companies throw at you next, or which model gets pulled offline on a Friday. Your process will carry you through. Stay agnostic, use SWE-bench to decide what is worth testing, refactor on every release, and know your costs. The models will keep changing on us — that is exactly why a steady process beats a favorite tool every single time.

Want the full framework?

If you want the complete framework for building, shipping, and pricing a SaaS product in this AI era, my free 5-day email course walks you through the whole process.

If you want to work with me directly on your build and your tooling decisions, my private coaching program is open.

Stay agnostic out there.