Weekly Column
We Trained an LLM So Bob Wouldn't Have To
MSPs drown daily in data they legally can't touch or use for much besides information. Customer networks, credential dumps, incident timelines, ticket histories: the kind of stuff that would evaporate a client's terms of service the second you uploaded it to OpenAI's API. So we trained a model. Deployed it locally. Moved on.
It was the right call. Necessary, even.
But that doesn't mean Bob, my tier 1 tech, should be doing the same thing.
Why We Did It (The Real Reason)
Commercial LLMs are difficult to use in the MSP space. Every customer contract had a clause about data residency or third-party handling, and Initech's legal department would've had an aneurysm if we'd suggested piping Acme Corporation's ransomware response logs through ChatGPT or Claude's API. So we took a base model, fine-tuned it on sanitized (but representative) data, and ran it on-prem.
Did it solve a real problem? Yes. Could we have done it differently? Probably. Was it more expensive and operationally heavier than just using Claude with good prompts? Absolutely.
But the trade-off was defensible. Data stays inside the perimeter. Compliance doesn't explode. Customers don't sue us because we got cute with their IP.
That's the threshold. If you can't use an existing service for legal, compliance, or security reasons, then perhaps it's time you train a model.
But Then There's Bob
Bob performs tier 1 support. Bob handles password resets, printer connectivity, and the occasional "my email isn't working" for hours daily. Bob saw us deploy this model and decided he needed to train one too.
For what, exactly? Bob wants to "automate first-response ticket categorization." Which is a real problem. But Bob's solution was to:
- Spend four weeks learning PyTorch, which is admirable.
- Copy and paste tickets verbatim (without speaking to anyone).
- Build a fine-tuned model on his personal laptop.
- Tell the team he was "implementing AI" to improve efficiency.
Bob did not:
- Ask whether Claude's API, with a five-prompt system message, would've solved this in 45 minutes.
- Check if we had compliance concerns about his data sources (we did).
- Calculate the cost of his time versus the actual value delivered.
- Test whether the thing actually worked better than the alternative.
This is what happens when organizations treat AI training as a checkbox instead of a decision tree. "We have data. We have engineers. Therefore we should train a model" is not a strategy. It's cargo cult engineering.
The real question is simpler: Can I use an existing service for this? If yes, do that. If no, and there's an actual constraint (legal, operational, data sensitivity, cost at scale, or specialized domain accuracy), then you evaluate training. Not before.
Bob had no constraint. Bob had impatience and a blog post he'd read about "custom models performing better." Bob optimized for sounding impressive in the team Slack, not for shipping value.
And here's the kicker: Claude plus a good system prompt would've solved his problem in a day. He'd get 85% of the way there. The last 15% isn't worth four weeks of labor.
We trained a model to solve a problem. Bob trained a model because he'd heard that's what people do with data, and he wanted to participate in the trend. Bob should've asked a simple question first: "Does this service already solve this?"
Most of the time, the answer is yes. Use Claude. Use GPT. Use whatever. They're good enough for the vast majority of work. They're maintained by hundreds of people, constantly improved, and not your operational responsibility.
The bar for "we need to train our own model" is high. Not impossible, but high. You need a genuine constraint. Not curiosity. Not novelty. Not resume-padding.
So: did we do the right thing? Yes. Does Bob need to replicate our work? Absolutely not.
The gap between "we solved this problem" and "everyone should solve this problem the way we did" is where a lot of organizational energy gets wasted. Bob's spending four weeks on something that has a three-hour solution.
That's not innovation. That's just expensive.
— Daniel