From support tickets to evals: a support engineer's path into applied AI

Everyone wants to build with LLMs right now. Fewer people want to answer the boring question underneath it: how do you know the thing you built is actually right?

I spent seven years answering a version of that question, and I didn’t realize it counted as AI work until I started doing AI work. As a Tier 3 support engineer I closed north of 5,000 advanced production issues. Every one came down to the same discipline. A customer hands you an answer they’re sure of, the logs hand you a different story, and your job is to find which one is true before you touch anything.

That is the job in applied AI too. The model hands you an answer it’s very confident about, and your job is to find out whether it’s true. If you’ve done support at a high level, you already have the instinct the field is short on. Here is how it maps.

Plausible is not the same as correct

The first thing support beats out of you is trust in a confident explanation.

A customer writes in: “Renewals stopped working after I updated the plugin.” That’s a hypothesis dressed up as a fact. Nine times out of ten the update is a coincidence and the real cause is a webhook that stopped reaching the gateway, or a fixed renewal date the platform never supported natively, or a grace-period setting quietly rebilling on the wrong day. The plausible story and the correct story point in opposite directions. Fix the plausible one and the ticket comes back in a week.

LLMs are plausible-story machines. That’s their whole design. They produce the most likely-sounding continuation, and most of the time likely-sounding and correct overlap enough to feel like magic. The gap between them is where the entire discipline of applied AI lives. A model will tell you a WordPress hook exists because a hook with that shape should exist. It will cite an API parameter that reads perfectly and was never real. It sounds exactly like a customer who’s certain the update broke their renewals.

A support engineer reads that output and feels the same itch they feel on a ticket: prove it. That reflex, refusing to accept a clean-sounding answer until it’s grounded in something you can check, is the core competency. Most people learning to build with LLMs are learning it from scratch. You learned it from customers at 2am.

Root cause is still the whole game

In support you learn fast that fixing a symptom is worthless. If a bug happens twice, it isn’t a one-off, it’s an architecture problem, and patching the surface just moves the failure somewhere you’ll find later.

Here’s a real one. We had a site throwing memory-exhaustion fatals on a rule-checking path. The easy fix, the plausible fix, is to raise the PHP memory limit and move on. The customer is happy for a day. The actual cause was a regression that called get_posts() with numberposts => -1, so a query that should have returned a handful of rows was loading every matching post into memory on a site with a lot of them. Raising the limit treats the fever. Bounding the query kills the infection. One fix returns as a ticket; the other never comes back.

Now put that next to a RAG pipeline returning wrong answers. The plausible fix is to rewrite the prompt until the bad answer goes away on your test question. Feels like progress. But if retrieval is pulling the wrong chunks, or the knowledge base has two documents that contradict each other, no prompt on earth fixes it. You’ve papered over one example while the failure sits in the architecture. The move is identical to the memory bug: stop tuning the surface, find the layer where the problem actually lives, fix it there. Retrieval quality, chunking, source freshness, contradictions in the corpus. That is the query loading every post into memory. Find it, bound it, done.

Support engineers do this instinctively because we’ve been burned by the alternative. You already distrust the quick fix. In AI work that distrust is the difference between a demo that impresses a stakeholder and a system that holds up in production.

Evals are just reproduction steps for a fuzzier bug

Here’s the part I’d tell anyone coming from support. You already know how to write evals. You’ve been writing them for years under a different name.

What’s the first thing you do with a vague bug report? You turn “it’s broken” into reproduction steps. Exact input, exact environment, exact expected result versus what actually happened. You don’t argue about whether the bug is real, you build the smallest case that settles it. Then you fix, re-run the case, and you know it’s fixed instead of hoping.

An eval is that, for a model. You take the fuzzy claim (“the assistant gives good answers”) and refuse to accept it until you’ve built cases that can prove or disprove it: known input, known good output, a way to score whether the model got there. Then when you change a prompt or swap a model or update the knowledge base, you re-run the suite and you know whether you improved things or quietly broke six other answers to fix one.

Most teams skip this. They ship on vibes, tweak the prompt when someone complains, and have no idea whether last week’s fix caused this week’s regression. That is a shop with no reproduction cases and no regression tests, exactly the shop a Tier 3 engineer walks into and immediately distrusts. Evals are the part of AI engineering people most want to skip, and they’re the part a support background makes you take seriously. Don’t skip them. They’re the reason you can trust anything downstream.

Knowledge curation is a support skill with a new name

There is a role that sits between raw support and building AI systems, and it’s where a lot of this comes together: deciding what an AI is allowed to know.

If you build an assistant on top of a knowledge base, the assistant is only as good as the base. Garbage in, confident garbage out. Someone has to decide which answers are correct enough to feed the model, which are outdated, which two articles quietly contradict each other, and which “helpful” advice would make a real operator wince. That someone is making the exact judgment call support engineers make every day: is this answer actually right, or does it just read right?

The test I keep coming back to is simple. Read a piece of content as the person who has to live with the consequences. Would a real operator running a real site nod at this, or wince because it’s generically correct and specifically useless? Generic AI-shaped advice fails that test constantly. It’s the content equivalent of a plausible answer: smooth, confident, and not grounded in what actually happens when you run the thing.

Provenance is part of it too. Where did this claim come from? Has anyone with real context confirmed it, or did it drift in from somewhere unverified? Support teaches you to trace a claim back to its source before you repeat it to a customer. Knowledge engineering is that habit turned into a system, where nothing gets promoted into what the AI trusts until someone who’d catch the wince has checked it.

What actually transfers

If you’re a support engineer looking at applied AI and wondering whether your experience counts, here’s the honest accounting.

The pattern-matching transfers. Five thousand tickets is five thousand reps at telling signal from noise, and that pattern library doesn’t evaporate because the domain changed. The root-cause discipline transfers directly, since it’s the same refusal to accept a surface fix. The reproduction-and-regression instinct is evals with the serial numbers filed off. And the judgment about whether an answer is genuinely correct versus merely convincing is the rarest thing in the whole field, and you’ve been drilling it against real stakes for years.

What doesn’t come for free is the mechanics. You have to actually learn how retrieval and embeddings work, how to structure an eval suite, how models get served, and what latency and token cost do to your design. That’s real study and there’s no shortcut. But mechanics are learnable in months. The judgment underneath them takes years, and that’s the part you already have.

I’m skilling into this deliberately, working through RAG, evals, agents, and the tooling around them, and shipping the work one piece at a time rather than claiming the title first. The reason I’m confident it’s the right move isn’t that AI is hot. It’s that the actual daily work turned out to be a job I’ve been doing all along, against Stripe webhooks and memory limits instead of model outputs.

The tools change. The question doesn’t: is this answer actually true, and how do you know? If you’ve spent years answering that for customers, you’re closer to applied AI than the job titles make it look.