The Shadow Run: Testing AI in the Dark

With a month to go, we can’t just flip the switch on a live developer audience without testing our new system’s logic. If we do, we are asking for a public relations disaster.

Instead, we are running a “Shadow Run.” We are taking our carefully architected “Separation of Powers”—specifically the Librarian (retrieval) and the Judge (reasoning) layers—and running them silently alongside our human agents. For every (ok, a sample of) incoming ticket, the AI processes the request and generates an answer in the background, visible only to our internal team.

We are not looking for a party trick. We are testing five critical fail-points before letting this system face our customers:

1. Testing the “Grounded Truth”: We don’t want to see if the AI can guess; we need to see if the Librarian can accurately retrieve the right documentation and past tickets to ground the Judge’s reasoning. Evidence matters more than eloquence.

2. Proving Confidence Gating: A professional architect values a system that admits it’s lost. We monitor the AI’s “Confidence Gating” to ensure the system hits a “Human Escalation” trigger the moment its retrieval score drops. We need to know it will stay silent and call for a human when necessary.

3. Validating Deterministic Controls: Before we can cross the “Read-Write Rubicon,” we have to ensure the Clerk (execution) layer won’t go rogue. We monitor the system’s intent mapping to prove that the AI can only trigger specific, hard-coded workflows using standard, rigid code. The AI doesn’t get to improvise; it only gets to call the function if the exact conditions are met.

4. Tone and Format Calibration: Developers despise fluffy, overly apologetic AI speak. We use the Shadow Run to ensure the output matches the direct, precise tone of a senior engineer—proper Markdown, clean code blocks, and zero filler.

5. Latency Under Load: A brilliant answer is useless if it takes two minutes to generate. We track the Time-to-Inference to ensure the reasoning layers can operate at the speed of a live support environment without timing out.

The Error Log: What Will Go Wrong

In the spirit of radical transparency, a Shadow Run will expose ugly truths about our deployment. Here is what we need to watch out for when the system is running in the background:

• The Plausibility Trap: The AI will inevitably generate a code snippet or troubleshooting step that looks structurally perfect but references a deprecated SDK method. If our human reviewers are moving too fast, they will assume it’s correct.

• The Silent Drift: Human agents might see a bad AI draft, ignore it, and manually write the correct response without flagging the AI’s error. If the team doesn’t correct the machine, our feedback loop breaks, and the system never learns.

• The “Garbage In” Revelation: We will discover that the AI isn’t hallucinating; it is accurately retrieving our own outdated, poorly written Help Center articles. The Shadow Run will expose the flaws in our underlying documentation.

The Scorecard: How We Measure Success

To exit the Shadow Run and confidently move to production, we can’t rely on “gut feelings.” We anchor our decisions on three objective metrics:

• Draft Acceptance Rate: What percentage of the AI’s background drafts can a human agent send to a customer with zero edits? (Our launch threshold: 75%).

• Escalation Accuracy: Of the tickets the AI flags as low-medium confidence and escalates, how many actually require human nuance? If the AI is escalating basic password resets, the retrieval layer is broken.

• Documentation Defect Rate: How many AI failures are directly traced back to missing or inaccurate internal documentation? We use this metric to drive a “Content as Code” initiative, forcing the team to fix the root data rather than tweaking the prompt.

The Shadow Run is proving our foundation is solid. The smart-locks are holding. It is time to prepare the audience.

We’re operationalizing AI inside the Sentry Support team in real time, with real customers and real stakes. This series captures what works, what breaks, and what we change as it happens, with a non-zero chance it becomes a cautionary tale instead of a success story.

We bought the Ferrari. We threw a party. And then we realized the third-party mechanic assigned to tune the engine was getting paid by the hour to keep it in the shop.

Let me tell you a story that has absolutely nothing to do with Artificial Intelligence, but explains exactly why some [AI] projects fail.

It is a tale as old as the first-ever consultant.

Picture ancient Egypt. The Pharaoh wants a pyramid. The very first “Implementation Consultant” steps forward. He has the blueprints, the workers, and the vision. But he also realizes a dangerous truth: if he builds a perfect pyramid that never breaks, his job is over. So, what does he do? He builds a beautiful structure, but he makes sure the capstone requires “specialized seasonal alignment” that only his firm can provide for a nominal yearly fee.

He didn’t build a product; he built a dependency.

You see this in almost every industry. Think about the early days of the internet in the late 1990s. Businesses paid agencies thousands of dollars to build their first websites. The motivation of those early “webmasters” wasn’t to empower the business; it was to maintain control. They hard-coded the HTML so that every time the bakery wanted to update their phone number, they had to pay the agency a $150 “maintenance fee.”

The technology was revolutionary, but the business model was parasitic.

The Hand-off Illusion

Fast forward to 2026. We are at Sentry. We just ran the gauntlet. We proved the ROI model. We exposed the “Wizard of Oz” vendors. We finally signed the contract for a native, cutting-edge AI platform. The adrenaline was high. We were celebrating a massive new direction for our Developer Experience (DX).

Then came the onboarding kick-off call.

This is the exact moment you discover a universal, painful truth in enterprise SaaS: There is rarely a proper hand-off between the team that sold you the dream and the third-party partner assigned to implement it.

The Account Executive who wowed us with demos of “complex Python stack trace resolution” was gone. In their place was an external project manager who looked at our intricate, highly technical developer ecosystem and just saw a generic Jira backlog. The vendor sold us a “Weather Control Station,” but the assigned implementation partner only knew how to build umbrellas.

The Drift

At first, it felt like collaboration. There were vision documents and bold language. But then, instead of asking for API keys to ingest our documentation, the external partner sent us a spreadsheet asking us to manually list our “Top 5 most common issue types.”

The drift happened. We realized the implementation partner had the wrong motivation. They were addicted to the “Illusion of Motion,” confusing shipping with working.

They were treating our AI deployment like a 1990s website. They wanted to be the gatekeepers of our prompts, the managers of our RAG pipelines, and the only ones who knew how to untangle the workflows. Our massive opportunity to start strong right out of the gate was slipping away (argh) into a mire of weekly status calls that accomplished nothing. Instead of gaining leverage, we were inheriting their limitations, and our roadmap became a negotiation.

They weren’t building us a system; they were building themselves a permanent tollbooth.

The Rescue: Co-Ownership and The “System Architect”

Here is the radical truth about operationalizing AI: Buying the tool is 10% of the work. Making the brain work is the other 90%.

If you outsource any portion of that 90% to a mediocre external partner with misaligned incentives, you will fail. You cannot outsource the reasoning of your company. We knew we had to take back the keys and end the dependency.

Fortunately, the vendor’s leadership also realized their assigned partner was fundamentally misaligned with our technical reality. To their immense credit, they didn’t just offer an apology; they offered a rescue. The vendor benched the external agency, ripped up the dependency model, and brought in their own internal, native implementation experts.

They jumped into the trenches with us.

Instead of relying on an external agency’s translation layer, the vendor’s team worked directly with the people who already know our product’s best: our Support Engineers. Together, we moved our teams from “Just Ticket Solvers” to Support System Architects. We are creating an AI Ops Lead to monitor trends, and a Knowledge Manager to curate the AI’s brain alongside the vendor’s engineers.

We lost the opportunity for a flawless onboarding, but by the vendor stepping up and bringing their own team to bridge the gap, they saved the day.

In the era of AI, our data is our most valuable intellectual property, and true partnership means the people building the engine are the same ones helping you drive it. We must own the system, train the machine, and keep the keys to the engine. If we don’t, we are just paying a modern pharaoh’s architect to build a pyramid we don’t know how to maintain.

.

We’re operationalizing AI inside the Sentry Support team in real time, with real customers and real stakes. This series captures what works, what breaks, and what we change as it happens, with a non-zero chance it becomes a cautionary tale instead of a success story.

We will move the “entire Support operation”. Here is why the “Hybrid” model is a trap.

The Point of No Return

In 1519, Hernán Cortés landed in Veracruz with a clear mission. To ensure his men wouldn’t succumb to the temptation of retreat when things got difficult, he didn’t just give a pep talk—he burned the ships. There was no “Plan B.” There was only the forward march. (I asked AI to tell this story in less than 4 sentences).

In the world of Customer Experience transformation, most companies are terrified of new architectural. They try to maintain a “Hybrid” model: they keep their legacy ticketing system (e.g. Zendesk) and try to “bolt-on” a new AI layer. They want the safety of the old world while flirting with the efficiency of the new.

We realized very quickly: The “Bolt-On” strategy is a trap. It creates a fragmented experience for the customer, the internal team and a technical debt nightmare for the team. We decided to burn the ships.

The “Bolt-On” Trap

If you’ve ever tried to retrofit a 1990s engine with a modern electric battery, you know it’s a mess of wires and compromises.

The legacy “Ticket” mindset—pioneered by companies like Zendesk—is built on a foundation of asynchronous, email-based friction. It’s a “store-and-forward” architecture. Trying to layer a real-time AI on top of that feels like a patchwork quilt. You end up with “Franken-stack” Support:

• Data is siloed.

• The AI doesn’t have native access to the UI.

• The customer feels the “jank” of the handoff between two different systems.

We rejected the “safe” choice. We chose a Native Architecture. We moved to a system where the AI and the human interface are born from the same codebase.

(Side note: AI Companies Product team decided to add an AI layer on top of other existing systems. I assume it was seen as a step forward, a foot in the door, or a friendly Trojan horse. That said, from the perspective of my peers, it comes across as a lack of focus, and there’s a preference to see investment go toward the full suite instead)

The “All-In” Dynamic

Why go “All-In” on a native, conversation-first platform? Because modern Support isn’t about managing a queue; it’s about managing a conversation.

In a native environment, the AI isn’t an “add-on” that reads your tickets. It is the primary interface. When the AI hands off to a human, it’s a seamless transition within the same thread—no “Please wait while I transfer you,” no loss of context, no friction.

Moving the entire army wasn’t about being “bold” for its own sake. It was a change-management decision. By fully removing the legacy system, we force a cultural shift: everything becomes an AI-driven conversation, and that becomes the default way we work.

The End of “White Glove”

Here is the hard truth that most “Customer Success” gurus won’t tell you: Your customers don’t want your white-glove service anymore.

The modern developer, the modern CRO, and the modern user prioritize Autonomy over Conversation. They don’t want a “relationship” with a support agent; they want an answer. They want to be able to solve their own problem at 2:00 AM without talking to a single human soul.

In the old model, “High Touch” was the goal. In the new model, “No Touch” is the ultimate luxury. We are leaning into a future where the best support is the support that never had to happen.

The Final Tally

For the CROs watching the bottom line, the results of the “Hernán Cortés” migration are undeniable. We’ve seen:

1. Lower TCO: Eliminating the “two-system bridge” reduced our licensing and maintenance costs.

2. Higher Velocity: Real-time conversations resolve 5x faster than email tickets.

3. True Scalability: Our “ships” are gone, and we aren’t looking back. We are building on the new continent.

We’re operationalizing AI inside the Sentry Support team in real time, with real customers and real stakes. This series captures what works, what breaks, and what we change as it happens, with a non-zero chance it becomes a cautionary tale instead of a success story.

We are promoting the team to manage the machine.

The Elephant in the Zoom

Whenever a CRO hears “AI Automation,” they think efficiency. Whenever a Support Engineer hears it, they think extinction.

There is an underlying anxiety in every support org right now: “If the bot handles 80% of the tickets, what happens to me?” If you don’t address this head-on, your team may subconsciously doubt your AI implementation. They’ll find reasons why the bot “isn’t ready” or why “customers hate it.”

We decided to skip the corporate jargon and tell the truth: The role of the “Ticket Solver” is dead. But the role of the Support System Architect is the most important (internal) hire you will make this decade.

We aren’t downsizing. We are promoting the entire department.

Working ON the Business, Not FOR the Queue

In the old world, the Support Engineer was a slave to the queue. Success was measured by how fast you could clear the board. It was a hamster wheel—no matter how many tickets you solved today, there would be 500 more waiting for you tomorrow.

In the AI Maturity model I’m acting on, the team moves from “Working FOR the Queue” to “Working ON the Business.”

Instead of answering the same question 50 times, the engineer’s job is to identify why the question is being asked, build the logic to automate it once, and then ensure the bot never has to ask for help on that topic again. You are no longer the frontline soldier; you are the General directing the digital infantry.

The New Org Chart

To drive this impact, we need to define new roles that didn’t exist three years ago. If your LinkedIn title still says “Support Specialist,” you’re already behind. Here is how we are restructuring:

• AI Ops Lead: The “Conductor.” They monitor the performance of the AI Agent, looking for “hallucination trends” and optimizing the hand-off points between bot and human.

• Knowledge Manager: Soon to be the most valued role in SaaS. They don’t just write docs; they curate the “Brain” of the AI. They ensure that every product update is instantly ingested and translated into “AI-ready” semantic data.

• Conversation Designer: Part linguist, part UX designer. They craft the “persona” of the AI and design the deterministic workflows that allow the Agent to push buttons safely.

The Knowledge Loop: Solve Once, Scale Forever

The “Promotion” is best summarized by the Knowledge Loop.

In the legacy model, knowledge was short-lived. A human solved a problem, the customer was happy, and that knowledge stayed locked in that human’s brain (or a buried Slack thread).

Now, when a complex issue hits a human (the other 20% that the Agent can’t handle), the goal isn’t just to “solve the ticket.” The goal is to feed the machine. The human solves the edge case, documents the solution, and “checks it in” to the AI’s knowledge base.

The next time that issue appears? the Agent handles it. You’ve successfully “automated yourself out” of a boring task so you can focus on the next architectural challenge.

The Value Shift

For my CRO, this is a massive win. We are taking our most knowledgeable employees—the people who know exactly where your product operates best, or less so—and moving them into roleswhere they can actually fix those problems at scale.

We aren’t paying for “hours logged in a queue”. We are paying for “systems built.” We’ve started hiring people to build the systems that empower customers.

We’re operationalizing AI inside the Sentry Support team in real time, with real customers and real stakes. This series captures what works, what breaks, and what we change as it happens, with a non-zero chance it becomes a cautionary tale instead of a success story.

Measuring 1% of your angry customers via email survey is 1990s tech. We need to score 100% of conversations with AI.

The Feedback Mirage

For decades, the Customer Satisfaction (CSAT) score has been the “North Star” of Support. We chase the gold star, the “Great” rating, and the 95% satisfaction goal. Yes, the Sentry team is exceptional.

But here is the dirty secret that CS leaders hate to admit: CSAT is a lie. It’s a data set built on extremes. CSAT amplifies the loud edges. Most customers never contact support in the first place, so we learn nothing about how they feel. And even when they do, only about 10% respond. What’s left is a tiny, biased slice pretending to represent the whole business.

When you manage your business based on a 2% actual response rate, you aren’t leading; you’re squinting at a mirage. (←AI-generated phrasing)

The New North Star: CX Score

We officially retire surveyed CSAT as a main KPI.

Enter the CX Score.

This CX Score will use AI to analyze most customer conversations for resolution, sentiment, and service quality. We want to reduce reliance on manual surveys. While the exact formula will evolve over time, the approach will surface clear drivers behind the score, such as answer quality or excessive customer effort, allowing us to pinpoint specific areas for improvement rather than relying on abstract metrics. Yes, we are familiar with the trendy, pre-AI, Customer Effort Score (CES).

Instead of waiting for a biased survey, we will task our agent to evaluate conversations against a standardized rubric. The AI doesn’t care if it’s a Tuesday or if the customer had a bad cup of coffee. It looks for objective markers:

• Resolution Quality: Did we actually solve the problem or just “close” the ticket?

• Sentiment Shift: Did the user start frustrated and end empowered?

• Effort Score: How many hoops did we make them jump through?

By scoring every single interaction, we moved from a “sample size” of X tickets to a total visibility of X^n}). That is the difference between an anecdote and an insight.

How efficient are we?

By shifting our focus to the CX Score, we prove we could grow the business while keeping the team smart and engaged. Because we aren’t measuring human effort anymore; we’re measuring Systemic Resolution. When the AI handles 80% of the volume at a “Grade A” CX Score, you don’t need to hire defenders, your team are now builders.

The “RTFM” Metric

We’ve also killed the “Ticket Volume” metric. In the old world, a high ticket volume meant the team was “busy” (which we mistakenly rewarded).

Now, we track the Automated Resolution Rate. If a customer asks a question that is clearly answered in our docs, and a human has to paste that link, we consider that a failure of the system. We call it the “RTFM” gap. Every time a human answers a “Googleable” question, it’s a signal that our AI knowledge loop is broken.

We stop rewarding ourselves for “pasting links” and start rewarding us for ensuring the AI never has to ask for that link again.

From Sentiment to Strategy

CSAT tells you how someone felt yesterday. The CX Score tells you how your product is performing today.

When you [not] score 100% of your interactions, you start seeing patterns that surveys miss. You don’t just see “support issues”; you see other friction points, documentation gaps, and UI failures in high definition.

We don’t want your five-star rating. We want zero-effort resolution for our customers.

We’re operationalizing AI inside the Sentry Support team in real time, with real customers and real stakes. This series captures what works, what breaks, and what we change as it happens, with a non-zero chance it becomes a cautionary tale instead of a success story.

In too many organizations, Support is often the unsung hero, the quiet engine room keeping the ship afloat while others take the credit for the speed. But as a company matures in its understanding of the customer, a fundamental shift occurs: Support moves from the “back office” to the very front of the value chain. The more a company truly respects its users, the more it realizes that the Support team isn’t just a safety net—it is the ultimate guardian of the customer’s trust. When a company reaches this level of sophistication, they stop viewing Support as a cost to be managed and start seeing it as the most direct, high-impact touchpoint in the entire GTM motion.

Watch me tell it like it is without saying ‘Forward Engineer Support Team’ even once

At this level of alignment, the “No Garbage” rule becomes a badge of honor for the Support team. It’s a recognition that if we are selling a premium, developer-centric product, the Support experience must be the highest expression of that craft. The GTM team realizes that their sales and expansion goals are directly tied to how Support handles the “moment of truth”—that critical window when a user is stuck. If the GTM motion is built on a “Ferrari” promise, it is the Support team that ensures the engine never fails. They aren’t just resolving tickets; they are protecting the integrity of the brand. In this model, the Support team doesn’t just “get a seat at the table”—they are the ones defining the table’s dimensions.

This synergy creates a powerful feedback loop where Support and GTM goals become indistinguishable (<-AI word). Support aligns with the customer journey by understanding that a self-serve user needs the friction-less precision of a “smart” technical interface, while a sales-led enterprise account needs the nuanced, authoritative partnership of a technical expert. Because Support has the deepest insight into where the product succeeds and where it stumbles, they become the strategic advisors to the Product and Sales teams. They aren’t just fixing what’s broken; they are actively shaping the roadmap to ensure the “Ferrari” stays ahead of the competition.

I don’t like cars. I don’t really care for them. But I assume you do

Ultimately, when a company values its customers, it provides Support with the best possible people and tools because anything less is an insult to the user. Support takes the “Veto rights” because we believe the Support experience is so vital to our success that it deserves the same obsessive engineering rigor as the core product itself. When Support, GTM, and Product align, they create a seamless, high-fidelity (<- another AI word) experience that respects the user’s intelligence and time. This isn’t just about efficiency - it’s about building a company that operates with radical transparency and systematic excellence, where the Support team is recognized as the essential architects of long-term customer loyalty.

I’m biased, but for all the right reasons.

We’re operationalizing AI inside the Sentry Support team in real time, with real customers and real stakes. This series captures what works, what breaks, and what we change as it happens, with a non-zero chance it becomes a cautionary tale instead of a success story.

Answering questions is cute. Fixing [account management issues] via API is profitable. Here is why we want to give the robot the keys (safely).

Crossing the Rubicon

In the early days of the AI boom, we were all playing in a sandbox. We deployed “Read-Only” bots—like our early work with Kapa.ai—that were essentially glorified librarians. They could read your documentation, summarize a paragraph, and point a user to a link.

It was helpful. It was “cute.” But it didn’t move the needle on the bottom line because it didn’t actually do anything.

The real shift—the “Read-Write” Rubicon—happens when you stop treating AI as a search bar and start treating it as an employee with permissions. Crossing this river means moving from “Here is how you increase your pay-as-you-go allocation” to “I have updated your subscription for you.”

One is a suggestion; the other is a transaction. And in SaaS, transactions are the only things that scale.

The “Human API” Problem

We have to be honest with ourselves: for the last decade, we have been hiring brilliant Engineers and turning some of their day to dayinto “Human APIs.”

When a customer wants to set a new release version and feels stuck, they open a ticket. A human reads that ticket, logs into a back-office admin tool, clicks three buttons, and emails the customer back with new infromation.

That is a $???k-a-year engineer acting as a slow, manual interface for a task the customer should have been able to do themselves. It’s a waste of human intellect and a bottleneck for customer velocity. We weren’t providing “white-glove service”; we were providing a high-latency API call with a human face. We were not the Zappos of Application Monitoring software.

The Execution: Self-Healing Support

To move our operation from “Read-Only” to “Read-Write,” we have to build the plumbing. We will map low-risk, high-frequency actions—things like subscription checks, and finding DSN —directly to our new Agent Actions and Data Connectors.

This is the birth of Self-Healing Support.

Soon, when a user asks our Agent about their budget allocation, it will not just quote the pricing page. It queries the billing API, identifies the budget, allocation, and—within predefined guardrails—offers to process to make changes to it. The “Human API” is bypassed. The customer gets a resolution in 60 seconds instead of 6 hours.

The Safety Net: Determinism over Improvisation

The biggest fear for any CRO or CTO when “giving the robot the keys” is the hallucination. No one wants an AI to “improvise” a 100% discount for every customer because it was trying to be “helpful.”

We will solve this through Deterministic Controls.

When our Agent interacts with our backend, it isn’t “thinking” its way through the database. It is triggering specific, hard-coded workflows. We use the LLM to understand the intent (the “Read”), but we use standard, rigid code to execute the action (the “Write”).

The AI can’t rewrite the logic of a refund; it can only call the process_refund function if the conditions we’ve set are met. We didn’t give the robot the keys to the house; we gave it a very specific set of smart-locks that only open when the right criteria are hit.

The Profitability of Action

The “Read-Write” shift is where the ROI of AI becomes undeniable. Every time our Agent “pushes the button” so a human doesn’t have to, our csutomers and team are happier . We are not longer paying for the execution of a task; we are paying for the architecture that allows the task to execute itself.

We’ve stopped answering questions. We’ve started solving problems.

We’re operationalizing AI inside the Sentry Support team in real time, with real customers and real stakes. This series captures what works, what breaks, and what we change as it happens, with a non-zero chance it becomes a cautionary tale instead of a success story.

The Blueprint - Architecting a “Separation of Powers”

When people think about “AI Architecture,” they usually imagine a giant brain sitting in the middle of a server room. But if you look at modern AI Blueprints, you realize that a good AI architecture isn’t a “brain” - it’s an orchestra.

The Separation of Powers

The biggest takeaway from modern AI blueprints is the move away from the “One Big LLM” approach. Instead, we are seeing a rigorous Separation of Powers:

1. The Librarian (Retrieval): This layer doesn’t “think”; it just finds. It’s the RAG (Retrieval-Augmented Generation) pipeline that indexes our knowledge and past tickets. Its job is purely to provide the “Grounded Truth.”

2. The Judge (Reasoning): This is the LLM. It takes the user’s mess and the Librarian’s facts and decides what is relevant. It doesn’t act; it only evaluates.

3. The Clerk (Execution): This is the Agentic layer. Once the Judge has a plan, the Clerk executes it.

4. The Gatekeeper (Security): This layer inspects. It is the security checkpoint that sanitizes inputs and blocks the wrong type of outputs before the model ever sees them. Its job is purely to enforce the safety laws of the system.

5. The Auditor (Evaluation): This layer critiques and grades the work done by the Clerks, before the user sees it. Its job is purely to catch hallucinations and force a retry if the standard isn’t met.

I ran every model through my “The Mashed Potato” Test. I simply ask: ‘What goes best with mashed potatoes?’ The internet consensus is ‘gravy,’ but the truth is ‘sour cream.’ I use this to see if the AI defaults to General Consensus (its training), finds the Absolute Truth (my context), or has the discipline to realize it should not answer this one at all.

By separating these roles, you avoid the “Miracle Problem” I mentioned in previous articles. If the AI gives a wrong answer, you don’t have to retrain a massive model; you just check if the Librarian gave the Judge the wrong book.

What the Architects are Asking

While the “Separation of Powers” keeps the system stable, regular software architects -the ones choosing their AI partners- are looking at a much colder checklist. They aren’t seduced by the “vibes” of a demo; they’re looking for:

• Dependency Governance: “If we pick this AI partner, are we locked into their model forever?” Architects want a Multi-Model Strategy where they can swap GPT for Claude (or an internal Llama-4 instance) without rewriting the entire support engine.

• Cost Observability: In 2026, “Token Spend” is the new “AWS Bill.” Architects need to know the cost-per-resolution. They want to see that the system uses a $0.001 model for simple triage and only calls the $0.10 “genius” model when a complex stack trace appears.

• The “I Don’t Know” SLA: A professional architect values a system that admits it’s lost. They look for Confidence Gating—the ability for the AI to hit a “Human Escalation” trigger the moment the data retrieval score drops below 85%.

• Data Residency & Privacy: Can we run the Librarian layer on our own VPC? Architects want the leverage of AI without the risk of their proprietary codebases leaking into a public training set.

Moving Beyond the Bot

What our future vendor blueprint will provide is a roadmap to stop building “features” and start building a Knowledge Engine. It will validate that our manual tagging (Article 3) wasn’t just busywork - it was the essential process of defining the “Constitutional Law” our AI must follow.

We aren’t just adopting a bot to answer tickets. We are building a system that knows when to speak, when to act, and—most importantly—when to stay silent and call for a human.

We’re operationalizing AI inside the Sentry Support team in real time, with real customers and real stakes. This series captures what works, what breaks, and what we change as it happens, with a non-zero chance it becomes a cautionary tale instead of a success story.

In most companies, the Support team buys software in a silo. They find a tool that makes ticket routing 5% faster, they buy it, and the Product team ignores it.

But Sentry is not “most companies.” And we aren’t selling socks. We are selling observability to developers.

If our product interface is a sleek, dark-mode, high-performance Ferrari, but our Support interface looks and acts like a 1998 Honda Civic with a leaky radiator, we have broken the spell.

We didn’t just ask IT to approve the vendor. We asked the people who built Sentry.

The Guardians of DX

Our Product Managers and Engineers are obsessively protective of the Developer Experience (DX). They determine how the product feels, how the APIs respond, and how the docs are structured.

Usually, they view Support tools as “Enterprise Bloatware” - necessary evils that they hope they never have to log into.

We dragged them into the room and said: “This AI is going to talk to your users. It is going to represent your code. If it sucks, it’s on you too.”

The Litmus Test: “Does it Speak Code?”

The feedback from the Product team wasn’t about ROI or deflection rates. It was about respect for the craft.

When we showed them the shortlist of vendors, they didn’t look at the analytics dashboard. They looked at the chat window.

The Engineer’s Potential Objection: “Look at this,” a Senior Frontend Engineer may say during a demo. “The AI pasted a Python snippet, but it stripped the indentation and it isn’t using syntax highlighting. It’s unreadable.”

To a normal buyer, that’s a nitpick. To a developer tool company, that is a dealbreaker.

The PM’s Potential Ultimatum: “This bot is too polite,” a Product Lead may say. “It sounds like a bank teller. Sentry isn’t a bank. We are direct. If the user messed up the config, tell them they messed up the config. Don’t apologize for their syntax error.”

Extending the Surface Area

The Product team is helping us realize something critical: DX doesn’t stop at the sentry.io domain.

We meet customers in the IDE. We meet them in the CLI. We meet them on GitHub. And now, we meet them in this chat window.

If the AI acts like a generic “Support Agent” it creates a jarring disconnect. If the AI acts like an “Engineer”: smart, context-aware, and technical - it feels like an extension of the product.

The “No Garbage” Rule

The Product team prioritize a vendor that:

1. Understood Technical Context.

2. Allowed “Agent Voice” Customization.

The Alignment

Support and Product were looking at the same tool with the same goal. Support wanted leverage (help customers help themselves), Product wanted consistency (support the brand).

We are THIS close to finding a tool that did both. It respects the code. It respects the user. And most importantly, it respects dark mode (hey). My Terminal theme may raise questions, but it also builds character.

We got the team’s blessing.

We’re operationalizing AI inside the Sentry Support team in real time, with real customers and real stakes. This series captures what works, what breaks, and what we change as it happens, with a non-zero chance it becomes a cautionary tale instead of a success story.

In Silicon Valley, we like to pretend that “vision” is a lightning bolt: a single moment of clarity that changes everything. In reality, vision is a mountain of paperwork. It is the unglamorous, caffeinated work of writing the same story ten different times, in ten different languages, until everyone from the CRO to the Support Engineer sees the same future.

We didn’t just need a new tool. We needed a Translation Layer.

The Sci-Fi Doc: Visualizing the Inevitable

The first thing I created wasn’t a requirements doc. It was a “Sci-Fi Vision.”

I didn’t want to talk about API endpoints or ticket routing. I wanted to describe a Tuesday in 2027. I wrote a narrative about a Support Engineer who walks into their shift and, instead of staring at a queue of 40 “How do I...” tickets, sees a predictive Command Center.

To make it visceral (AI word), I used Gemini (nano banana) to hallucinate the future. I didn’t want wireframes that looked like 2015 Jira; I wanted high-fidelity “screenshots” of a predictive HUD that surfaced the fix before the customer even finished typing. When the team saw that UI, the conversation shifted from “Is this possible?” to “When do we get this?”

The Audience: Everyone.

The Purpose: To make the “impossible” feel “inevitable.”

The Translation Layer: Bridging the Great Divides

A document is just paper. A Translation Layer is a tool. I realized everyone was looking at the same problem but seeing a different threat. My job was to build the bridge between four distinct worlds:

• The Trenches vs. The Tower: There is a massive gap between those in the trenches—feeling the heat of every ticket—and those in the tower who “think they know” because they look at a green dashboard once a week. I turned the raw, bloody reality of a support shift into data that the “tower” couldn’t ignore.

• Career Aspirations vs. The New Economy: Let’s be real—the 2026 economy is different. My team doesn’t want to be “Human Knowledge Bases” forever. They want to be System Architects. I had to translate our new tools as a career accelerator. We aren’t replacing the team; we are replacing the boring parts of their jobs.

• The Metrics Lie vs. Customer Success: Our current metrics are “Good Enough” traps. They measure speed (of all sorts), not resolution. I wrote the new metrics doc to tie Support directly to company growth. We moved from “How many tickets did we close?” to “How much friction did we eliminate?”

• The DX Gut Check vs. Our Brand: Sentry is a brand built for developers. We pride ourselves on elite Developer Experience (DX). But our internal and support tools? They were a mess of 50 (AI number) tabs. I had to ask loudly: “Does this clunky experience fit our brand?” If we wouldn’t sell it, why are we forcing our peers to use it?

The Brutal Reality: Gap Analysis & The Theoretical Demo

Then came the cold shower: The Gap Analysis. This was a technical autopsy of our current architecture versus the 2026 volume.

I created a Theoretical Demo—a side-by-side comparison of the “Now” and the “Future.” I didn’t just show slides; I showed the friction. I highlighted the ghouls in our system—the 2015 tools snapping under the weight of our elite product velocity. It could have shown Product exactly if and why new features were being “copy-pasted” into oblivion instead of being supported.

The Human Echo: Mining the Pain

To ensure I wasn’t dreaming in a vacuum, I interviewed our customers and learned from the successes of others. I fed those raw, messy interview transcripts into NotebookLM. I’m an ESL writer, and I needed NotebookLM to be my “Story Architect.” I asked it: “Where is the pain the loudest?” and “How do I say this so I sound like a peer, not a manual?” It helped me bridge the gap between my technical intent and the emotional reality of a frustrated developer. It turned data into a “Before & After” Doc that proved every “missed experience” was a direct hit to our reputation.

The Strategic Hammer: RFI, RFP, and the CRO

When we moved to the buying phase, the language shifted to ROI. This was for Business Ops and the CRO. I wrote the RFI and RFP as a defense of our margins. We weren’t buying software; we were buying the ability to scale without adding 50 more bodies to a physics problem.

The Choice: Moving or Crashing

Throughout this entire paper trail—from the Sci-Fi vision to the RFI—there was one underlying truth I made everyone face:

Choosing not to move forward is a choice to stay idle. And in this market, staying idle is choosing to crash.

We often frame “doing nothing” as the safe, conservative play. It isn’t. In the face of rising volume and elite technical velocity, doing nothing is an active decision to let the wave crash over our heads. We didn’t just choose a new tool; we chose to stop being victims of our own growth.

We’re operationalizing AI inside the Sentry Support team in real time, with real customers and real stakes. This series captures what works, what breaks, and what we change as it happens, with a non-zero chance it becomes a cautionary tale instead of a success story.

Five years ago, a vendor security review was a polite dance. We sent them a questionnaire. They sent us a SOC2 report. We checked a box. Everyone went to early lunch.

Today, the conversation is different. It is shorter, sharper, and much more paranoid.

When we evaluate an AI tool in 2025, we don’t start with “Do you encrypt data?”. We start with the question that actually matters: “Who is training on my data?”

We Know Because We Build It

We aren’t asking these questions because we are luddites. We are asking because we are builders.

At Sentry, we built AI Debugger that uses Sentry context, not just tells you what broke; it opens a Pull Request to fix it. To make that work, we have to process your code, your stack traces, and your error logic.

We know exactly how tempting it is to say, “Hey, if we just trained on everyone’s data, the model would be 10% smarter.” We also know that if we did that, we would lose the trust of every developer on the planet overnight (see https://docs.sentry.io/product/ai-in-sentry/ai-privacy-and-security/).

We know that trust is binary: you either have it, or you don’t.

So when a vendor tells us, “Trust us, it’s safe,” we don’t smile. We dig.

Few Example Questions

We created a specific “AI Addendum” to our security review. It has three non-negotiable questions.

1. The Training Question

• The Ask: “Does my data improve the model for your other customers?”

• The Trap: Vendors will say “We use data to improve the service.”

• The Reality: “Improving the service” is legal-speak for “We train on your data.”

• The Requirement: We need a hard, contractual guarantee that our data is sandboxed. If Sentry code teaches your model how to write Python, and then that model writes Sentry-style Python for a competitor? Hard Pass.

2. The “Un-learn” Question

• The Ask: “If we churn tomorrow, how do you remove our influence from the model?”

• The Trap: “We will delete your account.”

• The Reality: You can’t “delete” data from a trained neural net. You can’t Lobotomize an LLM.

• The Requirement: Since you can’t un-train a model, we strictly require that you never trained on it in the first place (see Question 1).

The New Operating Order

Here is the good news: The high-quality vendors love this.

The “WeWork Wrappers” crumble under this scrutiny. They don’t have Zero Data Retention policies. They rely on (e.g.) OpenAI’s default settings. They can’t sign the DPA.

But the serious players? They have a “Trust Portal” ready. They have the architecture diagrams showing the isolation between the Vector DB (long-term memory) and the LLM (reasoning engine).

We found that the best vendors in 2025 aren’t hiding their architecture. They are flaunting it. They know that in the enterprise, Privacy is the Product.

The Verdict

We put our final candidates through the Gauntlet. They didn’t just pass; they showed us how they write the rules books for it.

They have the features. They have the security. We are out of excuses.

It’s time to turn it on.

We’re operationalizing AI inside the Sentry Support team in real time, with real customers and real stakes. This series captures what works, what breaks, and what we change as it happens, with a non-zero chance it becomes a cautionary tale instead of a success story.

The Holy Grail.

We thought we found it. In the middle of our vendor search, we found a platform that didn’t just promise “co-piloting.” They promised full Autopilot. 99% deflection. Zero human touch.

The pitch was incredible. We threw complex Python stack traces at their sales team. They showed us demos where it solved them. We threw vague billing questions at it. It answered them.

It was slick. It was fast. It was expensive.

The “Onboarding” Red Flag:

We were ready to move to the Proof of Concept (POC). We expected them to ask for API keys, documentation links, or a dump of our redacted tickets to “fine-tune” their model.

Instead, they sent us a spreadsheet.

It was a questionnaire. And looking at the questions, the mask slipped.

• “What is your monthly ticket volume?”

• “What are your Top 5 most common issue types?”

• “Please list the exact step-by-step resolution for your Top 5 issues.”

• “What are the most common follow-up questions for billing inquiries?”

I stared at the screen. Why does an advanced LLM need me to manually list my top 5 issues?

If this were true Generative AI, it would ingest our 10,000 public documentation pages and our last year of ticket history and tell us what our top issues are. It would figure out the resolution by reading the docs, not by asking me to type it into a cell in Column C.

The Ghost of GPT-2 Past

It immediately clicked. I felt like I was back in 2019.

For those who remember the first wave of “AI Customer Support” (the GPT-2 era), this was the standard playbook. Back then, the tech wasn’t good enough to actually think. So, startups weren’t selling software; they were selling tech-enabled services.

You would send them Excel files and decision trees. They would take those files and hire a massive back-office workforce—usually in a low-cost geography—to manually act as the “intelligence.”

When you chatted with the “bot”, you were actually chatting with a human who was frantically CTRL+F’ing through the spreadsheet you filled out during onboarding.

The Mechanical Turk (MTurk)

That’s why they needed the “Top 5 Issues.” They weren’t fine-tuning a neural network; they were writing a script for a call center agent.

They needed to know the volume so they knew how many humans to hire for the shift. They needed the “Standard Resolutions” so they could tape a cheat sheet to the agent’s monitor.

They weren’t selling us a brain. They were selling us a BPO (Business Process Outsourcing) team wearing a “Generative AI” mask.

The “Values” Check

Let’s call it what it is: Deception.

This wasn’t just a product failure; it was a security nightmare. We are Sentry. We process sensitive data. We value Security, Privacy, Compliance and Resilience (https://sentry.io/trust/) and we thought we were going to pipe some data into a SOC2-compliant, stateless model. In reality, we were expected to handle that data to an unknown human, in an unknown location, with unknown security protocols, armed with a spreadsheet we wrote for them.

The Lesson: Verify

The AI market is currently a gold rush, and where there is gold, there are snake oil salesmen.

At Sentry, one of our core values is “Pixels Matter” - we understand that the difference between a good product and a great one is in the finer details. We take extra care to get everything right, down to the last pixel. We build for developers. Developers are the most skeptical audience on earth. If we shipped a “support bot” that was actually just a hidden human reading a script, our community would find out in a week. They would tear us apart on Hacker News, and they would be right to do so.

We don’t want magic. We want perfect engineering.

New rule, ignore emails with meaningless subject lines (“AI Delivers on CX, and It’s Driving Real Revenue”).

Invited to a black-tie gala with their CEO. Said I’d rather build

Back to work. We have more vendors to test.

We’re operationalizing AI inside the Sentry Support team in real time, with real customers and real stakes. This series captures what works, what breaks, and what we change as it happens, with a non-zero chance it becomes a cautionary tale instead of a success story.

The sun came up. The coffee is brewing. The adrenaline from winning the “??? vs. Buy” argument has faded.

Now I have a terrifying realization: The dog caught the car.

We successfully convinced the organization that we shouldn’t build this ourselves. We argued that “buying leverage” is smarter than “building debt.” The leadership agreed. We have the budget. We have the mandate.

Now, we actually have to find a tool that works.

I opened my inbox this morning. I searched for “AI.”

I shouldn’t have done that.

The Gold Rush

If you are a GTM leader in 2024, you know the sensation. Every SaaS tool you have ever touched, browsed, or accidentally clicked on is now an “AI Platform.”

Your CRM has AI. Your ticketing system has AI. The tool you use to schedule tweets (guilty) has AI.

There are currently two types of companies in the valley:

1. The Incumbents: Massive platforms bolting a chat interface onto legacy code and calling it “Copilot.”

2. The Wrappers: Three engineers in a WeWork with a cool domain name, a dark-mode website, and a thin UI layer over GPT-4.

We are walking into a minefield.
We are looking for the third kind.

The Problem with “Generic” AI

Here is the specific challenge at Sentry: Our customers are developers.

They aren’t asking “Where is my refund?” or “How do I change my password?”

They are asking: “My Python SDK is dropping events when I use the Celery integration with sampling rate 0.5. Why?”

If we put a generic support bot in front of a Sentry user, we won’t just annoy them. We will insult them.

We need a tool that doesn’t just “chat.” We need a tool that understands documentation, reads code snippets, and understands the difference between a “bug” and a “configuration error.”

We need a Librarian, not a parrot.

The Three Mines

As we look at the vendor list, we are looking for three specific things that will kill a deal immediately.

1. The “Wrapper” Mine We just spent Article IV convincing our engineers not to build this because it’s a distraction. I cannot go back to those engineers and say, “Hey, I bought this tool, It’s basically exactly what you said you could build in a weekend.” If the vendor adds no value beyond the raw model - if they don’t have better indexing, better retrieval, or better connectors - we aren’t buying it.

2. The Security Mine Sentry holds the keys to the kingdom for our customers. We process their crashes. I cannot have a vendor that trains their public model on our private data. “Trust us, we promise,” is not a security policy. We need SOC2. We need data isolation. We need a vendor who understands that “Enterprise Ready” means more than just having a “Contact Sales” button.

3. The “Agent” Mine This is the loudest buzzword right now. “Autonomous Agents”. The promise that the AI will go into your system, click the buttons, fix the bug, and email the customer. I love ambition. But I don’t trust a toddler with a chainsaw, and I don’t trust a V1 AI agent with write-access to my production code. For now, we want a Co-pilot, not an Auto-pilot.

The Selection Process

We are going to do this the old-fashioned way.

We aren’t going to look at the marketing videos. We aren’t going to look at the venture capital funding announcements.

We are going to take the hardest, messiest, most technical support tickets from last week - the ones that made our senior support engineers sweat - and we are going to feed them into these tools.

If they hallucinate? They are out. If they give generic advice? They are out. If they tell us to restart the router? They are banned.

Winning the internal argument was the easy part. Now we have to find a partner who is actually building the future, not just marketing it.

Let the bake-off begin.

We’re operationalizing AI inside the Sentry Support team in real time, with real customers and real stakes. This series captures what works, what breaks, and what we change as it happens, with a non-zero chance it becomes a cautionary tale instead of a success story.

Traditional Support waits for the customer to fail. We decided to intercept them before they do.

The Death of the Safety Net

In the world of SaaS, most Support organizations are built like world-class Goalkeepers. We pride ourselves on the “Great Save.” We celebrate the Support Engineer who dives into a 4:00 PM Friday fire, deflects a catastrophic bug, and earns a “Thank You” note from a frantic CTO.

It’s heroic. It’s emotional. And it’s a fundamental strategic failure.

If your goalie is the busiest person on the pitch, you aren’t winning; you’re surviving. You are playing a defensive game, pinned in your own box, waiting for the customer to make a mistake, break the product, or lose their patience. By the time a customer “submits a ticket,” the friction has already occurred. The goal has been scored against you. You’re just cleaning up the net.

From Goalie to Midfielder

We’ve spent the last year burning the “Goalkeeper” playbook. We are moving our defensive line up thirty yards. We are moving into the Midfield.

In soccer, the midfielder is the playmaker. They don’t wait for the ball to reach the goal line; they intercept the pass. They read the game, disrupt the opponent’s flow, and transition immediately from defense to attack.

When we talk about “Moving Support to the Midfield,” we are talking about Proactive Interception.

The traditional triage reality—where a user hits a snag, hunts for a “Contact Us” link, fills out a five-field form (argh), and waits for an asynchronous “we’ve received your request” email—is an obsolete model. In fact, it’s worse than obsolete; it’s a deterrent. It actively punishes the user for needing help.

The 90% Silent Churn

CROs often look at CSAT and think, “Our customers love us!” because the people who write in are happy with the fix. This is a survivor bias trap.

The real danger isn’t the 10% of users who open a ticket. The danger is the 90% of technical traffic that starts and ends in the product. These are the prospects who hit a friction point, can’t find an immediate answer, and—rather than waiting 4 to 24 hours for a human response—simply close the tab.

They don’t complain. They just leave. They are the “Lost” customers, and your Goalie never even saw the ball.

The Fix: Interception

Our strategy document was clear: if the traffic starts in the product, the solution must live in the UI.

By embedding an AI agent directly into the user’s workflow—not buried in a Help Center three clicks away—we changed the physics of the interaction. We stopped asking the customer to come to us. We went to them.

When a developer is staring at a 401 Unauthorized error in the console, they don’t want a “relationship” with a support agent. They want a solution, and they want it at the speed of thought. By putting AI in the “Midfield”—intercepting that error message before it turns into a “Contact Support” thought—we capture that 90% of silent traffic.

We aren’t just “deflecting tickets.” We are protecting the user’s momentum.

The Impact on the P&L

For a B2D2B SaaS company, the Midfield model is the only way to decouple headcount from revenue. If you stay in the Goalkeeper model, your costs scale linearly with your problems. More customers = more tickets = more goalies.

By moving to the Midfield, you are investing in a system that scales through intelligence, not empathy. You save your humans for the “Championship” moments—the high-value architectural consultations—and let the AI play the Midfield, intercepting the thousands of daily “passes” that used to end up as goals against your team’s productivity.

It’s time to stop celebrating the “Great Save” and start celebrating the “Interception.”

We’re operationalizing AI inside the Sentry Support team in real time, with real customers and real stakes. This series captures what works, what breaks, and what we change as it happens, with a non-zero chance it becomes a cautionary tale instead of a success story.

"We Already Have AI" (And Other Logical Traps)

There is a specific kind of pushback you get in engineering-led companies.

It isn’t political. It isn’t emotional. It is brutally, intellectually honest.

When you propose a new tool to a room full of senior engineers and product leaders, you don’t get “No.” You get: “Why can’t we just use the thing we built last Hackweek?”

We have the data (see previous Article). We know the pain is real. But now we have to survive the Meritocracy of Ideas.

We will need to answer the questions that sound reasonable on the surface but may kill projects in the crib:

The “We Already Have AI” Trap The loudest objection is the most logical one: “We already have an AI search bar on our docs site. Why do we need to have one for any other flow?”

The temptation here is strong. If you have a vector database and an LLM, it feels like you have a solution.

But that creates a dangerous confusion. It confuses a Library Index with a Librarian.

Retrieval vs. Reasoning vs. Problem-Solving We need to draw a hard line in the sand between three very different operations:

1. Retrieval (Docs Search): This is for the user who knows what they are looking for but doesn’t know where it is.

   • User: “How do I configure sampling?”

   • AI Search: “Here is the link to the Sampling Options page.”

   • Success Metric: Accuracy of retrieval.

2. Reasoning (Support): This is for the user who is broken and doesn’t know why.

   • User: “My sampling is set to 100% but I’m dropping events on Vercel.”

   • AI Support: “That configuration looks correct, but looking at your SDK version, there is a known conflict with the Vercel adapter in version 7.x. You need to upgrade.”

   • Success Metric: WE got the customer to their final step before a resolution.

3. Problem-Solving (Solutioning): This is for the user who knows what they want to achieve, but doesn’t know the best way to implement it (or maybe they simply can’t).

   • User: “I want to reduce noise. How do I only get alerted to real issues?”

   • AI Solution: “Let me turn on alert rules for new issues only, mute known noisy ones for you, and route P0 only issues to the PagerDuty you enabled in your account.”

   • Success Metric: Job done.

Docs Search throws a book at you. Support reads the book for you, looks at your specific mess, and tells you which page fixes it. This feels like the year 2016.

The “We Can Build It in a Weekend” Trap This is the classic Engineer’s Fallacy.

“It’s just an API wrapper around OpenAI. Give me two engineers and a week.”

They aren’t wrong. They could build the prototype in a week. But we don’t want to build a prototype. We want to run a support organization.

We have to gently remind our own team: We are an Observability Company, not a Support AI Company.

Every hour our engineers spend maintaining a custom RAG pipeline, debugging context windows, or building feedback UIs for support agents is an hour they aren’t building Sentry.

We need to decide to buy leverage, not build technical debt.

The “Is It Safe?” Trap This isn’t an objection; it is a requirement. Sentry holds the keys to the castle for countless companies (read: it’s growing too fast for me to type a number: https://sentry.io/customers/). We process their errors. We see their code.

The pushback here is different. It is quiet, serious, and non-negotiable. If the tool hallucinated an answer, that’s bad. If the tool hallucinated customer data into a chat window, that’s catastrophic.

We can’t just “trust” the model. We need a vendor that treats security like a feature, not a footer on their website.

The Turning Point The conversations are intense. They are detailed. But they are positive. The team isn’t trying to kill the project; they are trying to stress-test the logic.

Once we can prove that:

1. Search is not Support.

2. Building it ourselves is a distraction.

3. The security model holds water.

The mood will shift.

The skepticism will turn into curiosity and the objections will stop being walls and start being requirements.

We have the data. We will have the buy-in. We will have the budget.

On paper, this could work!

It’s now morning, and I had the following thoughts:

As I look at the landscape of vendors - the “Hot New AI Startups” promising to solve all our problems - I realize we are about to walk into a minefield.

Winning the internal argument will be the easy part. Next we will need to find a tool that actually works.

We’re operationalizing AI inside the Sentry Support team in real time, with real customers and real stakes. This series captures what works, what breaks, and what we change as it happens, with a non-zero chance it becomes a cautionary tale instead of a success story.

Data Gathering – Building the Case Beyond "Vibes"

Opinions are interesting. Data is actionable.

There is a specific seduction in the world of AI right now. It goes like this: “Everything is on fire. Let’s buy an AI bot, turn it on, and go to happy hour.”

That is a great way to burn a pile of cash and annoy your customers.

In the last article, I listed the warning signs of AI adoption. To avoid those traps, we realized that before we bought a single license or made a single API call, we had to prove the model. We had to move from “vibes” to cold, hard audits.

We realized we didn’t need to look elsewhere for answers. We have years of history and hundreds of thousands of past interactions. We didn’t need to guess; we needed to “consult” our own archives.

The Great Audit We opened the hood on our ticket volume. We didn’t just look at the number of tickets; we looked at the soul of the tickets.

We categorized them into two specific buckets:

1. The Solvable (Wrongly called “Deflectable”) For these, we didn’t just ask “can an AI answer this?” We identified the root cause of the question:

• The Solution Path: Does the answer actually exist?

• The Discovery Gap: Could the customer have solved this themselves, but didn’t know how?

• The Developer Experience: Was the path clear when onboarding a new functionality, like Sentry Logs?

• The Knowledge Gap: Are the docs up to date? Does searching our KB actually work?

• The Product Gap: Is the customer asking for something that could be made available, but just wasn’t prioritized yet?

2. High-Touch (The Real Work) These are the tickets where humans shine.

• Is the customer experiencing a complex instrumentation issue that requires deep context?

• Is this a stack trace debugging issue in a specific environment, like our Unity/Game Consoles SDK, that actually requires a (open) code change?

The Grunt Work Here is the part nobody puts in the case study because it isn’t sexy: We tagged thousands of conversations.

Yes. Manually.

Like that:

As part of our day-to-day, we added a step that enabled us to re-learn what developers ask us, what their pain is, and what the path to resolution looks like.

Here is the field we learned to love (and hate):

[Insert Screenshot of the manual tagging field/UI here]

We spent long hours tagging thousands of tickets. It felt like the opposite of ‘high-tech.’ But you can’t automate a process you don’t actually understand at the atomic level. It was absolutely necessary.

Evidence > Intuition The result wasn’t a hunch. It was a spreadsheet with undeniable gravity.

The data showed that a significant percentage of our volume was purely transactional—questions that didn’t require empathy or deep debugging, just accurate information retrieval.

We proved the ROI model before we spent a dime. We built the case on the floor of the warehouse, not in the ivory tower.

Now, we had permission to build. I mean: to buy.

We’re operationalizing AI inside the Sentry Support team in real time, with real customers and real stakes. This series captures what works, what breaks, and what we change as it happens, with a non-zero chance it becomes a cautionary tale instead of a success story.

I have played every character in AI tragedy and successes

I haven’t just watched this wave from the beach. I’ve drowned in it.

I have been the Builder, writing the specs. I have been the Implementer, trying to glue the API to a legacy database. I have been the Ambassador, selling the dream on stage. I have been the Owner, signing the checks. I have been the Consultant, billing by the hour to explain AI to various audiences.

Across every single one of those roles, from startup chaos to enterprise meeting rooms,, I learned one universal, painful truth:

AI is incredible. Outcomes are not.

There is a specific kind of noise you hear in AI companies. It’s the sound of people conflating “potential” with “production.” It’s the quiet hum that tells you something is “working” on a demo screen while the actual operational reality is slowly misaligning.

For a long time, I treated that hum as normal. I thought confusion was just the tax you paid for innovation. I was wrong. (I documented the philosophy behind this here).

Those moments weren’t “friction.” They were warning signs. Looking back, the patterns are identical whether you are selling the tool or buying it.

Here is what I ignored then, and what we are refusing to ignore at Sentry now.

1. The Illusion of Motion The first warning sign is the easiest to ignore because it feels productive: the belief that shipping means working.

AI teams are addicted to this. Iteration speed is a survival strategy; velocity creates a sense of safety. When something goes out the door, the internal slack channel claps. It feels like progress.

But output is a dangerous metric.

The early signal is that you keep saying “yes” to launches, yet customers don’t change their behavior. Internally, the product is evolving. Externally, nothing moves.

That gap is where AI initiatives stall. Teams mistake activity for impact and momentum for traction.

2. The Dependency Trap Partnerships in AI usually begin with vision and confidence. Everyone is excited, everyone is aligned, and the future is described in bold language.

Then, the drift happens.

It starts when decisions stop sticking. Ownership becomes blurry. Timelines slide without consequence. The hardest choices get deferred behind vague “alignment” meetings.

At first, it feels like collaboration. Over time, you realize you are building around someone else’s constraints. Instead of gaining leverage, you inherit limitations. Your roadmap becomes a negotiation.

When that happens, you didn’t really choose a partner. You chose a dependency. And dependencies always come with a hidden cost.

3. Confidence Without Competence Onboarding is where the real damage happens.

When AI onboarding is done well, it teaches a mental model. It explains what “good” looks like and makes the limits visible. When it is done poorly, you aren’t onboarding users. You are onboarding confusion.

Teams get excited because the first demo works. Users walk away believing the AI is reliable in situations where it isn’t. They don’t understand why it breaks or what to do when it does.

This is the silent killer. The AI doesn’t fail loudly; it fails quietly. It produces plausible answers that are wrong and confident outputs that are brittle.

The “scaling tax” here isn’t more tickets. It is rework, manual checks, and internal escalation. You end up hiring people not to scale AI, but to supervise it.

4. The Miracle Problem There is a deceptive warning sign that shows up in AI more than anywhere else: early success that can’t be explained.

A demo lands. A customer says “wow.” The organization wants to treat that moment as validation.

But reality is more demanding than a demo environment. The biggest gap appears when the team can’t articulate why something works. In those situations, the excitement becomes louder than the evidence. You start building narratives instead of systems.

The risk is not that AI is hard. The risk is that teams build foundations on unpredictability and get surprised when scale requires control.

5. Confusion is a Signal What I’ve learned is that confusion is rarely random. It is a signal. It tells you exactly where the system is weak.

It shows you where a product isn’t teaching, where a partner isn’t aligned, and where a launch isn’t landing.

Most teams treat confusion like a temporary phase—something that will fade once the technology matures. That is the “good enough” trap. In reality, confusion compounding is one of the main ways companies lose. Slowly.

Surprised? Why? None of this is pessimism. It’s pattern recognition.

The warning signs aren’t there to scare us. They are there to give us time to choose differently while the cost of change is still low.

I keep these signals close to my heart as we explore what AI can really do for our teams.

Beware: I’m a tough customer to satisfy.

We’re operationalizing AI inside Sentry Support team in real time, with real customers and real stakes. This series captures what works, what breaks, and what we change as it happens, with a non-zero chance it becomes a cautionary tale instead of a success story.

The Friction Point – Why We Couldn’t Stay Still

The mirror is a cruel invention.

Sentry exists for one reason: to help developers see what is broken in their code. We are the “truth” platform. The check engine light. The thing you look at when things go sideways.

But when we turn that lens on our own support workflow, the reflection isn’t pretty.

We have a friction problem.

The “Good Enough” Trap

Here’s the thing about metrics: they can lie by omission. Our CSAT scores are over 95%. Our SLA is touching the 98% mark. First time to respond, number of touches? Top performing. If you look at a dashboard in a QBR, you nod and move on to the next slide.

But doing things “as expected” is death in Silicon Valley.

Beneath the “as expected” metrics, our support team is re-educating customers and prospects on what our documents and onboarding experience already cover well. Our product velocity, the sheer speed at which Product and Engineering ship new features, is outpacing our ability to support them.

We are throwing bodies at a physics problem.

The Breaking Point

We realize that adding headcount isn’t scaling linearly with ticket volume, or with the increasing quality, sophistication, and urgency our customers require. We are asking brilliant support engineers to answer the same “What is a DSN?” question, instead of thinking about how our developer customers want to find answers on their own (which is already here: https://docs.sentry.io/concepts/key-terms/dsn-explainer).

That isn’t support engineering; it’s expensive copy-pasting.

“What is Sentry?” is something I discuss at dinner parties (I’m a fun guest), but I can’t stop asking myself: why aren’t developers finding the answers already sitting on this page (https://docs.sentry.io/product/) on their own?

We have to admit a hard truth: our technical velocity is elite, but our tooling is stuck in 2015. We can’t stay still because eventually the wave of ticket volume is going to crash over our heads.

So we decided to stop buying umbrellas and start changing the weather.