A small business owner we talked to this week set Meta's Muse loose on her inbox. It drafted a reply to a customer asking whether a product worked with an older model of their equipment. Muse said yes. It was wrong. Nobody had told it the answer, so it built one out of the surrounding text and answered like it knew.
That is not a bug report. It is the entire risk of putting a personal AI agent in front of a customer, and it is worth naming precisely instead of waving at generally.
Muse is a personal agent. It browses, it books, it shops, it manages your calendar and your inbox with your permission attached to each step. None of that requires it to know anything about your company.
It has no concept of your knowledge base. It has no memory of your last thousand support tickets. It cannot be pointed at your product documentation and told this is the ground truth, answer from here and nowhere else. A purpose-built support agent is trained on exactly that material before it goes live. Muse was never given the chance.
A well-built customer-facing AI system has a gate: if the model's confidence in an answer falls below a set bar, it stops and either asks a clarifying question or hands off to a person. That gate is not a nice-to-have. It is the difference between "I don't know, let me check" and a wrong answer delivered with total certainty.
Muse does not have that gate, because it was never built to need one. A personal agent that books the wrong restaurant is a minor annoyance you can fix yourself. A support response that confidently confirms compatibility, a return policy, or a price that doesn't exist is a commitment made in your company's name, to someone you may never get the chance to correct.
A confidence threshold isn't a slider you flip on. It's built by testing a model against a large batch of real, already-answered questions before it ever sees a live one, and measuring exactly where it starts guessing instead of knowing. That measurement is what tells you where the handoff to a human needs to sit. Skip the measurement and you don't have a threshold. You have a model that always sounds equally sure, whether it's right or not.
Meta built a real safeguard into Muse called Sentinel. It requires your sign-off before the agent takes a sensitive action, like sending a message, making a purchase, or completing a booking. That is a genuine control, and it works as designed.
But Sentinel approves or blocks an action. It does not evaluate whether the reasoning that produced the action was sound. If Muse decides, incorrectly, that your product supports a customer's use case, Sentinel's job is to check whether it's allowed to send that message, not whether the message is true. The permission layer and the judgment layer are two different problems, and Muse only solves one of them.
At Debsan we draw a hard line between two kinds of AI, because the businesses we work with keep confusing them. An AI copilot drafts a suggestion and waits for a person to check it before it goes anywhere. An AI operator is different: it is an AI system with write access inside a company's systems of record, positioned to execute the next step itself instead of surfacing a suggestion for a person to relay by hand.
Being an operator is not automatically good or bad. It's a design decision, and it comes with a price of admission. An operator earns write access into a customer conversation, a CRM record, or a support queue only after it has been trained on the company's actual material, tested against real historical cases before it ever sees a live one, and built with a threshold that makes it stop and defer when it isn't sure. Skip that groundwork and an operator doesn't save you coordination cost. It just moves the mistake closer to the customer, faster, and in your name.
Muse, for a business use case, is trying to act like an operator in front of your customers without having earned any of that. It has the permission model. It does not have the preparation underneath it.
None of this means AI has no place in customer service. It means the bar for putting AI directly in front of a customer is higher than the bar for using it to save yourself twenty minutes on your own calendar.
A system that earns that trust has three things Muse does not: it is trained on your specific knowledge base and past tickets, not the open internet. It is tested against a batch of your real historical questions before it answers a live one, so you can see its error rate before a customer does. And it has an explicit threshold below which it defers to a person instead of guessing.
This matters more, not less, for a manufacturer or a service business selling something physical. "Does this part fit my model" and "is this covered under warranty" are exactly the questions where a plausible-sounding wrong answer does real damage, because the customer acts on it before anyone catches the mistake. A consumer AI agent trained on the open internet has no way to know your model numbers, your warranty terms, or which parts of your own documentation are actually current.
This is exactly the kind of build an AI Forward Deployed Engineer does inside a company's actual systems, and it is the mechanism we described when we first drew the line between an AI copilot and an AI operator. Debsan doesn't sell Muse and wouldn't try to. It's a free consumer app doing exactly what it was built to do. What we build instead is the operator-grade version of the same idea, scoped to a company's own knowledge and tested before it ever talks to a customer.
If you've already let a personal AI agent draft a customer reply, go back and read it slowly. You'll usually find the sentence where it stopped knowing and started guessing.
Muse earns its place doing what it was built for: your errands, your inbox triage, your personal admin. It has not earned a seat in front of your customers, and the honest reason is simple. Nobody trained it on your business, tested it against your history, or told it when to stop and ask.
That gap does not close with a better prompt. It closes with the groundwork a real business-grade system requires before it gets write access to anything that touches a customer.
Can I let Muse handle simple, low-stakes customer questions?
Only if you are comfortable with an occasional confidently wrong answer reaching a real customer, since there is no confidence gate to catch it first.
Isn't Sentinel enough of a safety net?
No. Sentinel checks whether an action is allowed, not whether the reasoning behind it was correct. Those are separate problems and Muse only solves the first one.
What would it take to safely automate part of our support inbox?
A system trained on your actual knowledge base and ticket history, tested against real past questions before it goes live, and built with a threshold that hands off to a person when it isn't confident.
Where does Debsan fit if we're not ready for that yet?
We build the version of this that earns write access properly, scoped to your systems and tested before it ever reaches a customer, which is the AI Forward Deployed Engineer model applied to support instead of sales or finance.