An agent management system is the answer to that second question. It is the layer between a working prototype and something a business is willing to leave switched on.

Four things a prototype never has

• An identity. Every tool call happens as someone - the agent's own service connection, the person who started the session, or a credential supplied on the entry call. Which one it is decides what the agent can reach and whose name is on the audit row.
• A place for people to stay in control. Some actions should stop and wait for a human. That decision belongs in the agent's configuration, not in the wording of a prompt.
• A way to start that is not a chat box. Real work arrives on a schedule, from your API, or from an event in a system you already run.
• A record. Every model turn and every tool call, kept long enough to answer a question somebody asks next quarter.

Identity is the one people underestimate

It is tempting to give an agent one powerful set of credentials and move on. It works immediately, and it quietly makes every later question unanswerable: the audit log in the connected system shows one service account doing everything, so "who deleted that record" has no answer beyond "the agent did".

Binding a tool to an identity is a configuration decision in Yekar, made per agent and per integration. An agent can act as the domain's shared connection, as the person who triggered it, or with a credential handed to it on the way in and thrown away when the run ends. There is no fallback between those modes - an agent configured to act as its caller does not quietly borrow the organisation's stored credential when no caller token arrives.

Approvals are a control, not a prompt

"Ask me before sending anything" in a system prompt is a request. The model usually honours it. Usually is not a control, and the gap between usually and always is exactly the class of incident that gets agents banned from a company.

A safeguard that lives in the prompt is a safeguard the model can talk itself out of.

So gated actions pause the run and wait for a named approver. The agent's turn suspends, the person decides, and the run resumes from where it stopped - with the decision recorded beside the call it authorised.

The record is the product

Once an agent is doing work that matters, the transcript stops being a debugging convenience and becomes the thing you are actually buying. What did it read, what did it change, which model produced that answer, which version of the configuration was live at the time, and who approved the one step that needed a person.

That is the whole argument for a management system. The agent is the easy part. What lets you leave it running is everything around it.

Two ways to miss a document

A pure vector search misses the query that shares no vocabulary with the answer's phrasing. A pure keyword search misses the paraphrase. Both failures are common enough that picking one is picking which half of your questions go unanswered.

So the search runs both arms and fuses them, rather than choosing. A document that is a strong lexical match and a document that is a strong semantic match both surface; a document that is both surfaces higher. An organisation with no embeddings configured degrades to the lexical arm alone - which is a state, not an error, and the surfaces say so rather than pretending the results are complete.

The abstention floor

Fusing two arms improves the ranking. It does nothing about the case where the corpus simply does not contain the answer, because a ranking is relative and the top of a list of bad matches is still the top of the list.

That needs an absolute threshold. Below it, the agent is told the library has nothing on this, and answers accordingly - which reads as a worse demo and is a far better product. "I don't have anything on that in your knowledge base" is a sentence a person can act on. A confident paragraph assembled from unrelated chunks is not.

The measure of a retrieval system is not what it returns when it knows. It is what it does when it doesn't.

Citations land in the original file

• A citation that names a document is a starting point for a search. A citation that opens the document at the page the sentence came from is a verification. The difference matters most for exactly the readers you most need to convince - the ones who will check. The uploaded file is what gets served and rendered, not a reconstruction of it.
• Chunks carry their page and section, so a citation resolves to a location rather than to a filename.
• Citations are derived on the server from what the turn actually retrieved, so prose naming a file the agent never opened does not become a link.

That last one is a real class of bug. The displayed citations were never forgeable - but the sentence around them was, and a model that mentions a policy document it did not read produces a paragraph that looks sourced. Catching that is a check on the answer, not on the retrieval.

The actual list

• A credential expired, and the tool's error came back as a string the model treated as data - so the agent reported the outcome of an action that never happened.
• The agent produced output in the shape it was asked for on nine turns out of ten, and the tenth broke the consumer downstream.
• A tool returned an empty result and the model read the emptiness as an answer.
• A run was retried after a timeout, and the side effect it had already committed happened twice.

None of these are interesting. All of them are the reason someone turns the agent off.

Failure has to be a row, not an absence

The first thing that goes wrong with agent monitoring is counting only what succeeded. A dashboard built from successful runs looks complete and under-reports failure, because a turn that died is simply missing from it. You cannot see the shape of your failures in a table that only records the wins.

So every settled turn writes a row - its outcome, the tools it called, how many, how grounded the answer was, whether anything had to be repaired mid-turn. The row holds facts, never message bodies, which is what makes it safe to keep long after the transcript has aged out.

Repair beats refusal

When a turn produces output that does not match the shape the agent declared, or prose that cites a document the turn never retrieved, the useful move is usually not to fail the run. It is to tell the model what is wrong and let it fix it once - and then, if the second attempt still misses, to let the answer stand with the problem recorded rather than throwing away work the person is waiting on.

An agent that refuses on every imperfection is an agent nobody leaves running. One that repairs quietly and records the residue is one you can improve.

The recorded residue is the point. Every unrepaired finding is a countable fact against a real denominator, cut by the configuration revision that was live when the turn ran - which is the only way to answer whether last week's change made things better or worse. A flat average across a window that spans a fix averages the fix with the bug it fixed.