Strategy

Human in the Loop as a Service: Build or Buy?

Every agent team eventually builds a review queue, a notification pipe and an on call rota. What that really costs, and when a managed human endpoint makes more sense.

· 8 min read · by the Yoonet team

Somewhere in every agent product's second quarter, the same feature request surfaces: "we need a human to check some of these." What follows is remarkably consistent. A review queue gets built in a sprint. A Slack channel gets wired up. Someone volunteers to watch it. And six months later the team is running a small, unloved operations department that nobody costed, nobody staffed properly, and nobody wants to own.

This post is the build versus buy analysis for that moment — what the built version actually costs, what a managed human endpoint does and does not solve, and an honest account of when each answer is right.

What "just build a review step" actually means

The code is genuinely the easy part. Framework support for pausing an agent is excellent now — LangGraph's interrupts, the OpenAI Agents SDK's approvals — and a competent engineer wires either one up in a day or two. The queue UI is a week. That is the visible 20 per cent.

The invisible 80 per cent is an operations function:

  • Coverage. Approvals arrive when they arrive. Someone answers at 7am and on the Friday of a long weekend, or your agent's p95 latency is "Monday".
  • Consistency. Two reviewers, three interpretations. Now you need guidelines, calibration, and someone who notices drift.
  • Training and turnover. The person who understood the edge cases changes teams; their judgement leaves with them.
  • Attention economics. Engineers make expensive, unhappy, inconsistent reviewers. Interrupting deep work to approve a refund costs far more than the refund.
  • The audit trail. When a decision is questioned later, "someone clicked yes in Slack" is not an answer anyone enjoys giving.

None of this is hard the way distributed systems are hard. It is hard the way running a small call centre is hard — a different discipline that engineering teams neither enjoy nor get better at with practice.

What human in the loop as a service means

The service shape moves the operations problem behind an API. Your agent posts a task — a brief, a type, a return address. A trained person owns it, executes it, and the result comes back as structured data. From the agent's side the human is indistinguishable from any other dependency: called, awaited, answered. That is the whole of what hitl.ph is — one POST /tasks, a named specialist on Yoonet's managed floor in Balanga, a result on your webhook.

What you are actually buying is not the click. It is the rota, the training, the consistency, the named accountability and the coverage — the invisible 80 per cent — priced per task or per month instead of per hire. The managed versus crowdsourced comparison covers why the "named specialist" part matters more than it first appears.

When building is the right call

Buy is not always the answer. Build your own human layer when:

  • The judgement is your moat. If reviewing output is the core expertise of your business — a legal product staffed by your own lawyers — the reviewers are the product, not plumbing.
  • Regulation pins the human in place. Some decisions must be made by a licensed professional or an employee inside a jurisdiction. Outsource the preparation, not the signature.
  • Volume is tiny and colocated. Three approvals a week for a founder who is in the product all day anyway does not need infrastructure of any kind.
  • The context cannot leave. If a task can only be judged with full access to your internal systems, a brief will not carry it. (Though more cases survive being written down than teams assume — a good brief is a forcing function.)

When the service wins

The service shape wins when the human step is necessary but not differentiating: approvals with clear criteria, document checks, data judgement calls, phone confirmations, real world verification — the five task types, roughly. It wins hardest when volume is spiky (a rota sized for the peak idles at the trough), when coverage hours exceed your team's timezone, and when the alternative reviewer is an engineer whose time is worth multiples of the task price.

The arithmetic is rarely subtle. A reviewer hired at even modest cost, loaded and managed, has to clear thousands of tasks a month to beat USD $2.50 a task — and still leaves you owning nights, weekends, sickness and churn.

A one question test

Would you staff this step with a dedicated hire if the agent did not exist? If yes — because the judgement is the business — build, and treat your reviewers as first class colleagues of the agent. If no — because the step is operational glue — buy it as infrastructure, the same way you bought the model, the vector store and the queue. Your engineers did not want to be the human in the loop anyway.

Read next

Or start with the product: the hitl.ph API docs, the five task types and pricing.