Split Decision: Where to Insert the Human in an Agentic AI Workflow?

I checked. I fixed it. I checked again. It still wasn’t fixed — until I lost the one thing that could tell me for sure.

A “fixed” report from an AI system is a claim, not a fact — and the only way to know the difference is to check it against the original source while you still have access to that source. Once that access closes, “fixed” is the only version of the truth you’re left with, whether it’s true or not.

Last week I told you about the moment I realized I’d handed off more than a task — I’d handed off authorship of “done.” An AI agent extracted 281 files from my content library — the trainings and materials I’d built up from memberships to different services where I learn about AI — and I let a status update stand in for a real check. The lesson was: keep your judgment in the loop.

Here’s the part I didn’t tell you yet: keeping my judgment in the loop wasn’t actually the missing piece. I was checking. I spot-checked. I caught the first problem, had it rebuilt, and believed it was resolved. And it still went wrong — twice more — before I lost the one thing that would have let me check again for good.

If last week’s piece was about noticing when you’ve stopped checking, this one is about what happens when checking itself isn’t enough.

Key Takeaways

A file can look complete — titled, populated, present — and still be hollow where it counts. “It exists” is not the same test as “it’s right.”
Fixing a flagged problem doesn’t guarantee the fix held everywhere else. AI systems can resolve what you pointed at and quietly regress on what you didn’t.
Verification has an expiration date. The moment you lose access to the original source, your last “fixed” report becomes the permanent record — whether or not it was ever true.
The question worth asking before you trust “fixed” isn’t “did I check?” It’s “could I check again, right now, against the original — not against what the AI told me?”

When the Check Passes and the Work Still Fails

I’d had access to my content library for a long time. Between the podcast, the newsletter, and everything else running through Omni Mindfulness, I never had the bandwidth to go through it by hand — so extracting it in bulk with AI wasn’t a shortcut I took carelessly. It was the right call. Reducing that friction was smart. (If you read last week’s piece on friction, you know I stand by that instinct completely.)

The first large batch came back looking done. Files existed. They had titles. They had content. What they didn’t have was the actual substance of the originals — they were shallow summaries wearing the shape of full documents. Not missing. Not broken in an obvious way. Just… thinner than what was actually there. The kind of wrong you don’t catch by glancing at a file list.

I had it rebuilt. I spot-checked the results. I believed it was resolved.

Then I went back to the original pool and found scattered patches — not all of it, not none of it — that were still improperly extracted, even after the rebuild that was supposed to catch everything.

Fixed it again. Spot-checked again. Believed it, again.

Moved on to new batches — 25 files, then 50 more. Went back to check those, too, and found that nearly all of them were also broken, in the same way, on work the rebuild was specifically supposed to protect going forward.

By the time I understood the real scope of it, I’d lost access to the source library entirely. No re-pull. No re-verification. No going back to the original to settle, once and for all, what was actually true.

This is what agentic AI means in a moment like this: a system carrying out a multi-step task — extract, evaluate, rebuild, re-extract — across many files, on its own, without a person approving each individual step along the way. That’s exactly what makes it powerful for a solopreneur with more content than hours. It’s also exactly what makes “I checked it” not the safeguard it sounds like, if what you checked was the system’s own report of its work rather than the source underneath it.

The Evidence

This isn’t just a me-problem. Research on AI systems performing repeated, multi-step revisions keeps finding the same pattern: fixing what’s flagged doesn’t protect what wasn’t.

A 2026 ACL study testing five deep-research AI agents across multiple rounds of revision found that agents reliably fixed the specific feedback they were given — but regressed on 16–27% of previously correct content and citation quality outside the requested scope, on every single round (Chen et al., “Beyond Single-shot Writing: Deep Research Agents are Unreliable at Multi-turn Report Revision,” ACL Anthology 2026). The fix worked. The rest of the work didn’t stay fixed.

A related preprint studying AI coding conversations found something similar at a larger scale: across 542 tasks and six models over multi-turn conversations, 40–73% of tasks lost previously correct behavior somewhere along the way (Huang et al., “Regression Accumulation in Multi-Turn LLM Programming Conversations,” arXiv preprint, July 2026 — not yet peer-reviewed). Their best mitigation wasn’t a smarter model. It was a verification gate: rerun everything that was previously confirmed correct after every change, and roll back the moment the pass rate drops. That’s a master list and a habit of re-checking the whole pool, not just the part you just touched — which, not coincidentally, is the exact habit I built after this happened to me.

And having a second person check the work isn’t automatically the fix, either. Research on automation bias — the tendency to trust an automated system’s account of its own performance — has found that adding a human checker doesn’t reliably catch more errors unless that person is checking against independent evidence, not just the system’s summary of what it did (Skitka, Mosier, Burdick & Rosenblatt, 2000; Lyell & Coiera, JAMIA 2017). The checking has to reach past the report and touch the actual thing.

Gartner has forecast that over 40% of agentic AI projects will be canceled by the end of 2027 — not because the technology fails, but for three compounding reasons: escalating costs, unclear business value, and inadequate risk controls (Gartner, June 25, 2025). That third one is this article.

What the Community Archive Says

I went back through the shared archive from one of those memberships — the community where other members post their own AI wins and misfires — looking for anyone who’d lived through exactly what I did: an AI declaring a batch fixed, twice, and being wrong both times. I didn’t find that exact story. What I found was a handful of different stories with the same problem underneath: an AI system doesn’t get to decide, on its own, that something is actually done.

One member described asking ChatGPT for a document and getting ninety minutes of confident narration about work being done — without ever receiving the actual document — until she pushed back directly. Another described losing hours of in-progress work to repeated mid-project access lockouts. A third watched an AI propose, criticize, rearrange, and then reverse its own recommended website structure in a loop that never actually landed anywhere. And one member’s agent ran up $817 in charges stuck in an autonomous retry loop with no circuit breaker to stop it.

None of those is my story exactly. All of them are the same story underneath: a system reporting progress, or resolution, that wasn’t verifiable from where the person was standing — until they either caught it by chance or paid for not catching it.

And in Sources, drop the club-named line entirely — no public citation needed for an internal/anonymous source:Four checkpoints now sit inside every agentic workflow I run:

Check the source, not the summary. A status update from an AI system is a claim. The original file, the original page, the original data — that’s the independent evidence. Go to the evidence, every time, not the narration of it.
Re-verify the whole pool after every fix, not just the flagged item. A master list, checked start to finish, whenever something is declared “resolved” — not just spot-checked at the edges.
Preserve your own copy of the source before you hand off bulk work. If there’s any chance access could close — a subscription, a shared drive, a site you don’t control — get a portable copy first.
Treat “I checked it” as a question, not a conclusion. Checked against what? If the answer is “against what the AI told me,” that’s not a check — that’s a second opinion from the same source.

Woman at the same wood desk, calm and focused, checking one translucent file card against a larger master-list panel on screen — representing a single item verified against an independent source

FAQ

Is this saying AI can’t be trusted with big tasks?
No. It’s saying the checkpoint needs to sit at the right place — before anything irreversible, verified against the real source.

What if I don’t have access to re-check something later?
That’s exactly the risk this piece is about. Get your own copy of anything you can’t permanently access, before you start, not after.

Does having a second person review the AI’s work solve this?
Only if that person checks against independent evidence — the original file, not the AI’s summary of it.

Is this the same lesson as last week’s article?
Related, not the same. Last week was about noticing when you’ve stopped checking at all. This week is about what happens when you are checking — and the thing you’re checking against changes or disappears before you’re done.

What’s the one habit that would have prevented this?
A master list re-verified in full after every “fix,” checked against the original source, with a portable copy secured before access could close.

The Close

I don’t regret extracting that library. Reducing that friction was the right call, and I’d make it again. What I’d change is where I put my trust while it ran — not in the last report that said “fixed,” but in a source I could still reach myself, for as long as I needed to.

As a solopreneur, the checkpoints you build once are the ones that hold up long after the access, the tool, or the moment has moved on without asking your permission.

You don’t need to master every AI tool to avoid a mistake like this one. You need the right skills, thoughtful guidance, and a place to keep learning as AI evolves — which is exactly what AMPLIFY YOU™ Club is for.

Amplify You Club membership card — $144/month, with Shilpa Lewis on a video call with community members, offering monthly AI workshops and ongoing support for solopreneurs

Join Amplify You™ Club

It’s a practical learning space for solopreneurs who want to use AI with more confidence — without losing their authentic voice or getting overwhelmed by every new tool. For $144, you get monthly workshop sessions on real-world applications, AI Skill Files and playbooks you can put to work right away, support and accountability as you implement, and a human-centered approach to using AI to amplify your expertise — not replace your judgment.