Compliance Is the Boring Part of Agentic Apps. Build It First Anyway.
Every team building agentic software right now is chasing the same question: can the agent actually do the thing? Book the flight, close the ticket, move the money, file the form. Fair enough, that's the fun part.
But there's a second question almost nobody asks until something goes wrong: if the agent did the wrong thing, could you explain why? Not "the model decided to," but an actual chain you could show a regulator or a nervous customer and have it hold up.
That's compliance. And it doesn't work the way it used to. Under the EU AI Act, the toughest obligations for high-risk AI systems, the exact things this post is about, like logging and human oversight, were originally due to land on August 2, 2026. That deadline just got pushed back to December 2027, after EU lawmakers agreed to give companies more time. So there's breathing room. But the direction hasn't changed, only the runway. Regulators are still heading toward requiring this, and building it in now costs a lot less than retrofitting it under a deadline later.
"We log everything" isn't an answer
Old-school compliance is basically about access. Who touched what, when, with what permission. You bolt on some audit logs, add role-based access control, write a retention policy, and a compliance officer signs off and goes home.
Agents break that, because what you're auditing now isn't a fixed path through code. It's a decision. Your agent doesn't run step four because a developer wrote if X then step 4. It runs step four because, given a prompt, a set of tools, and whatever context it pulled in, it decided step four was the move. Do the same thing tomorrow and it might pick step six instead. Same model, different day, different retrieved context, different outcome.
So when someone says "don't worry, we log everything," what they usually mean is they have gigabytes of raw data nobody can actually read. That's not an audit trail. A real audit trail explains itself: why the agent made the call it made, what it did, what information it used to decide, and whether any safety checks caught it or missed it. Someone outside your team should be able to read it and understand what happened, without you in the room to translate.
It's a bigger gap than most teams realise. One recent study found that 82% of enterprises had AI agents running that their own security teams didn't know about. Another found that while most large companies now have someone in charge of AI, only about one in eight of them actually think their AI governance is good enough.
Where I've felt this the most
I've spent most of my career in fintech and energy, and lately I've been pulled deeper into agentic and RAG-based tooling, so none of this is theoretical for me.
Take fraud detection. If an automated system flags or blocks a transaction, "the model was pretty confident" is not going to satisfy a regulator. You need a defensible line from signal to decision, something you could put in front of someone whose job is to distrust you by default.
Or take tax filing. An agent that submits or amends something on someone's behalf can't just be right most of the time. It needs a clear line: what it can decide on its own, and what has to wait for a human to check it first.
Different domains, same lesson: compliance isn't something you log after the fact. It's something you design into where the agent's autonomy actually ends.
What's actually worked, in practice
A few things have made a real difference for me, as opposed to just generating paperwork.
First, track what the agent did, not what it said. A chat transcript is for humans reading in the moment. What you'll need later is a record of every action the agent actually took: which tool it called, with what inputs, what came back.
Second, keep that record somewhere the agent can't touch. If the agent has write access to its own logs, it can quietly edit its own history, whether through a bug or a compromise. Store logs separately, with separate credentials.
Third, don't let the agent do everything on its own just because it can. Give it a wide range of tools if you want, but put a hard stop in front of anything irreversible, moving money, filing something official, deleting a record. Make it wait for a human to confirm first.
Fourth, keep track of what changes. The prompt, the model version, the data it pulled in. If someone asks in six months why the agent did what it did, "here's the current code" won't answer that. You need to know what things looked like on the day it happened.
And fifth, remember who you're really building this for. Engineers want to know "did it work?" Compliance wants to know "who allowed this, and can you prove it?" Those are different questions, and they need different answers.
The unglamorous truth
None of this demos well. Nobody's tweeting about your audit trail. It's not going to be the thing that gets your launch noticed. But it's the thing that decides whether your agent is still running in a year, or whether it's the reason a client's legal team stops taking your calls.
Teams that build this in from the start get to keep shipping autonomy. Everyone else ends up rebuilding it after the incident that forced the conversation, usually at a much worse time, under real regulatory pressure, and with a lot less goodwill in the room.
I write about building in fintech, security, and AI-native tooling. If you're working through similar problems in your own agentic systems, I'd genuinely like to hear how you're approaching it.
