You hired an AI agency six months ago. You've had plenty of Zoom calls, seen some impressive-looking dashboards, and received at least three decks explaining what they're building. But your business hasn't changed in any way you can actually measure.
That's not bad luck. That's a pattern.
The AI agency market exploded fast, and a lot of shops set up shop without the substance to back up their pitch decks. Some are well-intentioned but under-skilled. Others have figured out that confused clients don't ask hard questions. Either way, you're the one paying for it.
Here are five specific signs that your AI agency is burning through your budget without delivering anything real. Each one comes with a specific question you should be asking right now.
1. They can't explain what they built in plain English
Ask your agency to describe what they've built. Not in a deck, not in a demo. Just tell you. If the answer sounds like: "We've implemented a multi-modal retrieval-augmented generation pipeline with semantic vector embeddings and a transformer-based orchestration layer" and they can't follow that up with "which means your customers get answers without waiting for a human rep," you have a problem.
Jargon is a professional tool. But it's also a hiding spot. When someone truly understands what they've built, they can explain it to a non-technical person in two sentences. Agencies that can't do this either haven't built something coherent, or they need you to stay confused so you don't realize that.
A good agency does this differently: They define terms before using them. They can explain value in business outcomes, not architecture diagrams. They'll say "this saves your sales team about four hours a week on follow-up emails" before they talk about any technology.
The question to ask: "Forget the technical details for a second. What has this actually changed about how our business operates?" If they pivot back to process and tools instead of results, you have your answer.
For a broader look at what AI agents actually are and what they can realistically do, this breakdown from Empowerment AI is worth your time before your next agency meeting.
2. You're three months in with nothing measurable
Three months is enough time to show something real. Not a finished product, necessarily, but a measurable change in a specific metric. Ticket resolution time down. Lead response rate up. Manual hours eliminated. Something.
Gartner has repeatedly flagged that a majority of AI projects fail to move from pilot to production. A big reason: no success criteria were defined at the start. When there's no agreed-upon definition of success, agencies can stay in "building phase" indefinitely while billing you every month.
Here's what this looks like in practice: You hired them to reduce customer support volume by automating FAQ responses. Month one was "discovery." Month two was "integration planning." Month three is "we're finalizing the training data." Meanwhile, your support inbox still has 400 tickets a day and nothing has changed.
A good agency does this differently: They agree on KPIs in the first two weeks, in writing, before any development starts. They set 30-day, 60-day, and 90-day milestones. If they're behind, they tell you why and what they're adjusting. Not just that they're "making progress."
The question to ask: "What specific number should have changed by now, and what is it currently?" Watch whether they can answer that without looking anything up.
If you want to understand what real ROI from AI automation looks like, our breakdown on AI automation ROI for small businesses gives you the benchmarks worth holding an agency to.
3. They keep expanding the scope instead of finishing what they promised
This one is insidious because it often sounds reasonable in the moment. "We discovered that to make the chatbot work properly, we also need to rebuild your CRM integration." "The original API won't support what we need, so we're switching to a different approach." "We found a much better solution, but it requires an additional phase."
One surprise is normal. A pattern of them is a business model.
Some agencies deliberately underprice the initial engagement to win the contract, knowing they'll recoup on change orders and scope expansions once you're already invested. Harvard Business Review has documented this pattern extensively in IT and consulting contracts. AI agencies are no different.
The tell isn't one scope change. It's that the scope always expands but the original deliverables never actually ship.
A good agency does this differently: They define a clear scope of work in the contract, with specific deliverables and acceptance criteria. When something genuinely needs to change, they tell you before it impacts the budget, not after. And they still deliver the original thing before asking for more money to build something new.
The question to ask: "Show me the original scope. What percentage of it is complete, and when will those original deliverables be done?"
4. Their "AI solution" is mostly people doing the work manually
This is the Wizard of Oz problem, and it's more common than the industry wants to admit. You're paying for an "AI-powered" system. Behind the curtain, a team of contractors is manually reviewing outputs, routing requests, or doing the actual work that the AI was supposed to handle.
The agency isn't lying, exactly. They're building toward automation. But the timeline is fuzzy, the manual work is hidden, and you're being billed as if you have a working AI system when what you actually have is an expensive staffing arrangement.
A real scenario: You hired them to build an AI that categorizes and routes incoming support tickets. They demo a slick interface. But behind it, someone is actually reading tickets and tagging them manually while the "AI layer" is still being trained. You don't find out until three months in when they tell you the model "needs more data" and they need to extend the contract.
McKinsey's State of AI research found that many organizations overestimate their level of AI adoption because manual processes get labeled as AI-assisted. Agencies exploit this ambiguity constantly.
A good agency does this differently: They're transparent about what's automated and what's not at every stage. They give you a clear roadmap showing when manual oversight transitions to full automation, with the specific conditions that trigger that transition.
The question to ask: "Walk me through exactly what happens when a [specific input] comes in. At which steps is a human involved, and which steps does the system handle on its own?" Get them to map it out step by step, not at a summary level.
5. They won't give you access to your own data and models
This is the most serious one on this list. If the AI system your agency built lives entirely in their infrastructure, runs under their API keys, and you have no direct access to the underlying data, models, or configurations, you don't own what they built. You're renting it.
When the contract ends, or when you want to switch, or when they raise their monthly fees, you find out you can't take anything with you. Your customer data, your conversation history, the fine-tuned model they trained on your data: it's all theirs.
This isn't always intentional predation. Sometimes agencies just work this way out of convenience. But the effect is the same: you're locked in, and leaving becomes expensive and disruptive.
Data ownership and AI security go hand in hand. If you don't control where your data lives and who can access it, you have a security problem on top of a vendor lock-in problem. These aren't separate issues.
The FTC has increasing interest in how AI vendors handle consumer data, and regulators in the EU are actively enforcing data portability rights under GDPR. You have legal standing to ask these questions, not just business standing.
A good agency does this differently: They build in your infrastructure or give you clear documentation on how to export and migrate everything they've built. They use your API keys, your cloud accounts, your storage. When the engagement ends, you have everything and they have nothing you didn't choose to share.
The question to ask: "If we ended this contract today, what would we walk away with? Can you show me where our data lives and confirm we have export access?" If they hesitate or redirect, that's your answer.
For a broader perspective on evaluating AI vendor relationships, this HBR piece on build-versus-buy decisions covers how to think about dependency and use in AI contracts.
What to do if you're seeing these signs
First: don't panic, and don't blow up the relationship before you have a plan. Some of these problems are fixable if the agency is willing to be honest about them. Schedule a direct conversation (not another status update call) and ask the specific questions from this list. Watch whether they give you straight answers.
If that conversation doesn't produce clear answers, you have enough information to make a decision.
Second: know that better options exist. There are AI agencies that set real KPIs upfront, work transparently, build in your infrastructure, and can explain what they built without making your eyes glaze over. They're not mythical. They're just harder to find if you don't know what to look for.
Our guide to choosing the right AI agency covers the pre-hire questions in detail. And if you're evaluating chatbot versus voice AI options specifically, this comparison will help you scope the conversation before you talk to any agency.
If you're ready to find agencies that have been vetted against these criteria, browse our agency directory or take the two-minute quiz to get matched with agencies that fit your specific use case and budget. Every agency in our directory has been reviewed for the things this post covers: transparency, data ownership, and measurable delivery.
You hired an AI agency to get results. That's the only reasonable bar. Hold them to it.
Not sure if your current agency is delivering? Take the quiz and we'll match you with agencies that set real KPIs and give you access to what you've paid for. Takes about two minutes.
