Lessons from rolling out agentic AI across a real engineering organisation
Generative AI is reshaping software engineering—but not in the way most people expect. The narrative often goes like this: give developers an AI assistant, plug in a code agent, add GPT-5, and voilà—productivity skyrockets.
Reality is far more nuanced.
At LoopMe, our 20-person Data Science team sits inside a 400-person adtech company, and we’ve spent the last 18 months operationalising GenAI for real engineering work. We’ve lived through the hype cycle, the scepticism, the false starts, and the breakthroughs. And we’ve learned what truly boosts velocity—and what simply doesn’t.
This isn’t a theoretical article. It’s based on actual adoption, real code, real pull requests, real frustrations, and real wins.
Here’s what happened.
We started early—but usage was uneven
Our first experiments began with JetBrains AI Assistant inside PyCharm. Some developers used it constantly. Others ignored it. Most used it like a slightly cleverer StackOverflow: helpful, but not transformative.
Then agentic tools emerged, like JetBrains Junie, and access broadened to the frontier models: GPT-5, Claude 4.5, Gemini 2.5.
Surely now everyone would embrace AI coding? They didn’t. Better tools alone didn’t change behaviour. We needed something else.
We made AI usage visible—but safe
We introduced two simple conventions in every pull request:
- ##AI: <percentage>
- ##Junie: <percentage>
This showed how much of the code was AI-assisted. Then we built (using agentic AI!) a Python script to scrape all pull requests after each sprint so we could track usage trends over time.
Crucially, we didn’t use this for micromanagement. We didn’t publish a leaderboard of “low adopters.” We didn’t shame anyone.
Instead, at our bi-weekly Data Science meeting, we recognised the top three “AI wizards”. Nothing heavy. No pressure. Just positive reinforcement.
Did it increase adoption? Yes—significantly.
Making AI usage visible but not punitive was one of the biggest cultural unlocks.
We discovered something bigger: GenAI requires a different way of coding
The biggest surprise was this: using an AI assistant effectively changes how you think about coding, not just how fast you type.
Traditional coding: “What’s the function I need to write?”
AI-accelerated coding: “How do I frame the problem so the agent can solve 80% of it without derailing itself?”
Teams had to learn a new skillset:
- How to define achievable, bounded tasks
- How to avoid overly vague or overly ambitious prompts
- How to detect when the AI is spiralling into over-engineering
- How to guide the agent back to the real objective
- How to evaluate its output as ruthlessly as you’d evaluate a human
The people who mastered this new mental model saw the biggest productivity gains—sometimes shaving weeks off tasks.
AI became more than boilerplate: it suggested entirely new ideas
Many assume coding agents are mainly good for scaffolding, refactoring, and tests. That’s not what we saw.
We repeatedly got suggestions like:
- “This smoothing can be stabilised with a Laplace prior.”
- “This optimisation resembles a convex projection problem—try XYZ.”
- “Use sparse matrices here; complexity drops from O(n²) to O(n).”
These weren’t regurgitated snippets from StackExchange. They were genuinely creative algorithmic insights. The models, with their vast exposure to techniques and patterns, often offered ideas none of us had considered.
When that happened, the team started to trust AI as a thinking partner—not just a typing assistant.
We also learned what absolutely doesn’t work
1. Expecting usage to rise automatically as models get better
Switching to GPT-5 didn’t magically increase adoption. Nor did adding Claude 4.5 or Gemini 2.5. Tools don’t change behaviour. Rituals do.
2. Letting AI usage remain a private, individual habit
Without visibility, adoption stalls. People think “no one else is doing this,” or “maybe this isn’t allowed,” or simply forget to use it.
Making usage socially normal—without pressure—was essential.
3. Expecting AI to control its own scope
Agents happily generate complexity: elegant abstractions, nested class structures, entire architectures that solve a problem you don’t have.
Humans still need to:
- keep the scope tight
- prune complexity
- recognise dead ends
- apply context
GenAI accelerates everything—including going in the wrong direction.
The biggest surprise: junior team members adopted GenAI fastest
We’ve had several data analysts transition into full data scientists. They learned most of their heavy coding with GenAI tools. For them, “AI-first coding” isn’t a shift—it’s the default.
Meanwhile, some experienced engineers were slower to adapt. Not because they’re less capable, but because their muscle memory is stronger.
This mirrors what we hear across the industry: the next generation of engineers will expect AI-first workflows by default.
Gamification helped more than we expected
Every two weeks, we showed:
- The trendline for AI usage
- Sub-team progress
- The top three AI power users
This turned adoption into something fun and social. Everyone improved, and usage rose naturally—without any of the politics or resentment that naming-and-shaming would have caused.
So… did productivity actually improve?
Early signs say yes. We’re seeing:
- More high-quality, peer-reviewed code
- Faster prototyping
- Faster refactoring
- Faster “first viable attempt” at new tasks
- Significant speedups in complex optimisation work
Anecdotally, several multi-week tasks were completed in days. Quantifying pure productivity is tricky, but the qualitative evidence is strong.
GenAI isn’t just a tool—it’s a new engineering discipline
The organisations that treat it as such will move faster than those waiting for “the perfect agent” to arrive.
The teams that win will be the ones that:
- Make usage visible and celebrated
- Teach engineers how to think differently
- Use AI for thinking, not just typing
We’re still early in this transition—but the velocity curve is bending in the right direction.
If you’re adopting GenAI in your engineering team, I’d love to hear what’s worked (and what hasn’t) for you.
—Leonard Newnham, Chief Data Scientist
