• LinkedIn
• Twitter
• Instagram
• YouTube
• RSS Feed

Apple •Spotify• Pocket Casts •Youtube •Overcast •RSS

What’s up folks, welcome to our 4 part series of Crawling through the dungeon of martech architecture. You’ve arrived at Part 3 : The Correlation Masquerade.

Summary: Your warehouse is clean, your agents are running, everything looks like it’s working, and that’s the trap. This floor is where AI mistakes correlation for causation and scales the mistake at machine speed, eroding revenue behind a green dashboard. We unpack the fix: holdouts, guardrails, and a causal context graph that proves what actually works.

In this Episode…

• Why Agentic AI Optimizes for the Wrong Thing at Scale
• The Boomerang Effect on AI that Erodes Revenue
• Why Marketing Attribution Data Can’t Tell AI Agents What Actually Caused the Result
• How Bad Signals Masquerade as Evidence
• BOSS BATTLE: The Correlation Boomerang Archer
• Reducing Exposure While the Foundation Is Built
• Building a Causal Memory Layer With a Context Graph
• Achievement unlocked: Causal Evidence Layer Established

If you’re just joining, go back to parts 1 and 2, where we demoted the CRM, built the warehouse, engineered the context layer, and built the shared meaning infrastructure that keeps agents from misinterpreting what they read.

Episode 1: CRM Gravity: We conquered the source of truth and discovered that the data warehouse replaces the CRM with portable audiences.

Episode 2: The Eye of Context: We learned why AI fails without context engineering, built the shared meaning infrastructure, and dug into why the industry built the wrong kind of meaning infrastructure in 2012.

Episode 3: The Correlation Masquerade: Today, we escape the correlation trap and build the causal memory layer that separates agents that optimize correctly from agents that confidently scale the wrong behavior.

Episode 4: The Dispatch Tower: Next, we tackle the governance chaos of 30 vendors all claiming authority, and confront the interface decision that most organizations already made without realizing it.

Let’s continue our descent.

Okay so we’re making our way down to the third floor with blood sweat and tears. But we’re feeling good. Our data is clean-ish. You’ve built a context bundle that we’re proud of and we collaborated on it with multiple people and shared definitions. We’ve got a nice big fancy data warehouse as our source of truth.

Our warehouse holds a complete record of what happened. We can query patterns, correlations, historical campaign data, audience behaviors, outcome signals: all of it is available. 

But the problem we’re about to find out is that none of what we’ve built so far can tell an agent whether the thing it’s optimizing for was ever the right thing to optimize for. None of it explains why an intervention worked, or whether it worked for the reason the model assumes it did.

Let’s step through.

FLOOR 3: THE CORRELATION MASQUERADE

The layout of the correlation masquerade is like a high speed train to nowhere. 

You’ve spent two whole floors meticulously cleaning the “atoms” of your historical customer data and building a sturdy warehouse so that you can let AI and agents loose on the data. Maybe you’re starting to play with ‘next best action sequences’, building propensity models predicting the likelihood that certain cohorts of users will churn, maybe running reinforcement learning loops on historical context and doubling down on your best campaigns. 

Everything looks like it’s working… until the world rumbles and you realize you’re still in a trap.

The layout of this floor is a room where every single action comes back wrong. And it’s not technically AI’s fault, they’re just optimizing for a finish line that’s actually just the end of the first heat of 8 heats. It’s a trap.

Why Agentic AI Optimizes for the Wrong Thing at Scale

Jason Dobbs, the Head of Marketing Ops and GTM Engineering at Kumo describes it like this:

“A warehouse is a record of what happened. It’s not a rulebook for what an agent should do next. If you let a generic agent optimize directly on historical correlations with unbound authority, you can absolutely scale the wrong behavior. A product that correlates with high LTV does not necessarily cause high LTV. Prediction is not policy. Once you cross into action, you still need guardrails, business rules, approvals, evidence that it’s actually driving business outcomes.”

JASON DOBBS, episode 221

An AI system that treats prediction as their gold standard will happily optimize for proxies while the actual outcome you care about degrades.

The Prediction Trap

Tobias Konitzer spent years studying this failure as a computational social scientist before bringing that lens to marketing. His argument is that predictive models describe what’s already happening, and marketing is about changing what’s going to happen. Those are different jobs, and the data warehouse doesn’t distinguish between them by default.

“The nature of predictive models is they represent the status quo. Someone is going to churn, or someone is not going to churn, but that is the status quo. And there is really no point for marketing if everything is just status quo. There is no marketing role here.”

TOBIAS KONITZER, Episode 212

He describes a CRM head at a billion-dollar outdoor brand who found that high LTV customers had a strong correlation with viewing a specific product, a pair of jeans. The obvious response was to push those jeans into the welcome flow. The correlation ran in the wrong direction, though. Those customers already had high LTV before they saw the jeans. The jeans showed up alongside the relationship, long after it was established.

Scaling that logic into the welcome flow pushed irrelevant products onto a broad cohort with nothing in common with the original high-LTV segment. The analysis was reproducible and data-supported; it just never verified whether the jeans caused the high LTV or merely accompanied it. An agent running the same logic would scale the error across every customer who matched the surface pattern, efficiently and invisibly, before anyone stopped to ask.

Tobias calls this lazy thinking, and risky thinking: analysis that feels like rigor because it’s data-supported and reproducible, but skips the question that would disqualify the conclusion.

The Boomerang Effect on AI that Erodes Revenue

When you let agentic AI loose on the data warehouse it has access to a TON of data. That’s amazing. But raw data and then events and actions often leads to predictions that are based on correlation, and not causation.

For example, let’s say you task an agent with improving the number of free users that convert to paying users. The agent sees in the past that a discount campaign to a certain cohort of active free users on a certain day resulted in a high % of paying user conversions.

The agent concludes: this campaign works. Scale it.

But what it can’t see: those active free users were already the most likely to convert, they were on the edge of paying with or without the discount. The campaign didn’t cause the conversions. It just correlated with them. The agent found the easiest pattern in the data and called it a lever.

Run that campaign at scale and a few things happen. You hand discounts to users who would have paid full price. You train your most engaged cohort to wait for an offer before converting. And your conversion rate dashboard looks excellent while your revenue per conversion quietly erodes.

The boomerang: by the time you see the damage in the numbers, the agent has already run the same logic against every cohort that matches the pattern. It didn’t fail. It succeeded, at the wrong thing, at speed, before anyone asked whether the signal was causal.

Simon Lejeune, VP of Growth at Wealthsimple, made this question the default response to any campaign result shared without an incrementality breakdown. At Wealthsimple, it became policy:

“We launched a promotion and dropped the price by 10 percent and sales went up by 25 percent during Black Friday. I’m like, okay, so what was the impact of the campaign? Well, the sales went up by 25 percent. I’m like, is that the impact of the campaign? It’s almost a meme at Wealthsimple. Every time someone shares results and I don’t see the incrementality analysis, I answer in the thread: okay, but what’s the impact? Especially when you start playing with pricing and discounts and incentives. If you give free money to people, they’re gonna do something. But are you sure they weren’t gonna do it anyway?”

SIMON LEJEUNE, Episode 182

That question is the one agents don’t ask. They see the result. They scale the pattern. The discount worked, by the only definition the data gives them.

The Boomerang Effect

Here’s Tobias Konitzer, VP of AI at GrowthLoop again. And remember that his lens is a computational social scientist who spent a decade studying how decisions backfire at scale.

“If you put a gen on audience building and you just ask this thing, give me an audience that causally minimizes churn, it’ll give you an audience of highly engaged people that always buy. Why? Because correlation, they never churn, right? But causally it’s nonsense. It’s probably the worst audience that you can pick.”

TOBIAS KONITZER, Episode 212

That is a system reading a warehouse full of correlations and confidently producing the wrong answer. The highly engaged buyers it surfaces were never going to churn. Sending them retention offers is irrelevant at best and alienating at worst. You’ve spent marketing budget reinforcing a relationship that didn’t need reinforcing, while ignoring the customers who were actually at risk.

Let’s dive into another example. Sundar Swaminathan spent 5 years building marketing data science teams at Uber, and his team ran exactly this test on their largest performance channel. Facebook correlated strongly with signups. Customer acquisition cost was swinging 10 to 20 percent week over week with stable spend. The instability was a signal. They ran a 3-month incrementality test on the entire channel.

“It came back that Facebook was virtually non-incremental. At that point, no one had thought about it. No one had really explored what would happen. And I love that we were able to do that. And no one batted an eye. They were like, this is what the data says. Let’s go try it out. Ran it for 3 months. And then we turned it off and gave back Uber about $30 million a year.”

SUNDAR SWAMINATHAN, Episode 153

The correlation had looked strong, the spend in that case was netting a positive ROI because attribution said the channel was working. But when put to the test, reality said that it wasn’t actually working. The signups were coming anyway.

Why Marketing Attribution Data Can’t Tell AI Agents What Actually Caused the Result

I don’t want to spend this whole episode arguing about attribution but it has a big role to play here. 

My main beef with is when attribution gets promoted beyond its job. The point is not that multi-touch attribution is useless. The point here is that attribution has to know its job. Attribution can help a team find patterns worth investigating, but it cannot be promoted into causal proof.

Journey Data Can’t Show the Counterfactual

Nadia Davis made this distinction well in our episode on attribution data quality. She’s VP Marketing at CaliberMind. Her view is that attribution can still be useful as a filter, especially when you’re trying to separate meaningful activity from channel clutter. But it still depends on the discipline of the data underneath it.

”It’s a framework and it’s a tool to answer certain questions. You have to be responsible with the data that you have and responsible in a way where you understand what it means. You understand the limitations. You set those expectations with others who may be less data savvy. And if you don’t have that data stewardship or that data acumen and you don’t know how to help people see that, it doesn’t matter what framework you use, it will all fall apart eventually.”

NADIA DAVIS, Episode 184

She separates surface reporting from the analytical work that makes attribution worth running:

“Reporting is just whatever number you log into your tool. Analytics is where you dive deeper, you investigate and you kind of try to understand what’s working, what’s not working.”

That distinction matters once agents enter the chat. If the touchpoint was never logged in a way that is auditable, our model can’t reason about it. If the model only sees visible touches, the agent treats visibility as importance. 

Attribution can help you do a lot of things. It can decide where to look and spend experiment dollars. It can tell you the visible path to purchase. It can give you real time signals on what paid campaign has the most engagement. It can tell you what content shows up most frequently in buying journeys… but it cannot decide what actually caused the conversion.

Rajeev Nair, co-founder at Lifesight and a causal inference researcher, describes the same failure from the measurement side:

“Marketers love to throw around the word causal. It sounds sharp. Scientific. Confident. But most of what gets labeled causal in marketing is just dressed-up correlation. If you’re not randomizing, balancing groups, and removing bias, you’re just chasing noise in fancy charts.”

RAJEEV NAIR, Episode 176

Rajeev’s billboard example makes the structural problem concrete. Picture a physical store with a billboard directly outside. Every customer who walks in has seen that billboard. Attribution data shows correlation: billboard exposure precedes purchase. The causal question is different. Those customers were walking into the store regardless. The billboard didn’t bring them. The data records what happened next to what. It can’t record what would have happened without the intervention. Add in-store discounts and the trap deepens: customers already showing high intent redeem the offer, the discount gets credit for the conversion, and the agent learns to replicate the pattern.

“Your bottom of the funnel would always be highly correlated with your outcomes. Because in the whole journey, more people are engaging with the bottom of the funnel before they transact. If you have a billboard right in front of your store, every user that walks into your store sees this billboard. Can we say that this billboard is driving the walk-ins? If you are offering discounts and people are redeeming those discounts and making a purchase — they were already showing high intent by walking into your store. Is the discount incrementally driving more outcomes? These are very difficult problems to address from journey data alone.”

Attribution data without controlled testing can’t tell you what caused an outcome. An agent that reads attribution correlations as causal signals will confidently build the strategy that made last quarter look good, for the wrong reasons. The metric you actually care about quietly degrades while the proxy looks clean.

This is counterfeit truth at the causal layer: data that is technically accurate, systematically correlational, and causally useless, with an agent scaling it at speed before anyone realizes the error.

Least Wrong Is the Real Goal

There is a harder argument here that deserves a fair hearing.

Constantine Yurevich, CEO and co-founder at SegmentStream and a skeptic of the measurement industry’s most confident claims, argues that marketing may never have true causation at all.

“In marketing we  have only correlation. It’s not true that we have causation.”

CONSTANTINE YUREVICH, Episode 166

His reasoning: even the gold standard of A/B testing relies on a methodology to validate its results, and most of those methodologies are themselves correlational. You are using the broken thing to validate the broken thing. Medical research, with full laboratory control and randomization, still struggles to prove causation cleanly. Marketing operates in a world with infinite uncontrolled variables: competitive noise, market trends, seasonality, economic conditions, and psychological differences across millions of customers. The variables balanced in your last experiment were different from the variables present in the next one. The replication problem is structural.

Constantine’s practical response is to work with reality rather than fight it. Track correlations systematically. Build frameworks that identify which patterns are stable and which are volatile. Accept that “least wrong” is a better target than definitive, and build toward it.

That framing matters for what the causal memory layer actually is. Building it produces better evidence than correlation alone: a record of what changed because you acted, across controlled conditions, over enough time to separate signal from noise. The agent operating on correlational patterns will scale what the data supports, regardless of whether that pattern holds under the conditions you’re in now. “Least wrong” is a discipline. Agents need it built in.

How Bad Signals Masquerade as Evidence

There’s a couple other elements to this that are worth flagging. Let’s talk about fake traffic… and fake data. Sometimes the signal is polluted before the agent ever starts optimizing.

Fake Traffic Looks Like Intent

Jordan Resnick, Director, Marketing Operations at an AI-powered tax solution vendor and former Ops guy at Atlassian and Cheq, came on the show to talk about fake traffic and machine customers. His warning was that bots do not just visit pages anymore. They scroll, click, submit forms, and create behavioral trails that look like intent until you inspect the sequence.

“I tend to look for what’s not explainable in analytics. Are they moving through the site faster than a human buyer would? Are they clicking on everything in patterns that don’t make sense?”

JORDAN RESNICK, Episode 203

He puts a number on what that scale of bad input costs:

“If 20 to 40 percent of the internet is fake, then 20 to 40 percent of those clicks you’re paying for are never going to buy because they’re fake.”

That is correlation masquerade at the input layer. The dashboard says engagement. The warehouse records behavior. The agent sees intent. But the business gets synthetic demand dressed up as signal.

Proxy Metrics Can Lie

Sometimes fake data isn’t necessarily from fake users or fake bots… but it emerges in the form of fake proxy metrics. The masquerade is the one every lifecycle marketer has felt at least once: the proxy metric looks alive while the actual customer experience is broken.

Jeff Lee, Lifecycle Marketing Technical Lead at Calm, has a brutal lifecycle example of this. His team once accidentally sent roughly 19 million essentially blank emails. The campaign generated an insanely strong open rate because people were super confused and the tracking pixel was still loaded. By the dashboard’s first read, the campaign looks like it broke new records. By any human read, it was obviously broken.

“ I accidentally sent 19 million emails that were basically blank. And we got a massive open rate!”

JEFF LEE, Episode 158

That is the correlation masquerade in its simplest form. When the sensor lies, the agent does not need bad context to make a bad decision. It only needs permission to optimize the proxy.

Alone, this is a recoverable mistake. But Tobias calls it the boomerang effect for a reason: the intervention backfires causally, damages the customer relationship, and takes weeks to surface in your metrics. By the time you see it in the data, you’ve done it to millions of people. And agents don’t run one campaign at a time.

“Agentic can accelerate the things that are good. But it will also accelerate the things that are bad to be really plainspoken. And it doesn’t distinguish between the two if you don’t.”

TOBIAS KONITZER, Episode 212

The acceleration problem has a second layer: if the agent is scaling the wrong thing, how do you know?

The Auditability Problem

The boomerang is worse than it looks. Correlational mistakes at agent speed are hard to catch, and in a system built on correlations alone, there’s often no way to determine what logic produced the output.

Tobias talked about the difference between RL reinforcement learning-based decisioning systems and agents running directly on warehouse correlations. With RL-based decisioning, the optimization path is traceable. You can at least examine what the system learned, what reward signals shaped its behavior, and why a particular intervention was chosen for a particular customer state. There’s baked in auditability.

“The nice thing about decisioning systems, about reinforcement learning, is that I can audit them. I can look in there and I can find out: this is why we did this. What agentic AI does, particularly if you let it go wild on decisioning, you can’t audit in this way. And if anybody claims anything else, it’s a lie.”

TOBIAS KONITZER, Episode 212

An audit trail matters most at scale, when the campaign has reached millions of people and someone in legal asks why a particular customer received a particular message. Building the causal memory layer before agents run autonomous campaigns is what makes that question answerable.

BOSS BATTLE: The Correlation Boomerang Archer

The boss on this floor is the correlation boomerang archer: it’s a rebound system that reads a warehouse full of accurate historical data and produces the wrong answer at scale, confidently and efficiently, because nobody told it the difference between correlation and causation. Defeating it requires building a causal record of what actually changed because you acted.

*Before the boss fight: check your inventory for how to battle this one. 

[RARE SKILL] Causal Inference Protocol: Requires randomization, balanced groups, and bias removal. Without this equipped, you are reading noise and calling it signal. With it, you can distinguish what actually caused an outcome from what merely correlated with it.

[EPIC WEAPON] Controlled Experimentation: Holdout groups, matched-pair analysis, incrementality testing. The only framework that produces signal an agent can safely act on. Swings for causation every time instead of settling for the easiest pattern in the data.

[LEGENDARY] The Agent Context Graph: Snapshot every experiment (customer state, intervention, actual uplift, world conditions at the time). Build this record across every experiment that runs, indefinitely. The teams who started 2 years ago already have a temporal advantage you cannot buy off the shelf.

Let’s put the items to work.

Reducing Exposure While the Foundation Is Built

That causal memory layer is the destination everyone talks about. But it takes time: experiments that run, results that accumulate, patterns that separate from noise. That work starts now and it doesn’t finish this quarter. Most teams will spend some time in the gap; agents running, causal infrastructure still under construction. 3 practices narrow the exposure while the foundation catches up.

While that foundation is being built, 3 practices narrow the exposure immediately. 

1. Add holdout groups to every agent-driven campaign, a percentage of the target audience that receives no intervention, so you can measure what would have happened without it. 
2. Set a rule that no agent scales any behavior beyond 10 percent of the audience without a human reviewing the assumed causal logic first. 
3. Document what the agent is optimizing for before it runs. If you can’t write down the causal chain it assumes, the agent shouldn’t be running autonomously yet.

The holdout discipline is easier to build than most teams assume. Simon Heaton, Director of Growth at Buffer, runs holdout tests as standard practice across Buffer’s lifecycle programs:

“Experimentation is a cornerstone of our team’s approach and methodology. The holdout test is a really great functionality, whether you’re trying to launch a new program and just validate will having something make a difference against not having anything, or trying to compare how a particular campaign influences the broader user lifecycle. The way it’s set up, it works really nicely. Because we have a delivery event fired for the holdout group, it makes cohorting the 2 groups exceptionally easy in Mixpanel so you can track differences in behavior between both groups.”

SIMON HEATON, Episode 133

The holdout doesn’t need to produce a statistically definitive result on day one. Its first job is exposure management. It tells you what the baseline looks like without the agent’s intervention. That baseline is the beginning of a causal record.

Even a small, underpowered test catches the thing that actually matters: a catastrophic drop before it reaches scale. Sundar Swaminathan held to this discipline at a travel tech startup running fewer than 200 signups per week:

“I pushed for that AB test because I didn’t want to deal with the CEO after. Even if it’s a poorly powered AB test, I’m willing to take the risk with a hundred people because I’ll see if something massive drops. If the result is saying with 60 percent confidence this thing worked, you can roll it out anyway. But at least now I caught the catastrophic drop — which was 30 to 40 percent with 95 percent confidence. That’s the stuff I’m trying to protect against.”

SUNDAR SWAMINATHAN, Episode 153

The first holdout doesn’t need to be perfect. It needs to catch the boomerang before the agent has already sent the campaign to 5 million people.

Building a Causal Memory Layer With a Context Graph

Okay let’s talk about causal memory and how to build it. 

How the Agent Context Graph Works

Anthony Rotio, the freshly named Co-CEO at GrowthLoop and a former Harvard computer scientist who led US marketing at AB InBev, calls it the agent context graph. The idea: snapshot every experiment. Where was the customer, what was the intervention, what was the actual uplift, and what was happening in the world at the time. Do that with every experiment that runs. Build the temporal record over years, not just the current data snapshot.

“The way we are trying to make this a verifiable domain is through what we call causality data, or our agent context graph. You start snapshotting: this is everything we know about Phil right now. Here’s the intervention or message we tried with him today. Here’s the outcome, the actual uplift in whatever KPI we care about. And you do that constantly with every experiment that you run. You build this context graph over time. Not just a snapshot of the data we have today, but at every single point in time: where was Phil? What was the world doing? What did we try to send him? Did it work?”

ANTHONY ROTIO, Episode 208

Once you have that temporal record, you can ask counterfactual questions: what would happen if we sent a different message, in a different moment, to someone in this exact customer state. You can run the test against a simulated customer population before a single message goes out. That’s the shift from reading patterns to reasoning about causes.

The warehouse still matters. The data is still the foundation. But the 2 systems answer different questions.

	Data Warehouse	Agent Context Graph
Records	What happened	What changed because you acted
Supports	Pattern reading and historical analysis	Causal evidence and counterfactual reasoning
Drift detection	Waits for performance collapse	Measures against a continuously grounded baseline
Time scope	Current snapshot	Temporal record across every experiment

That distinction is the difference between an agent reading patterns and an agent reading evidence.

The context graph also functions as a drift detector. As the market changes, as customers change, as the world changes, the assumptions from past experiments stop being true. The context graph lets you measure against a continuously grounded baseline instead of assuming last year’s signal still holds. You don’t have to wait for performance to collapse to know your model has drifted. You can see it before it does damage.

The agent context graph inherits the problem it’s meant to solve if the data populating it came from correlational sources. Tobias’s churn-audience example applies here: a context graph built from historical correlations will tell an agent that your most engaged customers are the ones most worth retaining, because their records show the strongest pattern of non-churn. The graph captures what correlated in the past. Causal evidence requires experiments that measured what actually changed because you acted.

The right inputs for initializing the context graph are experimental results: tests with holdout groups, documented interventions, and measured actual uplift. These entries give the graph a causal foundation to reason from. The first round of experiments is the foundation-building step. Each one populates the context graph with data it can reason from causally.

How to Start Building a Causal Record

Okay this sounds fancy… and the reality is that in a lot of cases if you want to do it right, it is. There is no shortcut here. The context graph has to be built from actual experiments run over time. The earlier you start, the more causal history you accumulate. The teams who started 2 years ago have causal history you can’t retroactively create.

So here’s a start pack:

• Week 1 looks like this: pick 1 campaign or experiment that already ran and document it fully: the audience state at the time, the message, the timing, the KPI you were measuring, the actual result, and what else was happening in the world (seasonality, competitor activity, product changes). 
• That’s your first context graph entry. Do it for every experiment from this point forward. The graph builds through practice, but only if you start capturing now. The teams with a 2-year lead built it one experiment at a time.

Here’s Rajeev Nair again, describes the order that makes this process work across longer horizons. The first model doesn’t have to produce perfect recommendations. It has to forecast accurately enough to earn trust in it:

“When we create a first model, you can use the model to create a forecast. You tell the model what you have planned to invest for next month or next 2 months. The model creates a forecast. See if the forecast is accurate. If the model is able to forecast well on what you’ve already planned for, then the model has generalized well on your historical data. Once you have built enough confidence on the forecast, then go to the optimization part — which is asking the reverse: I need to reach a revenue target. Give me the mix.”

RAJEEV NAIR, Episode 176

Most teams skip this step. They ask the model to optimize before they’ve verified it can predict. Building confidence in the forecast first is the check that makes the optimization trustworthy.

MMM plays a second role in this process: hypothesis generation. The model identifies which variables it is least confident about. Those are the candidates worth running real experiments on. Rajeev:

“MMM is a very good hypothesis generator. If you have 10 factors for which you need to measure incrementality, you need not test all 10. First have a process in place where you can generate intelligent hypotheses, so that you are using your bandwidth the right way — running the right tests, the most important ones. And for everything else, you could have quasi experiments, which is where your model comes in.”

The causal record and the model serve different functions. One builds the evidence base experiment by experiment. The other identifies where the evidence is thin and a test is worth the cost.

At Canva, Matthew Castino, Marketing Measurement Science Lead, runs this at global scale. His framing for the relationship between experiments and models:

“Experiments are core. The model is the best possible thing when you do not have an experiment, but you need both feeding each other.”

MATTHEW CASTINO, Episode 200

Canva uses MMM outputs to flag where confidence is weakest, designs experiments to fill those gaps, and feeds the results back into the model before the environment shifts. The test confirms or challenges what the model assumes. The model tells you where the next test should run. When the 2 methods produce contradictory signals on the same channel, Matthew says Canva investigates rather than picks a side. Contradiction is evidence that a model assumption needs testing.

The loop only works if it stays continuous. Creative changes, budget pacing shifts, and competitive dynamics move. The test that ran 8 months ago captured a version of the market that may have already passed. 

The context graph forces that refresh: every new experiment is a check on whether yesterday’s causal evidence still holds in today’s market.

Anthony Rotio says AI amplifies the quality of your feedback loop, whatever that foundation is. Teams with causal memory running benefit from that. Teams running on warehouse correlations have their pattern-matching scaled instead.

“When I say the cleanest feedback loop, it’s like you have to be snapshotting this over time. The earlier you start, the better, to have this foundation to actually do things better than just shotgunning a bunch of emails and burning your frequency caps and getting your customers really angry.”

ANTHONY ROTIO, Episode 208

The teams who have been building causal history for 2 years have a compound advantage that accelerates under AI. The context graph is what makes the loop clean.

Achievement unlocked: Causal Evidence Layer Established.

*The correlation engine read the warehouse with perfect accuracy and still scaled the wrong thing. That’s the masquerade. The agent context graph in your inventory is what breaks it.*

The Correlation Masquerade is cleared. The next floor is harder.

Since agents start learning from that causal memory, the next question is obvious: who watches whether the memory still matches reality, and who controls the interface where the agents receive their instructions in the first place?

Step through. 🫡

Episode Recap

Part three of the four-part Dungeon of Martech Architecture series drops you onto the floor where every action comes back wrong. You’ve cleaned your data, built the warehouse, and turned AI loose on it. Everything looks like it’s working. It isn’t.

This is the floor where agents mistake correlation for causation and scale the mistake at machine speed. An agent sees that a discount campaign converted free users, so it blasts the discount everywhere, never realizing those users were going to pay anyway. Revenue erodes behind a dashboard that’s flashing green. Agentic AI accelerates the good and the bad, and it can’t tell the difference.

The episode stacks the receipts: Sundar Swaminathan handed Uber back $30M/year by proving Facebook was non-incremental. Simon Lejeune turns “but what’s the real impact?” into a hiring filter at Wealthsimple. Nadia Davis, Rajeev Nair, and Constantine Yurovich pick apart why attribution data can flag patterns but never prove cause, and why “least wrong” beats chasing a causation that marketing may never truly have.

Then comes the boss: the Correlation Boomerang Archer: a system that reads perfectly accurate history and confidently fires back the wrong answer at scale. The weapons to beat it: holdout groups on every agent campaign, a hard 10% cap before any human reviews the causal logic, and the centerpiece: the agent context graph. Anthony Rodio explains how snapshotting state, intervention, and outcome over time turns marketing into a “verifiable domain,” letting you simulate 100,000 emails to a digital twin and answer the what-if questions before you ever
hit send.

Clean the floor, unlock the causal evidence layer and brace for the final, hardest floor: who watches the memory, and who controls the interface the agents take orders from?

Full episode ⬇️ or Back to the top ⬆️

✌️

—
Cover art created with Midjourney (check out how)

Apple •Spotify• Pocket Casts •Youtube •Overcast •RSS

Related tags

See all episodes

Future-proofing the humans behind the tech

Apple •Pocket Casts•Google •Overcast •Spotify •Breaker •Castro •RSS

• LinkedIn
• Twitter
• Instagram
• YouTube
• RSS Feed