I Analyzed 10,242 Website A/B Tests (Here’s What The Data Says) | Casey Hill, CMO, DoWhatWorks

Most website debates inside marketing teams come down to the same thing: one person's opinion versus another person's opinion, dressed up with enough confidence to sound like expertise.

Casey Hill does not operate that way.

As CMO of DoWhatWorks, Casey has access to something most marketers do not: a database built from thousands of real A/B tests run by the world's top brands, tracking not just what gets tested, but what actually gets kept. Companies like Adobe, Twilio, Stripe, and Glean are in there. The results are not always what you would expect.

Casey joined me on Marketing by Design to walk through what the data says, rank some real homepages live on air, and share the content philosophy that drives 30 to 40% of his company's closed enterprise business directly from LinkedIn. If you have ever argued about a website decision in a meeting without hard data to back it up, this episode is for you.

What DoWhatWorks Actually Does

Before getting into the findings, it helps to understand what Casey and his team have built.

DoWhatWorks tracks public A/B test data at scale. They look at what brands are testing across their websites, which versions those brands keep after the test concludes, and what variables seem to influence those outcomes. Company size, page type, industry, geography: it all gets factored in. Then that data gets surfaced to customers who are making design and content decisions and want something more defensible than internal opinion.

The practical use case Casey described hit close to home. Someone on his newsletter reached out and told him they sent one of his data-backed posts to their executive team to justify making a website change. That is exactly what this kind of data enables. It turns a design conversation into a business conversation.

Two A/B Test Results That Genuinely Surprised Him

I asked Casey what findings from the database had actually shocked him. Two came up immediately.

The first was logo bars. They are on nearly every SaaS homepage. They feel like table stakes. And according to the DoWhatWorks database, they lose in A/B testing at a very high rate. Casey said when he first saw that number, he did not believe it.

The second was video. Casey spent years at Bonjoro, a video email company, and came into this role genuinely bullish on video as a conversion tool. Then he looked at hundreds of tests comparing video against static imagery in hero sections. Imagery kept winning.

He was careful to add nuance here. Autoplaying GIF-style videos perform better than static videos that require a click. But the broader principle held: people do not always know what they are about to get when they click play, and on a hero section that uncertainty works against you. The expectation-to-reality gap is one of the core frameworks Casey keeps coming back to throughout the episode.

The Website Decision That Keeps Coming Up in the Data

One trend Casey has tracked across dozens of conversations with enterprise website teams is the rise of persona-based segmentation, and specifically, how brands are thinking about where that segmentation lives on the page.

Companies like MongoDB, Carta, Glean, and Sage have all made meaningful moves toward surfacing persona selectors higher on their homepages. MongoDB added an experience selector near the top of the page. Glean moved their segmentation section directly below the hero. Carta put their persona section right under the hero as well.

Casey dug into why job function kept outperforming industry and use case as a segmentation variable. The explanation he came back to was headspace. A VP of Product Marketing and a CFO both work at the same company, but they are coming to your website with completely different priorities. Segments built around role are more aligned with how people actually think about themselves and their problems.

His broader prediction is that this trend is pointing toward something bigger: websites that assemble themselves based on who is visiting. Not just a persona selector, but a fully curated experience based on real-time context. He used Lovable as a mental model. You tell it what you need, it asks clarifying questions, and it builds something specific to you. He thinks that is where enterprise websites are heading.

The S Through F Website Tier Ranking

One of the most useful parts of the episode is the live website ranking game we ran together. Casey walked through a handful of real homepages and pricing pages and gave his honest read on each one, grounded in test data where it existed and clearly flagged as personal opinion where it did not. Here are a few highlights.

Gumroad landed at C tier. The headline and subheadline gave a general sense of what the product does, but the differentiation was thin. The CTA language could be tightened to better set expectations before the click. And the visuals leaned more illustrative than product-focused, which tends to underperform in testing for B2B SaaS.

Strike-through pricing went straight to D or E tier. Casey's explanation was direct: it reads as salesy. The bigger the discount you're advertising, the more it signals to the buyer that the original price was not real. Trust takes the hit. The data across thousands of tests backs this up across both B2B and consumer brands.

PostHog earned A tier, which surprised some people in the room. Casey acknowledged that established marketing teams often critique PostHog's navigation as unintuitive. He agreed with parts of that critique. But his counterargument was distribution. PostHog is one of the most shared, linked-to, and talked-about websites in SaaS. The memorability creates its own momentum. He also called out their competitor comparison pages as some of the most honest he has seen, which builds trust in everything else on the site.

Headshot Pros closed out the game at D tier. Double banners, multiple eyebrows, competing visual elements, and social proof that raised more questions than it answered. Casey's core note was simple: if your header and subheader are doing their job, you do not need everything else fighting for attention at the same time.

Why Casey Barely Mentions His Own Company

The content strategy conversation was just as valuable as the website data, and it starts with a principle Casey has leaned into hard over the last year: own the category before you sell the product.

Almost all of Casey's LinkedIn content is about website best practices at large. He talks about what brands are doing, what the data shows, and what marketers should consider. He rarely mentions DoWhatWorks by name. The result is that he has become one of the most credible voices in the conversation, which means when people want to go deeper, they find the product on their own.

He also shared a specific tactic that I had not heard from any other CMO before this conversation. When he creates a post featuring a brand's website, he waits two to three days before reaching out to that company. He lets the post build momentum organically first. Then he contacts the team with something like: this has 40,000 views, just wanted to share it with you. The response rate from senior executives is close to 100%. Because the value is obvious and it is already proven.

Eight companies have since paid to sponsor his content after being featured. The CPM on those posts, because the algorithm already knows they perform, runs around five dollars compared to the eighty-dollar CPMs you might see on a bottom-of-funnel ad campaign. The distribution advantage compounds in both directions.

The Biggest Mistake in Social Proof

Casey's framework for evaluating social proof came from an unexpected place: a conversation with a biologist who studies cost signaling in animals. The harder something is to fake, the more credible it is as a signal. Elk antlers are hard to fake. A G2 badge from 2023 is not.

The practical applications he walked through were straightforward. Link your customer logos to actual case studies. Provide specific context around what a customer uses and how. Use video testimonials from named, searchable individuals at real companies. If you say you work with State Farm, make sure the viewer understands the scope and nature of that relationship, because without context, social proof can imply something different from what you intend.

His concern with badge walls and logo bars is not that they have zero value. It is that they are easy to fake, and buyers increasingly know it. When your social proof raises more questions than it answers, it is working against you.

What This Means for Your Website Right Now

The through-line across everything Casey shared is that most website decisions are made with more confidence than the evidence warrants. Teams argue about hero imagery, pricing page structure, and CTA language without access to what the data actually shows at scale.

The good news is that data is now accessible. Casey publishes it every Thursday in the DoWhatWorks newsletter. It is one of the most practically useful things I read as someone who builds websites for a living. If you want to start making more defensible decisions, that is the place to start.

You can also follow Casey directly on LinkedIn where he posts five to six times a week and consistently breaks down what real brands are doing and what the data says about it.

Listen to the Full Episode

This article covers the highlights, but the full conversation goes much deeper. We get into Casey's full career arc from sales intern to CMO, the GEO experiments he is running right now, and a live tier ranking of real homepages with Casey's unfiltered read on each one.

Find Marketing by Design wherever you listen to podcasts.

And if you want to see what high-converting website design actually looks like in practice, the Visual Swipe File is a free resource I put together for exactly that purpose: mmg.studio/visualswipefile.

‍