TESTING - Almost getting fired

Fyxer canceled after we tested 26 variants across 873,641 emails in two weeks. Before the test, we needed roughly 4,753 sends for a positive response. During it, we needed roughly 7,300.

I knew testing a bunch of unproven ideas could make the average worse while we found the ones worth keeping. I hadn't explained that properly, and the client was looking at worse performance.

Luckily, they gave me a chance to pick the winner and scale the F out of it.

There were strong winners inside the test. We concentrated on them, and I offered to front the costs during the notice period under a performance arrangement. They canceled the cancellation.

Gamma Prime is another example of why I look inside the average. In their first month, we sent about 58,000 emails, tested ten variants across three audiences, and got 53 positive responses. The average was roughly one lead per 1,094 sends.

Variant 3 needed 299 emails for a lead. Variant 1 needed 3,230. Another variant got 29 responses, and all of them were bots. Keeping all of those running equally would've meant paying to send a lot more of the weak copy.

Once we've got a winner, I like to give it 70% of the volume, use 20% to test changes to it, and keep 10% for new ideas.

For the hypothetical agency, that could look like this:

If nothing is producing useful outcomes yet, I don't have a winner to give 70% to. I'd keep looking for an offer and audience that work together. And if I only had 10,000 emails to send, I wouldn't split them across 30 little tests and expect a few meetings to settle which was best.

Gamma's strongest variant works out to about 334 positive leads per 100,000 sends. That gives us a reason to send more of it, but we still have to see whether the rate holds as we reach more people and whether those leads become customers.

I'd agree on the budget and the result we're trying to get before starting. We'd also agree when we'll make the next decision, so a test doesn't turn into paying for another month because nobody wants to say it isn't working.

Another story is us sending around 800,000 emails over four months for Eric Partaker and getting nothing.

Reply rates were twice our benchmark, but none of those conversations turned into paying customers. I told him I didn't think the channel worked for his business.

He disagreed, so we kept running the campaigns for free for a final push. It still wasn't the right channel. I didn't want him leaving with a maybe and spending more money trying to get the same answer.