Before you build a product it's essential that you test whether or not people want it. Part 1 described how to do initial research. This post explains how you can test your hypothesis with real data (and ideally make some money doing it!). Read below to see the results from the test and what I learned.
Last week I wrote an article titled “How to Validate Demand for Your Product.” In it I described my process for evaluating business ideas. I explained how I use Google Keyword Tool, Reddit, and Twitter to see if an idea is worth pursuing. That was Part 1. The next step in the process of business model validation is to test your hypothesis with real data. Enter: Conversion Rate Testing.
As some of you realized (and pointed out on HackerNews), the blog post that I wrote was in itself a business model experiment (meta, right?). In writing that post I was hoping to get traffic to my new website: Up Up Grow, an online marketing course. I wanted to test conversion rates and see if anyone actually wanted the course. So I thought it’d make sense for Part 2 of this series to share the results and explain how I did that.
First things first, here were the results from the test (and my first week of live testing):
Unique visitors: 15,862
Visits to course landing page: 720
Course sign ups: 13
Conversion rate: 1.8%
Email newsletter sign ups (leads): 120
An Introduction to Pretotyping
To most it seems kinda weird to advertise a marketing course that doesn’t even exist. It almost seems Donald Trump University-ish. But I’d argue it’s weird for anyone to do otherwise.
One of the people I mentioned in my last post was a mentor named Alberto Savoia. He taught me everything I know about business model validation. When I was a 19 year old college dropout trying to find my way in Silicon Valley he took me under his wing. At Google, he coined the term “Pretotyping” which is the idea that before you build a product you should test demand for it.
As an early engineer at Google he discovered that many of Google’s products weren’t taking off. People had heard of Google Maps, Search, and Adwords. But some of their products just weren’t working. So Alberto decided to figure out why.
During his research he learned that the process most engineers took was Build, Ship, Test. They’d come up with a great idea, and then immediately start solving the difficult technical problems. Then the design team would polish it up, and send it to marketing. Then they’d drum up a bunch of interest and ship the product. After an initial spike in users many products quickly became obsolete. Alberto guessed that the reason was because no one had taken the time to test whether or not people wanted the product in the first place. And thus pretotyping was born.
Coincidentally Alberto began his work at roughly the same time that Eric Reis began writing The Lean Startup. While Reis’ book went mainstream, Alberto’s attracted a more cult following. But the concepts are more or less the same: don’t build a product before you know people want it.
To Alberto (and Reis), a business should reverse their product development process: rather than rush to build a product, they should first test demand for that product. Then using a simple landing page and some marketing dollars they should get real data to validate their hypothesis.
Applying Pretotyping to Up Up Grow
With Alberto’s words in the back of my mind I decided to test demand for my online marketing course. I knew going into it that there was a lot of competition in the space. But still, I felt like there was a gap. So many people teach students how to market a product with a big budget, or consultants. I wanted to launch a course that founders could use to reduce their customer acquisition cost (CAC) to less than $100.
With Up Up Grow I skipped the process that I outlined in Part 1. I’d already spoken with about 100 people about the idea so I knew that there was some demand out there. But I wanted to make sure that those people weren’t just encouraging me. Any time humans rely on qualitative data they risk getting biased feedback. When I ran the idea by people they said they’d buy it, but how’d I know whether or not they were just being nice? What I needed was unbiased data to tell me whether or not I should spend weeks designing a course.
Building a landing page
Last week I launched the website using Squarespace, Unsplash (for images), and The Noun Project (for icons). I setup a Stripe account and listed my course as a product on the site. All told it took me about 2 hours to get the site live.
The next step was to get traffic to the page in order to test conversion rates.
In order to get traffic I wrote a blog post (part 1) and posted it to HackerNews. From my previous company, SimpleData, I knew that a HackerNews post could get tens of thousands of site visits if it hit the front page. So I wrote the post with that audience in mind. I looked at the front page and analyzed the headlines that got the most upvotes. The goal was to reverse engineer what had already worked. A blog post describing how to validate demand for a product seemed to make sense.
I hit publish last Thursday and crossed my fingers. I was at my parents house eating dinner when my phone started buzzing. By the time I logged on later that evening the story was #2 on the front page. A couple minutes later I got an email from Stripe telling me that the first student had signed up.
Over the next day I also got a constant stream of emails from Mailchimp indicating that people had signed up for the newsletter. So that night I decided to email them and see if they had any feedback on the course.
Hey new Internet friends,
In the last couple days you signed up for my newsletter after reading this blog post on How to Validate Demand for Your Product.
I have to say HackerNews is a wonderful place (minus the comment trolls of course). I posted the article on Thursday night, and three days later 15,000 of you have read the story.
Like many of you, I'm trying to validate a new business model here and I want to soak up as much information as possible. So I have a quick question -- Is there anything stopping you from buying the course right now?
One of my hunches is that the pricing isn't right. Or maybe the course curriculum (what you'll learn) isn't clear. Or maybe that picture of me scared you away :)
For the first 10 people that respond I'll give 50% off the already discounted $95 price as a token of my appreciation.
Anywho, I'm going to get off my computer and enjoy the holiday now. In the meantime let me know if I can help you with anything growth related!
The email got a 78% open rate, and 12% of subscribers responded. The responses ended up being GOLD. One person responded to let me know that they were confused about when the course would begin. I emailed back details and they purchased an hour later. Someone else responded that they wanted more information on what would be taught. Again, I responded and they converted. In total 7 newsletter subscribers converted, which was further validation that I was onto something.
Crunching the data
Of course the most important metrics I looked at were revenue and conversion rate. With $900 of revenue generated from a single blog post I knew the business model would be viable. Last week I spoke with a top-notch blogger that charges $250 for data-driven stories. With that in mind, I validated that the business could scale without me.
The conversion rate is important because it gives me a sense of what kind of customer acquisition cost I can expect in the future. Here’s a simple example to make that concrete:
Let’s say my friend runs a blog. He approaches me and tells me that he is willing to show my course to his readers. But he wants to charge me $.10 per click. Should I advertise with him?
At $.10 per click, 100 clicks would cost me $10 (duh). And if the same conversion rate (1.8%) held, I could expect to sign up 1 or 2 students, which would net $95 to $190 in revenue. That’s a no brainer.
At $1 per click I’d have to think harder about it. I’d acquire 1-2 students for $100. If the conversion rate slipped I’d barely break even.
Regardless of the exact numbers, my conversion rate gives me a benchmark to start plugging in numbers. In further research into the market if I learn that industry CPC costs are $5 then I know that I’m in trouble. Conversely if I learn about marketing channels like my friend’s blog that offer $.10 CPC prices then I know I’m in business. At that point my only bottleneck is capital to invest in the channels.
Of course, conversion rates change based on the channel and audience, but with this blog post I was able to get a ballpark estimate. Going forward I’ll be able to continue testing and ideally raise the conversion rate by targeting audiences better and writing more content on the site.
After validating demand, the next step is to start growing revenue. In order to do that there are a couple metrics to manipulate:
- Traffic -- Theoretically if I increase traffic by 10x I should get ~100 sign ups.
- Conversion rate -- If I continue to get the same traffic I could also try to double my conversion rate. Again, that should theoretically double the students I sign up per day.
- Course price -- I could also test the price. This is a classic microeconomics optimization problem to solve: if I double the price do conversion drop by more or less than 50%. If it’s less than 50% I should theoretically double my price and keep it there since net revenue would grow as a result.
In the next blog posts I’ll discuss marketing channels that I’m testing and a few paid ad campaigns I plan to run. As always I’ll provide as many examples and real data as possible. Sign up for the newsletter if you’re interested in hearing when that is published.