Aaron Larner is a Madison entrepreneur and CTO of The Art Commission. Aaron shares his experience A/B testing his database version tool DevJoist via Hacker News.
This past Monday I shared DevJoist on Hacker News. I posted around 10:30am CT and quickly reached the front page, hovering around spot #8 until about noon. I've summarized some of the data that I collected while DevJoist was on the front page.
The Tagline Test
Before submitting to HN I had set up an A/B test to determine how well different taglines converted to registrations. Here are the conversion results by tagline:
| Tagline | Visits | Pages / Visit | Visit Duration | Bounce Rate | Registrations | Conversion Rate |
|---|---|---|---|---|---|---|
| Collaborative Schema Editing | 379 | 1.74 | 0:01:15 | 65.17% | 1 | 0.0026 |
| Database Schema Versioning | 369 | 1.92 | 0:01:16 | 60.16% | 2 | 0.0054 |
| Database Versioning Done Right | 314 | 1.82 | 0:01:27 | 64.33% | 1 | 0.0032 |
| Version Control For Your Database | 397 | 1.76 | 0:01:22 | 65.99% | 3 | 0.0076 |
| Version Control For Your Schema | 313 | 1.76 | 0:01:27 | 59.11% | 2 | 0.0064 |
Version Control For Your Database and Version Control For Your Schema were the winners in terms of registrations. It's also interesting to look at the time it took people to register based on which tagline they saw:
| Tagline | Avg. Time to Register |
|---|---|
| Collaborative Schema Editing | 1.91 minutes |
| Database Schema Versioning | 1.19 minutes |
| Database Versioning Done Right | 2.88 minutes |
| Version Control For Your Database | 0.94 minutes |
| Version Control For Your Schema | 1.87 minutes |
Within the app, I added a little product tour. The tour consisted of small popups that highlighted the functionality of different elements within the app. Each column in the table below represents one of the tour popups. They were presented to users sequentially starting with "Save a version."
The numbers in the table below represent unique views of each tour popup
| Tagline | Start a project | Save a version | Projects dropdown | Diff and merge | Collaborate with other developers | Don't forget to register |
|---|---|---|---|---|---|---|
| Collaborative Schema Editing | 22 | 6 | 5 | 4 | 3 | 3 |
| Database Schema Versioning | 23 | 0 | 0 | 0 | 0 | 0 |
| Database Versioning Done Right | 15 | 6 | 4 | 4 | 4 | 4 |
| Version Control For Your Database | 27 | 0 | 0 | 0 | 0 | 0 |
| Version Control For Your Schema | 23 | 4 | 3 | 3 | 3 | 3 |
I thought it was interesting to see that almost everyone closed the first tour popup immediately, but the people who continued after the first popup were very likely to continue all the way to the end. 3 of the visitors who continued past the first popup registered. That's a 18.75% conversion rate, which is very similar to the 18.33% conversion rate of users who clicked the call to action button. In the end, it looks like the product tour has little to no effect on final registrations.
How To Run Your Own Tests Using Google Analytics
Here's how I ran my tests. For each new visitor I generated a random integer between 1 and 5 to determine which tagline that user would see. I then stored that tagline in a cookie on the users machine, so that they wouldn't see other taglines if they refreshed the page. When I rendered the page I insterted the appropriate tagline for the user and set a custom variable in google analytics (with their javascript API). I also sprinkled GAN event API calls throughout my javascript to keep track of a slew of other information, such as:
- When users registered
- Which tour popup boxes they saw
- etc.
With a mixture of advanced segments and custom reports within Google Analytics, I was able to come up with all of the data that you see above. See the full blog post on the DevJoist website for more implementation details here.
Keep up to date other DevJoist news. Follow me on twitter: @devjoist