Metrics Showdown: Mixpanel v. KISSmetrics v. Google Analytics

NOTE: This post describes Mixpanel, KISSmetrics, and Google Analytics as I used them in Spring and Summer 2010. At least Mixpanel has changed substantially since then. I’m planning to do an updated post.

If you’re building a new web app, you’d rather focus on the app instead of building complex data collection tools. You need a metrics solution along the lines of Mixpanel, KISSmetrics, or Google Analytics. But which should you use and why?

I evaluated Mixpanel, KISSmetrics, and Google Analytics with an early prototype of Slipstream, a new web-based Twitter client. I had previously built my own tool for collecting and reviewing user data but thought that I could save time using an existing solution. I picked these three tools because they were the main metrics tools I had heard of. I used most of the client-side features of all three tools.

What are you trying to learn?

Collecting too much data is as bad as not collecting enough because you won’t be able to act on it all. Most metrics tools can generate more data than you know what to do with so you need to focus on getting what you need up front. Before picking a tool, you should answer the most important question:

What am I trying to learn?

The answer affects which tool you choose more than anything else. For example:

• If you would like to justify your argument that a certain feature is being ignored, you can use Mixpanel’s event tracking to show that people aren’t using it, week after week.
• If you want to find where in your sign up process people are ditching your site, KISSmetrics could be a great choice because of it’s excellent funnels.
• If you need to show others that your social media efforts are more successful than any ad campaign you’ve tried, Google Analytics could give you a clue with it’s clear breakdown of traffic sources.

Once you know what you want to learn, it’s time to dive into the tools to see what will bring you closer to the answers.

Mixpanel pros & cons

Mixpanel is ideal for tracking arbitrary events in real-time. That means you can record when people do practically anything in your app. For example, when a user marks a tweet as a favorite in Slipstream, I can easily tell Mixpanel with a little bit of JavaScript:

mpmetrics.track("Added a favorite");

You can also add arbitrary properties to events. So if someone uses a particular feature like a slider that filters out tweets with a certain score, I can track the event (“Used slider”) and it’s property (“Tweets scored higher than”):

mpmetrics.track("Used slider", {
  "Tweets scored higher than" : slider_value
});

More impressively, the results can be viewed on Mixpanel.com in real-time. For example, I can see how often people are using my slider over the last few days:

and thanks to the “Tweets scored higher than” property, I can also see which values of the slider are the most popular:

Mixpanel is great for measuring retention, thanks to the following table. It helps me see which of my features, represented by each row as an event, are being used again and again over different periods of time:

Mixpanel also supports measuring funnels but I found it’s numbers were way off from what other tools recorded and, more importantly, the actual counts in my database. I don’t recommend using it.

This tool can’t measure and analyze traffic to your site, a chief strength of Google Analytics. Thankfully, the two can work together and the site and FAQs are helpful in explaining the differences between them.

Finally, Mixpanel has a somewhat cumbersome setup process. If you’re not familiar with metrics tools, it might take you some time to figure out where to integrate the Mixpanel code. There’s also no sandbox, so you’ll have to use up some data points against your monthly limit if you want to test that events are actually being tracked. Strangely, you can’t delete funnels or events so you’ll be stuck with test data. I recommend creating a test project when you’re setting up and trying to get everything working there. When you’re ready for production, create a new project and switch to it so your test data doesn’t get mixed in with your real data.

KISSmetrics pros & cons

KISSmetrics rocks at funnels. If you need to know how many visitors go from your landing page to pricing to sign up and how many drop out at each stage, it’s easy. You would first set up a few URL rules on the KISSmetrics site:

or via an event API very similar to Mixpanel’s:

_kmq.push(['record', "Viewed main timeline"]);

and then chain them together into a nice visual report:

Most importantly, the KISSmetrics funnels are very accurate. They consistently matched the traffic I measured with Google Analytics and matched up with the numbers in my database.

While their funnels are great, I didn’t find KISSmetrics useful for anything else. Even though their JavaScript API is almost identical to the Mixpanel one, there is no reporting interface that supports measuring retention. And while you can track properties for events, again like Mixpanel, I found those much less useful in my funnels.

KISSmetrics makes it easier to get set up with a really helpful debugger that loads an arbitrary URL on your site and shows you all the data KISSmetrics is collecting. This is much easier than the hoops I had to jump through with testing my Mixpanel setup.

Google Analytics pros & cons

Google Analytics showers you with page by page data about your site. You can see how much traffic you’re getting at any given point in time, where it’s coming from, and what pages people are hitting most frequently:

Google Analytics can measure funnels, which it calls Goals. They’re noticeably less accurate than the KISSmetrics funnels but not as far off as Mixpanel’s.

This tool also has event tracking but I found it very cumbersome to set up and could not get it to work. The Mixpanel and KISSmetrics APIs were much more straightforward for arbitrary events and properties.

If you just use Google Analytics to measure your traffic, it’s dead simple to set up: just include some JavaScript once and never worry about it again.

My Decision

I didn’t see a clear winner after poking around with the three tools so I decided to try all of them together. I’m really glad I did because this is the comparison table (based on my experiences from above):

	Mixpanel	KISSmetrics	Google Analytics
Tracking Events	Great	Poor	Poor
Measuring Funnels	Poor	Great	Average
Analyzing Traffic	Poor	Poor	Great
Real-time	Yes	No	No
Setup	Difficult	Moderate	Easy

They’re all great at different things! And since what you need to learn will change over time, you really need more than one tool in the long term.

My Setup

I’ve come up with a pretty flexible setup so that I never have to worry about sending unnecessary data to any of the services.

The Google Analytics script is only included in production so I’m not counting all the times I hit my site when I’m developing it. In a Rails app, that looks like this:

<%- if RAILS_ENV == 'production' -%>
    <script type="text/javascript">
    // Google Analytics include script
    </script>
<%- end -%>

I stub out Mixpanel if I’m not in production:

<%- if RAILS_ENV == 'production' -%>
    <script type='text/javascript'>
    // regular Mixpanel include script
    </script>
<%- else -%>
    <script type='text/javascript'>
    var null_fn = function() {};
    var mpmetrics = {
        track: null_fn,
        track_funnel: null_fn,
        register: null_fn,
        register_once: null_fn,
        register_funnel: null_fn,
        identify: null_fn
    };
    </script>
<%- end -%>

The else clause lets me use the entire Mixpanel API anywhere I want in the rest of my app without generating any errors because it silently fails. This is a lot cleaner than having to check if I’m in the production environment for each call to the API.

KISSmetrics is also set up to silently fail in non-production environments:

<script type="text/javascript">
    var _kmq = _kmq || [];
    <%- if RAILS_ENV == 'production' -%>
    // regular KISSmetrics include function
    <%- end -%>
</script>

Both Mixpanel and KISSmetrics allow you identify a user with something unique like an email address, username, or id before recording other data:

mpmetrics.identify("<%= current_user.login %>");
_kmq.push(['identify', "<%= current_user.login %>"]);

This helps with accuracy so that the same visitor isn’t counted multiple times in a funnel or for an event (unless they actually went through the funnel or triggered the event multiple times). While this makes for better reporting, neither Mixpanel nor KISSmetrics let you see all the events any given user triggered or all the funnels he or she went through. You can approximate that functionality by setting an identifier property with each event. So instead of just reporting that the currently logged in user viewed the main timeline, I can specify who it was with a property and be able to filter the reports by that property:

mpmetrics.track("Viewed main timeline", {"User" : "<%= current_user.login %>" });
_kmq.push(['record', "Viewed main timeline", {"User" : "<%= current_user.login %>" }]);

This can be a bit cumbersome but per-user data is very useful because instead of retention in the abstract, you get to see exactly who is coming back. I hope both Mixpanel and KISSmetrics add this type of functionality in the future without me having to always set a “User” property.

Conclusion

Metrics tools help you learn a lot very quickly but you should focus on what you really need to learn before diving into any of them. Don’t be afraid to use multiple tools; their APIs are very similar and they’re usually good at very different things!

2010

06/15

CATEGORY

tools

Write comment