Metrics Showdown: Mixpanel v. KISSmetrics v. Google Analytics

NOTE: This post describes Mixpanel, KISSmetrics, and Google Analytics as I used them in Spring and Summer 2010. At least Mixpanel has changed substantially since then. I’m planning to do an updated post.

If you’re building a new web app, you’d rather focus on the app instead of building complex data collection tools. You need a metrics solution along the lines of Mixpanel, KISSmetrics, or Google Analytics. But which should you use and why?

I evaluated Mixpanel, KISSmetrics, and Google Analytics with an early prototype of Slipstream, a new web-based Twitter client. I had previously built my own tool for collecting and reviewing user data but thought that I could save time using an existing solution. I picked these three tools because they were the main metrics tools I had heard of. I used most of the client-side features of all three tools.

What are you trying to learn?

Collecting too much data is as bad as not collecting enough because you won’t be able to act on it all. Most metrics tools can generate more data than you know what to do with so you need to focus on getting what you need up front. Before picking a tool, you should answer the most important question:

What am I trying to learn?

The answer affects which tool you choose more than anything else. For example:

  • If you would like to justify your argument that a certain feature is being ignored, you can use Mixpanel’s event tracking to show that people aren’t using it, week after week.
  • If you want to find where in your sign up process people are ditching your site, KISSmetrics could be a great choice because of it’s excellent funnels.
  • If you need to show others that your social media efforts are more successful than any ad campaign you’ve tried, Google Analytics could give you a clue with it’s clear breakdown of traffic sources.

Once you know what you want to learn, it’s time to dive into the tools to see what will bring you closer to the answers.

Mixpanel pros & cons

Mixpanel is ideal for tracking arbitrary events in real-time. That means you can record when people do practically anything in your app. For example, when a user marks a tweet as a favorite in Slipstream, I can easily tell Mixpanel with a little bit of JavaScript:

mpmetrics.track("Added a favorite");

You can also add arbitrary properties to events. So if someone uses a particular feature like a slider that filters out tweets with a certain score, I can track the event (“Used slider”) and it’s property (“Tweets scored higher than”):

mpmetrics.track("Used slider", {
  "Tweets scored higher than" : slider_value
});

More impressively, the results can be viewed on Mixpanel.com in real-time. For example, I can see how often people are using my slider over the last few days:

An event in Mixpanel
and thanks to the “Tweets scored higher than” property, I can also see which values of the slider are the most popular:

An event with a property in Mixpanel.
Mixpanel is great for measuring retention, thanks to the following table. It helps me see which of my features, represented by each row as an event, are being used again and again over different periods of time:

Retention in Mixpanel.
Mixpanel also supports measuring funnels but I found it’s numbers were way off from what other tools recorded and, more importantly, the actual counts in my database. I don’t recommend using it.

This tool can’t measure and analyze traffic to your site, a chief strength of Google Analytics. Thankfully, the two can work together and the site and FAQs are helpful in explaining the differences between them.

Finally, Mixpanel has a somewhat cumbersome setup process. If you’re not familiar with metrics tools, it might take you some time to figure out where to integrate the Mixpanel code. There’s also no sandbox, so you’ll have to use up some data points against your monthly limit if you want to test that events are actually being tracked. Strangely, you can’t delete funnels or events so you’ll be stuck with test data. I recommend creating a test project when you’re setting up and trying to get everything working there. When you’re ready for production, create a new project and switch to it so your test data doesn’t get mixed in with your real data.

KISSmetrics pros & cons

KISSmetrics rocks at funnels. If you need to know how many visitors go from your landing page to pricing to sign up and how many drop out at each stage, it’s easy. You would first set up a few URL rules on the KISSmetrics site:

A KISSmetrics URL rule
or via an event API very similar to Mixpanel’s:

_kmq.push(['record', "Viewed main timeline"]);

and then chain them together into a nice visual report:

A KISSmetrics funnel report.
Most importantly, the KISSmetrics funnels are very accurate. They consistently matched the traffic I measured with Google Analytics and matched up with the numbers in my database.

While their funnels are great, I didn’t find KISSmetrics useful for anything else. Even though their JavaScript API is almost identical to the Mixpanel one, there is no reporting interface that supports measuring retention. And while you can track properties for events, again like Mixpanel, I found those much less useful in my funnels.

KISSmetrics makes it easier to get set up with a really helpful debugger that loads an arbitrary URL on your site and shows you all the data KISSmetrics is collecting. This is much easier than the hoops I had to jump through with testing my Mixpanel setup.

Google Analytics pros & cons

Google Analytics showers you with page by page data about your site. You can see how much traffic you’re getting at any given point in time, where it’s coming from, and what pages people are hitting most frequently:

Traffic over time in Google Analytics.
Traffic sources and top pages in Google Analytics.
Google Analytics can measure funnels, which it calls Goals. They’re noticeably less accurate than the KISSmetrics funnels but not as far off as Mixpanel’s.

This tool also has event tracking but I found it very cumbersome to set up and could not get it to work. The Mixpanel and KISSmetrics APIs were much more straightforward for arbitrary events and properties.

If you just use Google Analytics to measure your traffic, it’s dead simple to set up: just include some JavaScript once and never worry about it again.

My Decision

I didn’t see a clear winner after poking around with the three tools so I decided to try all of them together. I’m really glad I did because this is the comparison table (based on my experiences from above):

Mixpanel KISSmetrics Google Analytics
Tracking Events Great Poor Poor
Measuring Funnels Poor Great Average
Analyzing Traffic Poor Poor Great
Real-time Yes No No
Setup Difficult Moderate Easy

They’re all great at different things! And since what you need to learn will change over time, you really need more than one tool in the long term.

My Setup

I’ve come up with a pretty flexible setup so that I never have to worry about sending unnecessary data to any of the services.

The Google Analytics script is only included in production so I’m not counting all the times I hit my site when I’m developing it. In a Rails app, that looks like this:

<%- if RAILS_ENV == 'production' -%>
    <script type="text/javascript">
    // Google Analytics include script
    </script>
<%- end -%>

I stub out Mixpanel if I’m not in production:

<%- if RAILS_ENV == 'production' -%>
    <script type='text/javascript'>
    // regular Mixpanel include script
    </script>
<%- else -%>
    <script type='text/javascript'>
    var null_fn = function() {};
    var mpmetrics = {
        track: null_fn,
        track_funnel: null_fn,
        register: null_fn,
        register_once: null_fn,
        register_funnel: null_fn,
        identify: null_fn
    };
    </script>
<%- end -%>

The else clause lets me use the entire Mixpanel API anywhere I want in the rest of my app without generating any errors because it silently fails. This is a lot cleaner than having to check if I’m in the production environment for each call to the API.

KISSmetrics is also set up to silently fail in non-production environments:

<script type="text/javascript">
    var _kmq = _kmq || [];
    <%- if RAILS_ENV == 'production' -%>
    // regular KISSmetrics include function
    <%- end -%>
</script>

Both Mixpanel and KISSmetrics allow you identify a user with something unique like an email address, username, or id before recording other data:

mpmetrics.identify("<%= current_user.login %>");
_kmq.push(['identify', "<%= current_user.login %>"]);

This helps with accuracy so that the same visitor isn’t counted multiple times in a funnel or for an event (unless they actually went through the funnel or triggered the event multiple times). While this makes for better reporting, neither Mixpanel nor KISSmetrics let you see all the events any given user triggered or all the funnels he or she went through. You can approximate that functionality by setting an identifier property with each event. So instead of just reporting that the currently logged in user viewed the main timeline, I can specify who it was with a property and be able to filter the reports by that property:

mpmetrics.track("Viewed main timeline", {"User" : "<%= current_user.login %>" });
_kmq.push(['record', "Viewed main timeline", {"User" : "<%= current_user.login %>" }]);

This can be a bit cumbersome but per-user data is very useful because instead of retention in the abstract, you get to see exactly who is coming back. I hope both Mixpanel and KISSmetrics add this type of functionality in the future without me having to always set a “User” property.

Conclusion

Metrics tools help you learn a lot very quickly but you should focus on what you really need to learn before diving into any of them. Don’t be afraid to use multiple tools; their APIs are very similar and they’re usually good at very different things!

About these ads
  • Trackback are closed
  • Comments (16)
  1. If you have a Rails app, Josh Krall’s analytical can help you set up many analytics accounts including GA, KM, Clicky, and Hubspot (no MP yet): http://github.com/jkrall/analytical

  2. Thanks for a well written and useful piece.

    • gon
    • June 16th, 2010

    You should be using Rails.env.production? instead of RAILS_ENV == ‘production’

  3. Hey Arthur,

    Thanks for doing this post. Our funnels are definitely more complicated then they need to be. There’s a few reasons why and we think we made some right initial decisions.

    We’re planning an update very soon with them. =)

    Feel free to email me any suggestions directly, we’re always listening.

    Suhail

    • artvankilmer
    • June 18th, 2010

    @Suhail Doshi

    Thanks for having a great tool otherwise! The complexity of your funnels isn’t what didn’t work for me, they just didn’t seem to be recording the right outcomes when compared to KISSmetrics, goals in Google Analytics, or my database. I had a fairly generic sign up funnel but the number of people who signed up (according to the Mixpanel funnel) was much less than the number of accounts in my db.

  4. Thanks for the info, I still have a lot to learn when it comes to analytics!

  5. Great info, thanks for this! The only thing I disagree with is that I’ve had great luck with event tracking in Google Analytics and found it very easy to setup. Not sure why you couldn’t get it to work but I would try it again.

    • Amitabh Mallik
    • August 19th, 2010

    Hi, This is Amitabh, i respect your experience, but in my experience, i can see Kissmetrics is excellent then others 2 like google paralytics and funnel paralytics. Tracking result of Kissmetrics is excellent.

  6. Thanks for doing this post. I had a fairly generic sign up funnel but the number of people who signed up

  7. I would love to hear an update to this. Google has done a lot of updates with regards to flexibility of events, though I’d also make a claim you can fake page views as well to track more custom things. Additionally they’ve done more with actual funnels instead of just goals that lets them compete a bit more, though I’m unclear of the level of accuracy.

  8. This is very interesting, You are an excessively professional blogger. I have joined your feed and look ahead to looking for more of your great post. Also, I have shared your website in my social networks

  9. If you are doing an update, also check out Totango.com. Would be interested in your opinion. I think they are a bit different in that they don’t only monitor and do cohort and funnel analysis, but also integrate with your CRM and e-mail marketing/automation to help you improve on SaaS and online business customer engagement.

  10. There is a fine line between too much data and too little. I have a problem with too much and it becomes overwhelming. Thank you for this informative post!

  11. I totally agree Miranda. With Mixpanel I found it way too complicated – Google has done a lot in the last year to make their analytics more accurate and easy to add event tracking, etc. Even though the marketer inside of me wanted to see things as Mixpanel presented them.

    Start with Google and master tracking your visitors and the events that trigger when they come to your website. Use this to see what your visitors are clicking on to increase your conversion rates.

  12. Not sure if you are still planning to update this post, but as a mixpanel user, I think I find it not too hard. We use a combination of mixpanel and GA.

  13. I’d also recommend checking out Klaviyo at http://www.klaviyo.com for true people analytics (as a caveat, I’m one of the co-founders).

    Klaviyo focuses on user tracking and analysis – letting you quickly and easily know exactly which users have been logging in, which haven’t, and which haven’t found key features. We also let you tie this data directly to action by scheduling automatic emails directly from Klaviyo. For example, one of our clients setup and automatic email that emails all users who haven’t finished setup or logged back in three days after sign-up and offers them help from their support team.

    We’ve used all of the other solutions (and still use GA) and I’d agree they all have a time and place – it just depends what you’re trying to do. If you are trying to understand individual behaviors and take action on underperforming users, Klaviyo might fit well.

Comments are closed.
Follow

Get every new post delivered to your Inbox.