Endeavor is pleased to make public the following transcript from a presentation at the 2011 Endeavor Entrepreneur Summit in San Francisco. The event, which assembled over 450 entrepreneurs and global business leaders, featured dozens of entrepreneurship-related presentations by top CEOs and industry experts.
Overview: Hiten Shah, CEO at KISSmetrics, discusses the state of data analytics today and the challenges ahead, and offers some advice on how to make good data-driven decisions.
Bio: Hiten started on the Internet by founding a Internet marketing consultancy, ACS. He then went on to create Crazy Egg, an analytics tool that visualizes the user experience on a website. Now with KISSmetrics he is building a data driven solution for to help online businesses make better business decisions. Hiten is passionate about helping other entrepreneurs and startup people.
Let me begin by giving some background on myself and how I think about metrics. I went to UC Berkeley undergrad and prior to that I only ever had one job which was a summer internship at a medical devices company in Orange County. From early on, I realized that I wanted to start my own company versus working for someone else.
Actually, it started when I was six years old. My dad said then that you shouldn’t work for anyone. He is a physician; he goes to work, he makes money for every hour that he works, and he says if you are an entrepreneur it’s not like that. You can scale your time better. Yes, it’s harder, but it’s usually a better life, and the funny thing he said is you usually use your brain more.
So when I got out of college, my current co-founder was still in high school and he had one customer paying $3,500 a month for SEO. He was very good at it. He knows everyone in the industry. Even before high school, he was trying to build his own website. So he basically got in touch with all the SEO experts at the time. That was when it wasn’t as crowded and there were a lot of good people, and bad people. He hired a lot of them. He’s very entrepreneurial. I actually stayed in college 5 and a half years and made some money while I was there so I decided to convince him that we should start a company. I didn’t know what I was going to do except that I couldn’t do anything like work for someone else.
So we started the company and then literally from the first year of starting the company we realized that consulting is great. You can make money, but it’s just like working for someone else except you are working for many someone elses. So what we decided to do, and probably me more than him (he’s very sales and marketing focused, I’m a lot more product and engineering and team and company focused because I studied organizational behavior in college) is we were going to spend as much money of the money as we could. We both still lived at home — I even got married and lived at home for two years just because I could pour more money into trying things — but we poured all our money into trying things on the internet. Prior to that I didn’t know much about the internet except that I could type 100 words a minute and I could download stuff on Kazaa like movies and music and stuff. So we needed to learn way more about the internet and how to build products. These were things like front-end technologies, back-end technologies, and design.
What we did was we hired 70 to 100 different agencies or individuals or development shops. We basically learned by spending money and trying different things. We tried building 20 different products and most of them failed very horribly and very painfully. My most famous one, for me, is a hosting company that we spent a million dollars trying to build, and it was all our own money, there wasn’t any venture capital or anything like that. We also started getting into social media. We were probably one of the first agencies to get into that. We had at one time 7 or 8 of the top 10 Digg users on payroll, not because we were doing anything shady, but because they understood what type of content got up on those social media sites and we needed to get their knowledge for our customers.
Basically the first product that worked for us – “worked” meaning it started making money – was a SaaS product called CrazyEgg. What it does is you put a Java script on a site and creates heat maps for where people are clicking on a page. So that’s how I got into analytics—a little accidentally because that was the first product that worked for us. Prior to that we were always using analytics tools because we were basically a marketing agency at the end of the day. As a result, there were a lot of pains and it wasn’t very visual and a lot of business owners couldn’t understand the actual data. So while we were working on that we tried to raise money because at that time people weren’t funding little toys. Now they fund a lot of little toys, but there’s a lot of reasons why. Probably because there is more of a market in a lot of the software as a service, really low touch, low price point stuff can scale faster now. That company was quite successful, but it’s a small business and it still runs.
While we were building that we came up with the idea for KISSmetrics. The thesis behind KISSmetrics has always been: what’s the biggest problem with analytics that the current companies aren’t solving? The question that we wanted to answer was why are people building analytics tools in-house even though all these other companies exist. This still isn’t a solved problem. We’re still not solving it enough today. Everyone keeps building these tools in-house to track data.
There are two things that we discovered, and this is where it gets into the metrics. One is still very much early adopter; the other one is pretty established now in terms of the concept. So the reason people are building analytics in-house without them knowing it or even us knowing it is that when you build analytics or track data in-house—i.e., on your own back end in your own servers without using a vendor—you are actually able to tie that data to each individual user or individual person. When you look at Google Analytics or any of those companies, they don’t let you do that. Even if they do let you do that, they don’t let you do it easily. So the thing was that a lot of people were like “Okay, I’ve had ten visits to my site; what I really want to know is that I have 2 people visit 5 times each.” So that’s the whole people problem. Once we started digging in, we realized that’s the problem. If we build a tool that tracks people and the data for people, we win. And it’s a very hard problem.
The reason it’s a hard problem is that if you have 10 million customers, my challenge is to let you see data on one customer and all their data. And in house, that’s a big chore too. So companies like Groupon that are hitting north of a 100 million customers sometime soon, they have their own big data warehouse infrastructure. They’ve used us for a while for a bunch of stuff, but they built their infrastructure in the same way that we’ve built ours. Meaning there is always a person, there’s events, which are things people do, and there’s properties, which are things about people. But you can’t pass data today into KISSmetrics without tying it to a person. And then some of the more advanced things people do with tracking data is when someone’s anonymous, they give him an anonymous cookie or an anonymous ID in the record, and they track all their activity even if it happens for a year, before they actually convert and become an identified customer. Identified means they have an email address or user ID that you’re able to attach to all that anonymous activity. Our system does all that automatically. Those are all the things we discovered are big problems. Problem is, they are really big problems on multiple fronts. One is people don’t understand Google Analytics can’t do this for you. People don’t understand that this is even a problem because they are just dealing with the data that they get from Google Analytics or whatever. The savviest people are just cobbling it together.
I was just talking to one of the smartest people I know and he was saying that even my tool isn’t good enough for him because he can’t really get at the data he wants, but he knows it’s there. And he’s cobbled together things. He is using Google Analytics, his CRM, two other systems, an in-house system, and he dumps it into Excel and that’s how he does his reporting. This is a company that is like 40 or 50 people now, probably doing about $300 or $400k a month and growing really fast every month and he’s got this cobbled-together system and it’s working for him. So our dream is to make it so that he doesn’t have to do that. We’re still a little ways away from that probably, but it’s getting closer.
So that one problem is that all your data needs to be tied to people. With social media and all that stuff, it becomes an even wider and exponential problem. Because now you can identify the person long before they are even a customer. So how do you go and take that data and tie it back to the person once they’ve converted and understand how many implicit touch points they had with your company or you before they even actually became a customer? These are all advanced problems that we’ll probably start seeing discovered in the next couple of years. I think fundamentally if you are not tying data to people, you are not able to get good intelligence on your data. That’s the one key thing that I would say. It’s like I said before: it’s harder to know how many people did something than how many times something was done. So the number of times people purchase is interesting, but what’s more interesting and actual is the number of people that purchase. Even e-commerce, even real world, all this stuff matters, because then you can have insights on where these people that are repeatedly purchasing come from. Those people are more valuable to you. And it’s answers to questions like that that you’re really trying to get.
Qualitative and Quantitative
People think that data is quantitative, but what we’ve learned by our business is that qualitative data is just as important as quantitative data, sometimes even more important.
Say you want to improve your product. You already have current customers. One thing is to look at the data and things you are doing on your current customers. Another thing is to go to them and talk to them. There’s a book that’s a cheat sheet to customer development that’s called custdev.com. I encourage you to at least read that book. It’s all about this kind of interview and qualitative process and talking to customers.
The key point is that when you talk to customers, when you do market research, you are trying to ask them things like if I built a better car for you and it did this, would you buy it. That’s the typical kind of thing market research says. This process is not like that. It’s not about will you buy what I make. It’s more about what are you doing today to solve this problem.
So in analytics, I actually gave you guys the whole story of my business. If we were focused on just trying to build something that only we wanted to build — if we were going to do social analytics, and we weren’t paying attention to what the real needs were — we would have a relatively small opportunity and small business on our hands. Once we realized that what we were doing was trying to figure out what the real deep pains are for people and trying to solve those, this whole customer development process helped us understand what those are. So whether you are social analytics, mobile analytics, online SaaS, or even an offline store, what matters to you is your customer. If we weren’t trying to pay attention, we wouldn’t have built a product that’s much broader and much wider, with bigger opportunity. I’m a big believer in this and just don’t bias people that you are talking to when you are trying to figure out what their pains are. Don’t bias them by telling them what your solution is, trying to sell them on your solution. This is not a sales thing, this is a discovery thing.
So take the example of wanting to build some products for your customers based on this idea that you have. There a few ways I might go about that. One way would be, I would go talk to these customers and just go find out what they are using the product for today. I wouldn’t even try to tell them “this is what it does.” Sometimes they even come to you.
The other thing I love to do is when you have customers and you actually have their data, show them something on their data. And they’ll come back and they’ll tell you more about it. That’s the qualitative piece.
The last piece — and this is way newer and this is something we are discovering ourselves – follows a concept called OKRs (objectives and key results). This was invented at Intel by John Doerr and his crew. It’s followed and practiced at Google. I believe Zynga has some form of it and Facebook has a mutation of it which comes from business school which is the idea of MBOs (management by objectives). What we’ve realized is that the most successful, the most data-driven companies are following some form of this. What that means is that currently this process goes all the way from executive down to highest level managers where all of those people, everyone on the team has an objective and there are key results tied to objectives. Key results are just metrics with goals.
What we are doing at KISSMetrics is going to be able to create an objective. Your objective could be to increase revenue. Your metrics tied to that objective if you are a SaaS business would be your sign-up conversion rate. Because when people sign up they are already in a trial; they are ready to pay, your revenue per day. So there is someone on the team, maybe multiple people on the team, looking at that objective right now, have a goal of when they need to improve it, know how they need improve it.
A lot of you mentioned this idea of “I have this data, I want to analyze it.” So that’s one problem. And that problem will get solved, but the real problem is “I have this data, how do I get it to everyone in the company, how do I let them see what matters to them and what they should focus on…”
You need to analyze what’s been done to understand how to improve it, but if you don’t know what you are trying to improve, you run into a lot of problems. The idea is to add workflow around your data. Not a lot of companies are implementing this. The objective on management team is X. Then what you do is you basically create all the ABCs of all those objectives that tie to the big one as you go down the company. Our goal is to go below even that management level, to go all the way down to the janitor if we can and make them have an objective.
Focusing on the Right Data
So here’s a question: how do you make sure you’re measuring the right things?
This is a question you should be asking about any metric you are looking at. How is it helping you make decisions? Today. Not even a year from now. Let’s go through an example. Let’s say it’s an online application. People sign up online, they sign up for a trial, then they upgrade and then they are an active user. What I usually start with is what are your assumptions. So the assumptions here are people visit the site, they sign up, they activate. Activate means they have had a happy experience. So if you are a project management tool, activation would be you have created your first project and invited one of your team members into the project. That’s an active customer. You want to track that and then you track that when people actually upgrade. Instead of drowning in balance rates and pages per visit and all that stuff, all you really need is to figure what are the key points that are basically making them pay. If they don’t go through these, they are not going to pay. I’d like to think of those.
After we figure out what those key actions are, we hypothesize what this business should look like or could look like. You get some hypotheses right off the bat, but you are not tracking anything yet. You are just in Excel playing with these numbers and seeing what is a viable business. A lot of times an additional thing you would do is figure out how much each additional visit to your website would cost you if you would buy traffic. At the end of the day you need to buy traffic. So the idea then would be baseline. Baseline means you are getting the actual data.
Some of you have asked about real-time analytics. I think it depends on your business. If you are a social game or even a company like Groupon and you run a test in five minutes 100,000 people are going to come into the site, you’ll get a significant result really fast. But how many companies actually have that type of volume today? Not that many. If you have that type of volume and you are able to make decisions at that speed of real-time, then sure, real-time data is great. We actually have a real-time view in our product, but when you tie all data to customers, you can’t do that in real time. Because if I have data in the system for a customer for the last year and all of the sudden in the last one to two hours, there is new data on these 100,000 customers, that takes about two to four hours to update. You can’t do real-time and personal level data. Personal level meaning this person first came from a year ago and being able to tie that to real time. It’s impossible, not impossible from an infrastructure point of view, it’s just really costly. No one is doing that today like that.
So I’d say if you are high volume and you need to do the testing, real time works. That being said, if you do a deploy and you have these key metrics that you are looking at and somehow those key metrics are off, there should be something telling you that something went wrong. That needs to happen in real time too. So I would not tell you that real time is useless, I would just say you want to know why you want to use it. Otherwise, it’s awesome. It’s cool to see people in real time.