It’s not about the outputs, it’s about the outcomes
Warren Buffet, the legendary billionaire investor put it rather succinctly when he said that “Accounting is the language of business” – it’s the numbers that shows how a business is performing and it’s the numbers that shareholders look at to understand how their investment is faring. It is also true that it is the numbers that our business leaders look at when considering their intranet return on investment.
One business leader in my organisation repeated the mantra that the “facts are friendly“, building on Buffet’s observation, but when it comes to intranet performance, the numbers can be misleading and in the wrong hands, be beautifully manipulated. It’s vital that we do measure intranet performance, but we should put energy into measuring the right things.
Lies, Damned Lies and Statistics
Here are some common examples of intranet statistics that I’ve quoted (and have been quoted at me) together with potential interpretations.
They say: “Look! Employees love my content, look at the page views per visit”
Translation: Employees clearly can’t find what they want
They say: “Our content is compelling, look at the average time per visit!”
Translation: They can’t make head nor tail of it. It takes them forever to figure out where to find it and what it all means
They say: “The search is so powerful. Our employees always use it”
Translation: Your navigation stinks
They say: “On average, 90% visit the intranet every month!”
Translation: 90% visit just once to collect their payslip. 10% of employees can’t even be bothered to do that.
Intranet outputs such as these are interesting enough, but will never tell the full story. Ask yourself a very important question for any of the stats that you routinely present to your leadership — ask, “so what?”.
The ‘So What’ test of relevance
If an article on your intranet gets high readership levels, why is that important? So what if 90% of your employees opened the article. Did they read it all, did they read any? Did they do anything with the information?
The inference is that if an article is well read, that your employees have absorbed and understood the information presented. The tempting leap of faith is to assume that employees have somehow done something with the information within, maybe changed a behaviour or realised a new business efficiency. Of course, sadly, however tempting, you shouldn’t jump to those conclusions.
For every statistic you publish, apply the ‘so what’ challenge. Why is that number of any value? What does it actually mean? Does it allow your leadership to understand the power of the intranet and their investment, or does it just serve to mislead and invite misinterpretations.
It’s not about the outputs, it’s about the outcomes
Outputs such as page views, or time per visit, are interesting numbers but don’t get to the heart of the matter. It’s the outcomes, or the results of intranet activity, that’s most important. When your internal communications team publish an article, the number of views (an output) is not nearly as interesting as the number of employees who, when asked, could recall the article content four weeks hence, or statistical demonstrations of how that communication changed behaviour (both outcomes).
These numbers are much harder to generate and report, but do provide better clarity of return on investment and of purpose of your intranet. They’re also not as susceptible to misinterpretation as more traditional intranet web statistics.
How do you measure your intranet performance?
Thanks for this thought provoking article Jonathan but I think page views alone are a valuable and useful measure of success.
For example, I would be almost willing to bet that you track how many page views this blog article gets and that the more views you get the more successful you consider the article.
Perhaps there are more meaningful outcomes that you could use to judge the success of your article – such as can people recall your article or an increase in speaking or consulting opportunities – but I know from own blogging experience that page views (and time spent on the site as well ) is a strong motivating force.
And is content posted on a blog much different from intranet content?
Thank you for your comments, Andrew.
You’re quite right that I DO measure page views for this blog, but I do so because it’s much harder for me to measure outcomes than it is for my intranet. I’m also acutely aware of how I can easily manipulate page views on my blog (by advertising on twitter and G+) for example and how those extra visits, do not reflect success.
You do give 2 great examples of outcomes — thank you. Speaking opportunities on the topic of ‘Perils of intranet measurement’ or maybe how many times my own article is quoted back to me by my management or other intranet professionals. I’ll endeavour to keep a tally 😉
I believe that measuring outputs is a great start, but that we should all look to measure outcomes if at all possible. That is significantly harder, but is a better reflection of success.
Of course, I forget this important point: page views don’t equal page reads or indeed, page understanding and action.
Outcomes of course are best. The tricky bit is finding a “signal” of successful outcome, something you can measure easily. This takes work, principally observing closely what people actually do with content. One thing I notice is a tendancy to produce a lot of measures across the whole of an Intranet. It’s only by focusing on the things that are really important that you can make time to measure properly.
You’ve just reminded me of a project I worked on for a research institute where one of their key measures of success was citations of published papers.
In other words, if your research paper was published in a journal and then cited or referenced by a lot of other researchers, this was considered a success. So it wasn’t enough just to get your research published – it had to be ‘endorsed’.
An idea worth thinking about with possible applications to intranets I think.
Absolutely, I like that idea. Citations is an outcome, papers published is a useful output.
Retweets, PageRank and backlinks are all effective outcomes for blog posts where you’re looking to gain consultancy or speaking opportunities. Clicks on ads are what most websites use ;o)
To get backlink reports on your intranet you’ll need a spidering technology that can provide these kind of reports back to you. GSA does it, for instance.
I’ve not used Yammer, but I imagine it provided retweet reports?
Comments, feedback, and ratings can all help show the value of specific content on a site.
And sticking ads on your intranet shouldn’t be rejected out of hand; local services information for people coming into your office can be extremely useful to users and a source of income for the team.
But ultimately pageviews should inveigh on the content; if it hasn’t been looked at in say a year, it shouldn’t be active on your intranet.
[…] Continuing financial crisis causes companies to drive savings and benefits from all of their spend. Intranets will have to prove their worth and demonstrate what’s being used and what’s waste. Good metrics that clearly demonstrate positive and productive outcomes – not outputs – w…. […]
[…] Facts are friendly. Take time to review your analytics as it will give you great insight on how your intranet ticks. Is it well visited? Where do people go? What are the results of their visits? For some further thoughts on intranet measurement, see our post on intranet outcomes. […]
Hi Jonathan,
Page views and time spent on a site are basic measurements that give some indication of what is happening but it can be misleading. You need to enrich this data with other more qualitative measures.
Take a news story for example. If you can rate the item, like it and share it with other people and that information be known to the owner that would help give a better picture.
If some people blog their opinions on what they thought about the news item or there is a discussion forum that covers the news item it helps to show what the impact has been in people’s understanding and changes in behaviour because of reading it.
Mayve an online poll to show people’s first impressions will help too.
My point is usage stats are important and shouldn’t be ignored. Having more information helps build a more accurate picture to show what the outcomes have been.
Mark
I like outcomes and also ‘actions’ as objective measures. In an intranet context, even in a blog context to be fair, hits, time on page etc are pretty useless stats. We know people rarely ‘browse’ intranets, they go there to complete tasks, so the effectiveness of doing so should be where the measures are. For basic news reading, getting to the end of the article is pretty much the only widespread objective – you can’t measure that with anything other than heat mapping can you?
Also, don’t forget the maxim: H.I.T.S – How Idiots Track Statistics. And the point that if advertisers paid Google based on actions from ads (e.g. sales/enquiries) and not simple clicks on them, Google would not exist.
Similarly, I did some work at a hospital where there were 18,000 staff. A manager was explaining to me in a research interview how he reprimanded one of his nursing staff because she was spending up to 4 hours at a time on the Sydney Morning Herald site. Obviously there was no way she was spending that time on the site, but that’s what his little stats tracking dashboard was telling him. In reality she’d often leave the page up as a browser tab while she was on shift (often for 13 hours).
Great measurement on intranets is hard. No doubt about it. Things like ratings, comments etc are all data, but don’t tell you much as not everyone leaves them. As suggested, you have to combine multiple raw data streams with insights from other means to get a somewhat clear picture of things.
Also, I *like* seeing great traffic on my blog, but it really doesn’t mean much and is not the reason I have the site. I look at Google Analytics maybe once every three months and the basic typepad stats only if I’ve posted something I know is likely to be popular and there could be a spike, which is rare. 😉
[…] offline comms channels in Google Analytics: Measuring success means looking at outcomes, not outputs. More often than not, that means looking at more than just page views. Here Luke Oatham gives a […]
I agree it’s the outcome that we should be concerned with. However, things are not so straight forward. Most organisations would readily produce quantitative statistics and use them as proxy to get a sense if info in intranet is useful to users. It’s hard to determine what are good stats to keep track of. e.g high count of page views does not necessarily mean that users use the info or find the info useful for their work. It could be an indicator which requires further steps to confirm if the info is indeed useful.
It is not easy to measure usefulness without taking further steps to ascertain as every user who view the same document/content may conclude the degree of usefulness differently as their needs are likely to be different. Even if we add in a rating of usefulness for the content, it’s subjective and does not give us the context of the user’s rating.
Conducting surveys and focus groups is more resource intensive and time consuming than generating statistics/reports from the system. Not all organisations are prepared to regularly conduct surveys or focus groups as more time and resources are required to study the feedback from the surveys and focus groups, synthesis and analyse them and come to a conclusion.
Thus, measures should not be just focused on quantitative statistics and should also include the qualitative aspects to provide a holistic view. Like I said, it’s not easy to do it well. Thus even up to now, there are still discussions among intranet professionals on what are the key measures that will show usefulness of intranet. There’s a LinkedIn Discussion on “Which KPIs are used in your intranet”: http://linkd.in/UahzKm
We cannot be ambitious to try to measure too much (may lead to analysis paralysis) and also cannot measure too little and can’t have a good sense. What are really considered very critical and relevant measures/KPIs (say, top 5 measures) to show an intranet’s usefulness?
[…] true but, Jon has missed an important point – one that he has actually made himself in the past, That its not about the outputs, it’s about the outcomes. We need to forget about measuring the number of likes itself and focus on what behaviors the […]
[…] turn is linked to decision-making ability within the organisation. We’ve talked before about measuring outcomes, not outputs – and Klout scoring is simply a measure of output. More robust analysis of influence within […]
[…] Before assuming that obvious stats and the twmeasures SharePoint or your tracking tool provides are good enough, remember the Intranetizen ‘so what?’ test: […]
[…] their language. Don’t talk about cost, talk about value. Don’t ever be tempted to include ‘time efficiency factors’ in your business […]
[…] Before assuming that obvious stats and the measures SharePoint or your tracking tool provides are good enough, remember the Intranetizen ‘so what?’ test: […]
[…] the goal is ‘hits’ and positive feedback. Measuring the wrong things means chasing outputs rather than outcomes and hinders the intranets ability to offer real benefit. Utility is the main driver of adoption; […]