Last Exit Westend

The other day I was getting curious about what people would have to say about the limitations of Web Analytics.

Besides the usual suspects (mostly of the kind: “Is the data trustworthy?”, “Is tool A better than tool B?”) some very inspiring posts came up: Shelby Thayer’s “Web analytics limitations … and a bright future” (here). From there I found a link to Jospeh Carrabis’ excellent post “The Unfulfilled Promise of Online Analytics” (part 1 here).

The main problem for organizations, it seems, is to put Web Analytics data into good use. Or, even worse: into any use.

The situation depicted in both posts is awkward, but so true: a lot of effort is put into getting tools deployed and getting insights from the data. A horde of analytically inclined people, most of them tech-savvy and knowledgeable in statistics, are looking at reports and are providing documents stuffed with insights and recommendations – and all they can do after delivering their stuff to their organization is to enjoy the silence.

The idea that Web Analytics data could provide a validation framework for “failing faster on the web” (Avinash Kaushik) seems to be less and less true these days: most marketing organizations are doomed to succeed – and Web Analytics data/reporting is only adding one particular bit of information to the agenda of the responsible marketing manager: “Yes. My campaign was a success! And this report proves it!”

The claim that Web Analytics could turn a company into a data-driven organization has become somewhat stale through repetition by now. Replacing gut feeling with solid measurements is a great idea, but the way data is commonly looked at by Non-Analysts in companies reveals a big empty space where the famous “Management buy-in” should be located.

Carrabis points out: “Whenever there’s a discipline of numbers it means there’s an evidentiary trail for decisions. Consider the political and psycho-economic meaning of this for a moment.”
And he explicates them in a brutal, but very precise way:
“If I have the option of taking advice from someone who goes with their gut then I really can’t be held accountable because there are no numbers, therefore from any evidentiary standpoint I’m pretty safe. Should things go sour it’s a political issue because there was no real evidence that we should have gone pro or con, we went with our guts, flipped a coin and took what came.

Even better, it was (point finger in some general direction) their gut feeling, we went with it, it flopped, it was their gut not mine, they’re out and I’m still good.

But if I go with hard numbers and my decision is in error? Now it’s psycho-economic and I’m the idiot and fool because I didn’t understand what I was doing. Both I and the group that helped me make the decision are forfeit.

So which is politically safer to place higher on the corporate ladder, to listen to and feel good about?”

We seem to look at a classical contradiction between organizations being run by common goals or by politics. Politics in organizations normally comes with a strong sense of hierarchical interests, while common goals are said to replace the hierarchical direction of a company with a more reasonable approach towards objectivity, clarity, and transparency.
That’s thought very romantically, of course.

“Accountability”, the possibility that somebody could be called over the coals, can be seen as a threat for the middle management. Nevertheless Web Analytics tools are being deployed on corporate web sites all over the place.
Maybe this explains a vast proportion of frustration and anger Web Analysts do face in organizations where Web Analytics systems are up and running.

Let’s take a closer look at the constituents of the limitations Web Analytics faces beyond politics.
1. The “fact basis” of Web Analytics data has two dimensions. The measured data is distinctive: a page has been visited or not been visited, a (tracked) link has been clicked or not. Not many people are doubting the binary character of Web Analytics data.
On the other hand tools are using certain techniques for capturing (measuring) data – and for consolidating them in a backend system.
The methods for postprocessing and cubing data are different from vendor to vendor and from tool to tool. Having two Web Analytics tools installed on one and the same web server often gives different numbers.

As a result, we have a contradictory picture on tools and figures: the figures can be doubted, but the measurement principle itself remains undoubted.

2. Everybody is talking about tools, but nobody talks about tagging.
By default, Web Analytics tools are capturing data on page level. Page view, visit and visitor figures are broken down to distinct file entities which historically do relate to URLs ending with the file suffix “.htm*”.
This poses two particular problems: first, many web sites these days are consisting of dynamically rendered contents which are shown in distinct frames or templates. Systems like this are constituting page views with an emphasis on “view”: every page is just a particular view on a conglomerate of contents loaded from a database. Clicks on elements can change the view without a new page being loaded.
Second: with the increase of web sites using Flash it became best practice that clicks in Flash movies are tagged as page views (the distinction between page views and events is not properly used).

As a result, we get a contradictory picture on what tools measure: page-level tagging is understood and taken well, tagged on sub-page and element level is yet lacking standards.

3. Web Analytics tools don’t measure things with relation to people.
“A visitor” is not “a person”, a visit (session) consists of a sequence of clicks. By nature, Web Analytics data is “blind” for sociodemographics and identities. Acquiring traffic through particular sources comes with a certain probability that the target audience has an affinity towards the contents behind an offer – but we can’t be totally sure about that.

To “increase our conversion rate” has a different flavour and appeal than “making more people buy from our web site”. Marketers and Web Analysts often don’t speak the same language – a good reason for misunderstanding one another.
“Segmentation” is yet another source of misunderstanding: marketers may have personas (based on socio-demographics) in mind when thinking about segmentation (“the “digital pro” is on average 36,1 years old, overproportionally male (66%), has a high degree of formal education and mostly lives in a household with two or more persons”); Web Analysts are thinking segmentation as grouping visits by certain formal criteria (“an “engaged visit” consists of at least five unique page views, lasts for more than 120 seconds, and includes at least one internal search”).

As a result, we get a contradictory picture on how users and their deeds are to be seen. The notion of “motivation” and “relevance” usually comes with respect to a person, a persona, or an observer, and doesn’t easily break down into formal visit criteria.

–
What the inclusion of sociodemographic data into Web Analytics can contribute to close this last gap is not yet sufficiently clear: services like “Quantcast” are traditionally focusing on the US, “Yahoo! Web Analytics” seems to catch up and is said to provide data for smaller markets as well (see for yourself this article for the .cz population as an example) – we surely will see more of this any time soon.

The degree of tagging and tracking needed for capturing events and elements has its particular challenges: it requires huge amounts of data which need to be looked at in the context of a visit/session (not with relation to an absolute count of elements on a web site being clicked most often).

“Sentiment” or “Intent” data (mostly captured with exit surveys and stored in separate data repositories nowadays) would need to be integrated with Web Analytics data (again enabling a formal segmentation, for example into “successful” / “unsuccessful” visits) to understand service bottlenecks beyond abortion rates in conversion funnels.

Politics and the technicalities of tagging left aside for a moment the biggest limitation in taking Web Analytics data into use is a lack of “semantic recolouring” of web data. The most important aspect is to bridge the gap between the idealtypic terms relating to people, persons, and personas, and the statistically significant counts and measures applied in Web Analytics.

This entry was posted on Monday, June 21st, 2010 at 10:27 am and is filed under Web Analytics. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.