Accepting the Inaccuracies of Analytics Data?

Hurol Inan, on his website Hurolinan.com

posts an article entitled

Best Practice KPI Reporting for the Online Channel

In it, he outlines the four factors to improve decision-making

factor number one: The data the reports are based on must be accurate, and free of bias. KPI reports that rely on inaccurate data sets are not only invalid but also dangerous as they may cause the wrong decisions to be made.

While I totally agree with what Hurol Inan has to say on the subject, I have a couple of concerns.

1. While improving accuracy is important, it is, I think, also important to understand the limits of the technology, and the resultant possibility of inacuracies that are virtually unavoidable. This is the natural result of a distributed tracking system that relies on users accepting our tracking cookies, following prescribed paths within expected timelines, and not confusing our tracking systems by returning to a site multiple times through multiple channels. For today, the best we can do is to work towards ensuring that report users understand specifically what the data represents, and that they understand the limitations and potential for error.

2. Accuracy also depends on the company´s definitions and practices. For instance, if you are running multiple ad campaigns, and a user responds to numerous advertisements before making a purchase, or returns through online ads, once already a customer, how do you properly attribute the return on investment for advertising spend? Having clearly defined and published policies on the hierarchy of tracking events is critical both for measuring behaviour, and communicating to stakeholders - both internal staff and external affiliates.

3. In my experience, too often, there is no common understanding of the specific definitions of the data being tracked, particularly among upper management, advisory boards or investor groups. A plan for educating all users on the meanings, definitions, limitations of dashboard reports and KPIs is critical for establishing the practise of trusting and acting on the data, and acheiving upper management support for allowing KPIs to shape strategy.