Predictive Analytics...Maybe the Models are More Accurate than I Imagined

I attended an Emerging Markets conference last week alongside a group of entrepreneurs, libertarians, and disheartened taxpayers from the first world, looking for ways to escape the long grasping arm of the tax man, the encroachment of inflation and the total destruction of their wealth.

Bill Bonner, in his opening remarks, complained to us that economics is a dismal science. He shared a joke about 4 economists who get lost in the woods, and sit down to calculate their whereabouts.

The punchline goes...."you see the second mountaintop north of us across the valley? Well, we're there."

Reminded me of a lot of marketing budgets I've been called in to assess over the years. The inputs looks sort of right...the outputs are believable as long as you don't look at them too hard, or compare them to the reality.

It also reminded me of another joke about an economist and an engineer stuck on a desert island with a crate of canned beans.

After the engineer loses an eye and the contents of a precious can of beans in a failed thermodynamics experiment over a roaring campfire, the economist leaps to his feet. He proclaims he has the problem licked...

"First off, assume a can opener..."

I set about reading on the subject of asset allocation for the purposes of risk management over the last few weeks for work, and because predictive models interst me.

The prevailing theory speaks of an optimal risk curve, a theoretical line along which the maximum return is achieved for each level of risk. This formula supposedly automates the process of selecting the best assortment of investments for each individual according to their risk tolerance.

The problem with this, of course, is that someone -- a person -- inputs the assumed level of risk for each investment, or investment class into the model. The key input.

Which of course assumes that the person inputting actually knows the level of risk involved in an investment.

"assume a can opener....."

So what you end up with is the clear illustration of a concept with little utility. With worse than little utility. With awesome destructive power.

This model formalizes someone's 'educated guesses' and gives them the weight of prognostication via clever formatting. The individual investor is seduced into turning over their hard-earned funds under the guise of hard science.

Hang on...I guess it really is an extremely effective and accurate model....

Effect at separating the investor from his money.

Which brought me back to thinking about how accurate predictive analytics models are in predicting -- and altering -- buyer behaviour. I guess it depends on which buyer's behaviour you seek to alter. The online shopper, or the marketing exec thinking about buying the product?

Brand Awareness is a Metric, NOT a Goal

OK just a quick one. Here it is 1:00 in the morning, and I have to write this because it's on my mind. I'll regret it when the alarm goes of at 6:something.

I read a whitepaper recently, and it started off by making the critical point that in order to design metrics, you must be clear about your objectives. OK we all agree.

But then this same document goes on to outline brand awareness as an objective.

Now at first, this didn't jump out at me.

But has brand awareness ever been a business objective? Do people go into business to maximize brand awareness? Or is brand awareness "quantified", a measure of your effectiveness at reaching your target audience and disposing them to consume your product?

The Acid Test for Custom Reports

OK, I'm speaking to the people who work in small entrepreneurial comapanies here. For those of us who chose to work in a dynamic entrepreneurial setting do it for the rush of immediacy. The ability to move quickly. to stay light on our feet, and make real change.

Lots of times, in performance meetings, someone at the table (ok, it's usually me) will pipe up and say "How hard would it to be able to just find out ..." It's a theoretical discussion only.

Someone has a new report request.

Maybe its something they've thought through, and is going to revolutionize the way we do our business.

Maybe its just idle curiosity.

The tech people at the table immediately tend to jump on the "How do we do this?" bandwagon, totally bypassing the "Is this something we should be allocating valuable resources to?" train.

And usually, at first blush, the answer looks and feels like ...pretty quickly.

But how often is that true?

Sometimes, what happens at that point, is someone pipes up and "authorizes" the report, based on the assumption that someone else can whip it up over sandwiches today at lunch.

Then that someone else spends the whole afternoon on it, because they ran into a few unforeseen stumbling blocks. Oh, and by the way, they have a couple of questions about how the reports should be formatted...so now followup meetings are being scheduled, and the person who first requested the report, is now piling on additional feature requests, unchecked. And the dominoes start to topple....

The project has outgrown the petrie dish, and is limping hideously through the corridors of your place of work, wreaking havoc. People have forgotten its original, innocuous status as a "theoretical discussion". The hours invested in it have infused the project with the value of human sweat.

Let's take a peek a few weeks down the road, and go ask how much time the new report is saving? A couple of possibilities (terribly overgeneralized of course)

1. The person who first requested the report has a new spring in her step, and has lost the haunted look that comes from too many hours manually calculating stuff that is better done by a computer.

This is the result you're going for. Congratulations.

That young genius is probably going to start poring over the reports and come up with a great recommendation that will materially alter how you do business...save you a gazillion dollars, and pay for the implementation 40 times over before next week's meeting.

2. The person who requested the report has relegated it to the pile of "stuff I don't need to babysit anymore." Ask her how it's progressing, and she'll pull a report for you while you wait. ....and oftentimes discover that the data is being pulled incorrectly or the report is garbled, or not in a format that is useful to anyone. ***By virtue of having been authorized in the first place, the project has been elevated to the status of "stuff worth doing properly"****. At this point countless additional hours may be sunken into the pursuit, before any kind of cost:benefit analysis happens.

3. The person who first requested the report is completely buried. The time requirement to analyze the implications of the new report is consuming them and they no longer seem to have time to stop and think about how much benefit the new information affords them(factoring in all of the data exceptions, annotating the performance anomalies that affect the data output, and closing the knowledge gap between the data and the information that data stands proxy for).

The Three Things That Should ALWAYS Happen Before Anyone Builds a Custom Report

1. Write a SPEC. Even for a little thing. This process is great for shining a light on the holes that are so easy to gloss over in discussion. We sometimes want to rush past this step. We know exactly what we want. We think we have expressed it clearly. We think there is no room for error.

I'm married to a system architect. I've tried asking him to just whip me up a report (I am a self-confessed data junkie). Without a written spec, he refuses, even when I assure him its quick and easy. Even when I say pretty please or bat my eyelashes.

Try this. If is a simple report, it will only take you a few minutes to write out the functional spec. Describe all the inputs, the data sources, and all the possible outputs, depending on what inputs the report receives. This will bring you a lot closer to a shared understanding with the people who are building the report.

If having the report is not worth the time it takes to properly specify how it works, you probably don't need it.

2. Remember that data is only a proxy for information. It is imperfect, and prone to misinterpretation. It can act as a red herring, or mask important trends. Be sure that everyone involved understands how the data is being calculated. Call the data what it is, not what it is supposed to represent.

Create a A data Glossary of Terms.

3. Things automated are easily forgotten. Ensure you build in a mechanism for following up on the results of the report.

Final Notes

There is a lot of room for assumptions when people toss an idea about over the boardroom table. But programming a custom report is an exact science. Many a programmer has misinterpreted the requirement, and built a report that does not meet the need. And often the report that DOES meet the need is a LOT harder to build.

Multivariate Test Conversion Page

Thanks so much for participating in my multivariate test....

Or, if you came straight here from somewhere else, and are willing to take 2 seconds to participate, I'd appreciate it. Just click the link.

Multivariate Testing, A Work in Progress

I am very excited about google's web optimizer, which makes multivariate testing easy and manageable. I have run various A/B tests for the pursposes of website optimization in the past, but always with unweildy manual systems. the variables very quickly spun out of control, and the results had to be manually processed and interpreted.

I have struggled for the last few days to set up a multivariate test on a dynamically generated site, with limited success (server problems have meant limited ftp access to try out the quick/easy wordpress plugin that I found after a little searching. Its called Google Website Optimizer from Content.Robot.)

So, I thought I would set up the experiment here instead. Of course, I'll have to pay all of my friends and loved ones to come and visit the site achieve statistically significant volumes, but it will at least give me practise at setting up the process.

So, first I have to identify what action I want people to take. I'll start simple and just ask them to

click this link to participate in my multivariate test.

The variables I'll test will include

1. put the link in the middle of the page versus the top of the page
2. including an image or not
3. 2 alternate headlines

And I'll create a "conversion page", to use the google website optimizer lingo, thanking site visitors for playing.

p.s. clearly I should have posted my conversion page first, so they would appear in order

Emerging Keyword Trends With Hittail and Blogpulse

How do you identify appropriate keywords phrases when they are only hours or days old?

Take the phrase "subprime crisis" for instance, or "global food crisis". These are subjects that have only recently been thrust into the public consciousness. Language is constantly evolving, but SEO experts need to be ahead of the curve.

Using keyword research tools like wordtracker or Trellian is excellent for understanding the "voice of the customer" - words that prospects use to describe the product or service that you offer- as long as the phrases are well established. But what about keyword phrases that are evolving or being coined today? By the time an SEO expert has found them, he is already behind in the race to get search engine position.

That is where tools like Hittail and Blogpulse come in.

I've written about Hittail a couple of times, but if you haven't downloaded it, you can get it here free: www.hittail.com.

Blogpulse is a tool from Neilsen Buzzmetrics. You can check it out at www.blogpulse.com.

Hittail give you a realtime stream of keywords, as well as the ability to easily repeat the search that brought your visitor. It allows you to identify emerging keyword combinations quickly.

What it doesn't do is help to expand your keyword list to related terms.

If the phrase is very new, the keyword research tools won't help you here.

So how can you identify related keywords to the longtail phrases Hittail shows you from your site traffic?

For this I have recently started to use Blogpulse. Results are fresh. timely. and framed in the language of the people who are most interested...niche bloggers and the site visitors who post their comments. I have to comb through the results, looking for useful terms and phrases. It is not neatly packaged like a keyword tool, and volumes are anyone's guess. But the wording in use by the bloggers and commenters is a good starting place for building out a list.

The site also displays helpful charts showing incidences of the phrase over time.

I am on the hunt for a more immediate keyword research tool. Can anyone suggest one for me?

I'm Trying Out Hittail - Part 2

I wrote a couple of weeks ago that I had installed Hittail on this blog, in order to get a better sense of what subjects my readers are interested in. I wanted to see which topics the Hittail engine would recommend that I focus on.

Well, after a week, I took the next step and installed it on several higher traffic blogs.

So far, the results are encouraging....and somewhat addictive for the compulsive web-analyst. (Did I mention I'm thinking of starting a support group for marketers who feel compelled to while away their nights studying emerging user trends?)

At the office, we've taken a couple of (I think) exciting steps to align our editorial and marketing efforts, and the Hittail application is playing a role in that process. It is helping us to determine which stories to feature, allowing us to craft more targeted headlines (both to improve SEO and the user experience), and increasing our awareness of which keyword phrases we should be monitoring position and optimizing for.

A Word on the Hittail Suggestions Tool

Hittail provides suggestions as to which keywords your website should focus on optimizing for. As a self-professed metrics junkie, I've wasted a few thoughtcycles pondering the algorithm that identifies words.

Hittail is about the longtail strategy, so the recommendations are not based on keyword volume. Some of the words it has selected are pretty obscure, so it doesn't seem to be any kind of keyword clustering. From what I can hypothesize, it seems to select words for which your site appears fairly low in the rankings, but for which you still managed to attract a visitor (implying that those who ranked above you provided unsatisfactory results for the motivated searcher). Presumably, I guess, if you could improve you rankings on these phrases, you would attract much more of this type of motivated/previously unsatisfied traffic to your site.

I have no real idea if this is how it works. I'll be posting to the forums at hittail.com to see if I can ferret out an answer. Meanwhile, I can recommend the tool, and report that it is proving useful for me on numerous levels. You can download it



Anyone else out there using hittail? How are you implementing its recommendations?