Multivariate Testing, A Work in Progress

I am very excited about google's web optimizer, which makes multivariate testing easy and manageable. I have run various A/B tests for the pursposes of website optimization in the past, but always with unweildy manual systems. the variables very quickly spun out of control, and the results had to be manually processed and interpreted.

I have struggled for the last few days to set up a multivariate test on a dynamically generated site, with limited success (server problems have meant limited ftp access to try out the quick/easy wordpress plugin that I found after a little searching. Its called Google Website Optimizer from Content.Robot.)

So, I thought I would set up the experiment here instead. Of course, I'll have to pay all of my friends and loved ones to come and visit the site achieve statistically significant volumes, but it will at least give me practise at setting up the process.

So, first I have to identify what action I want people to take. I'll start simple and just ask them to

click this link to participate in my multivariate test.

The variables I'll test will include

1. put the link in the middle of the page versus the top of the page
2. including an image or not
3. 2 alternate headlines

And I'll create a "conversion page", to use the google website optimizer lingo, thanking site visitors for playing.

p.s. clearly I should have posted my conversion page first, so they would appear in order

Emerging Keyword Trends With Hittail and Blogpulse

How do you identify appropriate keywords phrases when they are only hours or days old?

Take the phrase "subprime crisis" for instance, or "global food crisis". These are subjects that have only recently been thrust into the public consciousness. Language is constantly evolving, but SEO experts need to be ahead of the curve.

Using keyword research tools like wordtracker or Trellian is excellent for understanding the "voice of the customer" - words that prospects use to describe the product or service that you offer- as long as the phrases are well established. But what about keyword phrases that are evolving or being coined today? By the time an SEO expert has found them, he is already behind in the race to get search engine position.

That is where tools like Hittail and Blogpulse come in.

I've written about Hittail a couple of times, but if you haven't downloaded it, you can get it here free: www.hittail.com.

Blogpulse is a tool from Neilsen Buzzmetrics. You can check it out at www.blogpulse.com.

Hittail give you a realtime stream of keywords, as well as the ability to easily repeat the search that brought your visitor. It allows you to identify emerging keyword combinations quickly.

What it doesn't do is help to expand your keyword list to related terms.

If the phrase is very new, the keyword research tools won't help you here.

So how can you identify related keywords to the longtail phrases Hittail shows you from your site traffic?

For this I have recently started to use Blogpulse. Results are fresh. timely. and framed in the language of the people who are most interested...niche bloggers and the site visitors who post their comments. I have to comb through the results, looking for useful terms and phrases. It is not neatly packaged like a keyword tool, and volumes are anyone's guess. But the wording in use by the bloggers and commenters is a good starting place for building out a list.

The site also displays helpful charts showing incidences of the phrase over time.

I am on the hunt for a more immediate keyword research tool. Can anyone suggest one for me?

I'm Trying Out Hittail - Part 2

I wrote a couple of weeks ago that I had installed Hittail on this blog, in order to get a better sense of what subjects my readers are interested in. I wanted to see which topics the Hittail engine would recommend that I focus on.

Well, after a week, I took the next step and installed it on several higher traffic blogs.

So far, the results are encouraging....and somewhat addictive for the compulsive web-analyst. (Did I mention I'm thinking of starting a support group for marketers who feel compelled to while away their nights studying emerging user trends?)

At the office, we've taken a couple of (I think) exciting steps to align our editorial and marketing efforts, and the Hittail application is playing a role in that process. It is helping us to determine which stories to feature, allowing us to craft more targeted headlines (both to improve SEO and the user experience), and increasing our awareness of which keyword phrases we should be monitoring position and optimizing for.

A Word on the Hittail Suggestions Tool

Hittail provides suggestions as to which keywords your website should focus on optimizing for. As a self-professed metrics junkie, I've wasted a few thoughtcycles pondering the algorithm that identifies words.

Hittail is about the longtail strategy, so the recommendations are not based on keyword volume. Some of the words it has selected are pretty obscure, so it doesn't seem to be any kind of keyword clustering. From what I can hypothesize, it seems to select words for which your site appears fairly low in the rankings, but for which you still managed to attract a visitor (implying that those who ranked above you provided unsatisfactory results for the motivated searcher). Presumably, I guess, if you could improve you rankings on these phrases, you would attract much more of this type of motivated/previously unsatisfied traffic to your site.

I have no real idea if this is how it works. I'll be posting to the forums at hittail.com to see if I can ferret out an answer. Meanwhile, I can recommend the tool, and report that it is proving useful for me on numerous levels. You can download it



Anyone else out there using hittail? How are you implementing its recommendations?

Predictive Versus Descriptive Modeling with Analytics

I wanted to post about approaches to using web analytics data that recognize its limitations, and its power.

Many organizations use historical analytics data as a basis for forecasting future growth, and establishing performance goals and budgets. This applicaton for analytics data can blur the distinction between predictive and descriptive data. Understanding this difference is critical to an effective analytics program. It generally falls to the analytics professional to ensure that the difference is clearly understood within the organization.

I'm going to start out with a couple of definitions. What do I mean when I say predictive versus descriptive modeling?

Predicitive modeling refers to a mathematical model that can accurately predict future outcomes. For instance, I know that if I apply sufficient heat to water, the water will reaach 100 degrees celsius and begin to boil (barring slight variations for altitude which are also predictable). The rate at which this happens and the amount of energy required can be mathematically described.


Descritive modeling refers to a mathematical model that describes historical events, and the presumed or real relationship between between elements that created them. For instance, yesterday when I went to the store to buy milk, it cost me $1.00 a litre, last month it was 95 cents, last year it was 80 cents.. Based on historical events, I assume it will cost me roughly $1.05 to buy a litre of milk next month.

Web analytics falls in to the latter category. It is a set of descriptive, historical statistics.

Past Versus Future Performance

I direct marketing activities for a division of a financial publisher. The company has an outstanding track record for identifying and recommending market-beating stocks. Historically, they've made recommendations that represented a lot of money for a lot of investors worldwide.

But at the end of each financial report that we send out....and those of virtually any financial advisory service or market report is a warning to readers something like:

Past performance may not be indicative of future performance.


***Web Analytics reports should carry the same user warnings***


Have any of you ever sat in a management meeting in which company representatives have said something like, "We have the data to demonstrate the relationship between ad spending and revenues, so if we want to grow sales by 20%, we just need to ratchet up our ad spending accordingly."?

This kind of thinking fails to recognize the rate of change in markets, technologies and the competitive landscape, and fails to factor in the concepts of resource scarcity and the law of diminishing returns, as well as the potential from economies of scale. That's without even considering the inherent flaws in any data gathering and processing system.

So, if the operation of complex markets can not be accurately predicted by simplified analytics models, what are the more tangible uses of historical analytics data?

1. To identify broken systems. A significant change in performance data can often indicate a technical problem, overloaded systems, broken links, faulty logic etc.

2. To select between alternatives. Analytics is particularly apt for testing market responses to different offers, creative, or sales processes with A/B or multivariate testing. It can also provide a guide as to which channels or markets tend to be most lucrative or most cost effective among existing channels. (The challenge, then, is often to find ways to expand the more lucrative channel).

3. To flag new market opportunities. A careful study of web analytics data can reveal new opportunities for cost savings, revenue generation or operational improvements.

4. To extablish a meaningful dialogue with existing and potential customers. Web analytics data can help us to learn about customer needs, desires and propensities. It can teach us the language that the customer uses to articulate their needs, so that we can respond meaninfully. It also can give us parameters to personalize the user experience to better meet their needs and create loyalty, trust, and ultimately customer satisfaction

One final VERY POPULAR use for historical analytics data

And hey,...IF your company persists in falling in to the infinite resources analytics model trap. IF they continue to hold fast to the belief that past performance predicts future performance, you can always use your historical analytics data as a roadmap to all of the external factors that caused your company to veer off its charted course.

My Other Blog is Spam?

A few days ago, as I found myself writing here, and trailing off in to other subjects, I decided to set up another blog on a favourite subject of mine: offer strategy.

I went and set up the url http://offerstrategy.blogspot.com, and over the next few days, wrote a total of three articles on the subject of creating sales premiums that make sense. Designing offers that help to close sales.

The blog has a total of three external links:
1 to this blog.
1 between two articles
1 to an article on my company's website about the mortgage crash in the US (which is topical to the article, since I think it can be argued that the crash is the result of flawed offer strategy)

This morning I receive mail from blogger informing me that my blog has been flagged as a potential "spam blog"

I went and read about the defininitions of a spam blog, and it cited pirated content and volumes of nonsensical links as the general offenders. I am a little stunned to discover that my three little articles, and three little links could somehow have set off alarm bells someplace. I am curious to know how it could possibly fit the criteria, and if so, do the criteria make sense?

Anyways, I have put the blog up for review by the humans at Google. At least someone will be reading it now, as I hadn't gotten around to publishing it in any way...I was going to wait until there was a little more content.

Has anyone else had their blog unjustifiably locked as potential spam? Please share your story with me. Meanwhile, I'll climb back down off my soapbox now. Sorry for having strayed from the subject matter. To get back to analytics...I am currently working on an article for the WAA Blog about User Generated Content. Hope to share that next week.

I'm Trying out Hittail

A couple of days ago I installed the the Hittail real-time search hits application on this blog. You can register and get the code for your own site at http://www.hittail.com. I am running the freebie version of the application, for now.

I learned about Hittail on the Web Analytics Association site blog. If you are not a member, and you are interested in Web Analytics, I suggest you join the Association.

• the community is very helpful and supportive
• they organize educational events
• the forums are very active, and provide a good mix of practical information along with insights on where the industry is heading

Anyways, I installed Hittail here on this blog, and then, when everything seemed to go well, I also installed it via the wordpress plugin on http://www.moneyweekes.com. So far...no data on that site. Now, I am not panicking yet, because it took a couple of days for the stats to register for this blog. But when they appeared, all the historical data was there.

As a surprise bonus, in addition to reporting keyword traffic, the system informed me that my blog had been stumbled (I had been meaning to submit this blog to stumbleupon, but I'm delighted that someone else found the site and gave me the thumbs up), and received linkedin visits and traffic from the WAA today.

While all of those are great to know, I'm not sure how they will play in to the software's stated objective to help me identify which longtail keywords I should be optimizing for.

Ambar Shrivastava from Hittail described the primary benefit as follows:

"The idea is to provide actionable data that helps webmasters and bloggers better connect with their existing readers as well as attract new readers interested in similar topics to what you already have on your website. This differs from other analytics tools that mainly describe what is happening on your site with little information on how to improve it."

I ran into a couple of snags along the process of installing (what can I say, I'm tired...it's easy, but I was firing on half cylinders). And I can tell you that the team over at Hitbox fairly jumped at the chance to help me out. They get double thumbs up for customer service.

I have shared the application around the office, so I'll have more to report back shortly, I expect. We are all anticipating good things. I work for a publisher, so our sites are very original-content rich, which seems like a good fit for Hittail. However, the moneyweekes.com site is i Spanish. Not sure how the application will handle other languages.

One of our programmers looked unimpressed and muttered "um, you know you can get that data from your log files, right?"... Yeah, I know. But Hittail does digest the information for me and highlight key points...and theoretically it provides optimization suggestions based on my words. I say theoretically, because my search volumes don't merit any suggestions as yet. . And most exciting of all....it does it in almost realtime. So I can start getting a sense of when people are visiting the sites...what is the best time of day to publish my newsletter? How often should I be posting here? I read on their website that I should have about 100 pages minimum.

So if you found me through a searchengine (or maybe even if you didn't) chances are tomorrow I'll be thinking about your keyword search, and how I might better address the issues you seem to be searching for online.

Anyone else out there installed the Hittail code on their site? How is it working for you?

Knowing When NOT to maximize your ROI

I was just reading a fascinating discussion over on Jim Novo's blog about the effectiveness of social media sites versus chatrooms as channels for advertising.

One commentator made a suggestion that Jim could "do a bad job" with his cpc campaign...as in design ads to attract fewer, but more targeted clicks, as a means to better optimize his ROI. Of course, it makes perfect sense. It is like designing incentives that are only of interest to the audience that would buy your product.

Don't give away a palm pilot - everyone wants one of those. give away a demo version of your product, or a discount on purchase, or a whitepaper that compares and contrasts the features of your product versus the competition - something that removes barriers to purchase.

The advertiser is happier. The respondent is happier - because you are speaking to their specific needs. Can anyone tell me which stakeholder loses out in this equation? The publisher.

In my opinion, this is a relationship that must be considered when measuring ROI, that hardly ever makes it into anyone's calculations.

The supply of quality publishers depends on their ability to generate revenues from ads. I have had the experience at times in the past of negotiating so hard with a publisher, that they refused to renew the contract when it had run its course. So, ok...I got the client the best possible deal in the short term, but what were the long term effects? I had "used up" available channels to access my target audience.

A client runs SEM campaigns on Google and also a very successful Affiliate program.

They pay their affiliates 5% of revenues.

SEM on a cost per click basis that works out to roughly 20% of revenues.

Each represents approximately one third of sales.

I suggest to the client that they could afford to pay their affiliates 8%.

The first response is, of course,

Why pay 8% when their affiliates were doing it for 5%?

The most important step in the process is to recognize affiliates as an important stakeholder in the sales process. That one can "create" more lower-cost inventory by taking care of a particular stakeholder group, or encouraging its development.

So then the questions
1. how many more affiliates can be attracted at 8%?
2. how much can SEM campaigns be reduced in response? and what is the expected cost per acquisition at the new, presumably optimized (and therefore more cost effective) volume.
3. what is the associated cost of marketing to this stakeholder group to bring them on board in sufficient volumes?

It is a step...or rather a leap of faith, that requires some careful calculating, careful planning, and total commitment - that stems from the recognition of the multiple stakeholders that make your organization work.

Can you share stories of when it made sense for your business to pay MORE for customer acquisition?