Will be interesting to see how this goes over the next month or so. I agree with Joske that the delay *shouldn’t* make that much of a difference however a nice fast cart does however make it look a little more professional.
I don’t understand what’s so different about these carts that could possibly lead to a near doubling of sales. Honestly, I would check that your split testing is actually working correctly.
As far as I can see, the new cart loads a bit faster, shows your logo instead of a generic shopping cart icon, and doesn’t ask CD buyers for their ZIP codes. Do you really think those are significant enough changes to explain this?
If so, then imagine what could happen if you fixed your card rendering code to properly center the words and use a single font size!
No matter what caused the jump in conversions, congratulations, but if I was running this split test, I’m pretty sure these results would make me suspicious that some other factors are coming into play and that the A/B split wasn’t isolated as well as I thought.
Hmm. Has the amount of people that entered the screen in question stayed roughly constant? If there are a lot more people trying to buy a copy, maybe there’s an intermediate variable. (For example, some other change has increased the interest level of potential buyers, which both leads them to be more likely to try to buy and more likely to actually do so).
I am surprised that a statistically significant proportion of customers click on the cart without actually making a purchase.
On a design note, I think that including your logo prominently is a great touch. It makes it seem more reputable and inviting.
On another design note, I see that some of your pages seem to have that widget that automatically generates similar articles. How has that been working out for you?
Wow, if those figures are true I’m *flabbergasted* too. I’ll be waiting to see how the conversion optimizer results progress – I can’t quite bring myself to believe that it could make such a difference!
Though I have noticed another big difference between your old cart and your new one that might also help to explain it. I’m not sure if you’ve mentioned this big difference before. On your old cart, if you:
1. click the link to add Bingo Card Creator without a CD
2. close the cart
3. click the link to add Bingo Card Creator *with* a CD
You actually end up with a cart with *two* copies of Bingo Card Creator in it. Which, presumably, very few people would actually want, but, following your “just trying it on” theory (nice theory BTW), I think it might actually happen quite often.
I’m guessing this two-items-in-cart issue might be enough to put people off the purchase altogether – it’s no longer a nicely-greased sales funnel when you have to figure out how to remove items from carts…
Colin: I’ve terminated the original one (since I raised prices yesterday, as per my plan of a month ago) and have started a followup. I intend to wait until that produces statistically significant results, or one week, and then act on them. Early indications (i.e. like one day into test #2) indicate that if the first result was a fluke it appears to be a fluke that repeats when asked.
DanH: Read the article regarding decision paralysis. The long and short of it is cart outperforms non-cart for me, ergo site needs cart. 😉
Jesse: I checked, and I checked, and then I checked again. There is no error unless Google is deciding to throw a gremlin in the works to amuse me, and if they do they’re not adverse to spending a little money to do it. I should *not* be selling as much as I am right now, full stop.
Brian: An A/B test eliminates the “more people are seeing the cart” intermediate variable. The carts aren’t tested in serial (one after another), they’re tested in parallel (both at once — visitors see one or the other, randomly). This means that even if the traffic increases by a factor of ten over the period of your test it doesn’t matter, because half are still seeing the control cart and have the experiment cart.
With regards to the widget generating similar articles: do you mean my blog or my site? On the blog it is worthless and I want to turn it off. On my site it works fairly well, largely for funneling pagerank.
MB: Interesting thought. After you mention it, removing an item from the cart doesn’t seem user friendly either (check a box then click update then wait two seconds).