Week 7 — Human Computer Interaction — Running Web Experiments
A/B Tests, Split testing, ex. — Splitting the traffic of who comes to the site. Collect metrics (conversions, click throughs, etc.) 3 versions of a site - # of columns - getting more content “above the fold”. Page A / Page B / Page C — calculate % traffic (same for all 3) vs. new sales and see the change. Page C had a drop in sales!
Ways design makes a difference! Small changes in a design can make a real difference in the effectiveness of the site.
- The position and color of the primary call to action.
- Position on the page of testimonials, if used.
- Whether linked elements are in test or as images.
- The amount of white space on a page, giving the content space to breathe.
- The position and prominence of the main heading
- The number of columns used on the page.
- The number of visual elements competing for attention.
- The age, sex, and appearance of someone in a photo.
Content above courtesy of A List Apart http://alistpart.com/articles/designcancripple
Question: How do we measure the outcome and if it’s what we want? With tests!
“Our expectations are often wrong.” - Scott Klemmer
“You should follow me on twitter here” had the highest number of click through on Dustin Curtis’ blog.
Large Scale changes design: Make smal but consequential differences detectable. Small differences accumulate. Beware of anomalies: investigate further. Some of the changes you may see may be anomalies. Important to run real manipulations to avoid data anomalies.
Principles for Effective Inline Experiments
- Run experiments with equal number of people in each of the two conditions. This is the fastest way to detect an effect if your design is causing a change that matters.
- Initially, ramp things up. If your deign is too drastic, and it’s negative, then you don’t want to hit 50%. Ramp it up to 50% once you make changes along the way. If you don’t see a positive effect, then roll it back.
- Figure out what you want to measure. “Pick a meaningful yardstick”
- Run your experiment long enough for people to get accustomed to it. Sometimes furst use is not the same as what people are familiar with.
- Rules for random assignment: assignment should be consistent (same person should see the same interface every time they logon). It’s important that the assignment is random.
- By running controlled experiments you can be sure it’s your changes that are causing the effect. Run online tests, and combine it with in person studies.
Designer on the online age!
- Role has shifted to being about creating multiple alternatives.
- People are often too sure of themselves.
- Rapid experimentation can help you fail fast to succeed sooner.
Analyzing Experiments - Rate Comparison
3 Questions that you can ask and answer by analyzing your data:
- What doe smy data look like - graphically. Look for paterns.
- What are the overall numbers? Aggregate stats for quick summary. Usually mean and standard deviation.
- Are the differences “real”? Is the number significant enough?
Chi- Square Test
(Observed - expected)^2 / expected
- Critical values for chi-squared - our P value.
- As the chi-square number gets bigger, our the difference between expected and actual grows, then it is unlikely that this information could have been created by a unbiased coin.
- Degrees of freedom - number of choices you have minus 1. (ex. A 6 sided die degree of freedom is 5.)
The Null Hypothesis - our opening bid, is “we dont think there’s a different between our two conditions.” Ex. The coin is not loaded.
- Check whether the data falsifies the null hypothesis.
“To get a feel for your data, graph it all!” - Scott Klemmer