Kristen Reay

Week 7 — Human Computer Interaction — Running Web Experiments

A/B Tests, Split testing, ex. — Splitting the traffic of who comes to the site. Collect metrics (conversions, click throughs, etc.) 3 versions of a site - # of columns - getting more content “above the fold”. Page A / Page B / Page C — calculate % traffic (same for all 3) vs. new sales and see the change. Page C had a drop in sales! 

Ways design makes a difference! Small changes in a design can make a real difference in the effectiveness of the site. 

• The position and color of the primary call to action.
• Position on the page of testimonials, if used.
• Whether linked elements are in test or as images.
• The amount of white space on a page, giving the content space to breathe.
• The position and prominence of the main heading
• The number of columns used on the page.
• The number of visual elements competing for attention.
• The age, sex, and appearance of someone in a photo.

Content above courtesy of A List Apart http://alistpart.com/articles/designcancripple

Question: How do we measure the outcome and if it’s what we want? With tests!

“Our expectations are often wrong.” - Scott Klemmer

“You should follow me on twitter here” had the highest number of click through on Dustin Curtis’ blog. 

Large Scale changes design: Make smal but consequential differences detectable. Small differences accumulate. Beware of anomalies: investigate further. Some of the changes you may see may be anomalies. Important to run real manipulations to avoid data anomalies. 

Principles for Effective Inline Experiments

1. Run experiments with equal number of people in each of the two conditions. This is the fastest way to detect an effect if your design is causing a change that matters.
2. Initially, ramp things up. If your deign is too drastic, and it’s negative, then you don’t want to hit 50%. Ramp it up to 50% once you make changes along the way. If you don’t see a positive effect, then roll it back.
3. Figure out what you want to measure. “Pick a meaningful yardstick”
4. Run your experiment long enough for people to get accustomed to it. Sometimes furst use is not the same as what people are familiar with. 
5. Rules for random assignment: assignment should be consistent (same person should see the same interface every time they logon). It’s important that the assignment is random. 
6. By running controlled experiments you can be sure it’s your changes that are causing the effect. Run online tests, and combine it with in person studies.

Designer on the online age!

• Role has shifted to being about creating multiple alternatives.
• People are often too sure of themselves.
• Rapid experimentation can help you fail fast to succeed sooner. 

Analyzing Experiments - Rate Comparison

3 Questions that you can ask and answer by analyzing your data:

1. What doe smy data look like - graphically. Look for paterns. 
2. What are the overall numbers? Aggregate stats for quick summary. Usually mean and standard deviation.
3. Are the differences “real”? Is the number significant enough?

Chi- Square Test

(Observed - expected)^2 / expected

• Critical values for chi-squared - our P value. 
• As the chi-square number gets bigger, our the difference between expected and actual grows, then it is unlikely that this information could have been created by a unbiased coin. 
• Degrees of freedom - number of choices you have minus 1. (ex. A 6 sided die degree of freedom is 5.)

The Null Hypothesis - our opening bid, is “we dont think there’s a different between our two conditions.” Ex. The coin is not loaded. 

• Check whether the data falsifies the null hypothesis.

“To get a feel for your data, graph it all!” - Scott Klemmer