Business analysts might be able to learn something from quantum physicists. Consider this great article about a discovery that neutrinos might be able to travel faster than light—a feat thought physically impossible. It is a discovery that questions the most fundamental theories about the way the universe works (talk about promotion material!). But the scientists are not celebrating.
Why not? The statistical significance is indisputable (it represented a difference of six standard deviations). The tests were clean. The scientists have rechecked their work multiple times.
The reason they aren’t celebrating is they know the data may not represent what it seems to. Compare this to business, where we routinely celebrate numbers that are much fuzzier than these.
One useful trait many web analysts share is a deep skepticism of the numbers they promote. It is useful, because there are many times that we simply do not know what the data represents.
One reason for this uncertainty is that the measurement methodologies themselves can be wrong without us realizing it. To quote the article: “If you are measuring something incorrectly, it doesn’t matter that you measure it very carefully.”
Despite how carefully we measure things in business, there are so many steps in the measurement process that a slight systemic error can lead to numbers that do not really represent what we think they do. This does not mean that the numbers are wrong, just that they represent something different than what we thought.
When measuring people, we can’t ever take our numbers for granted. We have to be familiar with all the common measurement errors and try to account for them. It is not enough just to recognize that there are measurement errors and then choose to ignore them because we can’t do anything about them. We have to find ways (sometimes new ways) to fill in the gaps and get the data we thought we were actually going to get.
What is the Business Cycle?
Another looming reason for uncertainty is one that we don’t like to admit: we don’t actually understand the business and business cycle. We know that we sell widgets, but we don’t really understand what motivates people to buy them or when or how long it takes them or what all of the influencing factors are in that decision. We can’t create repeatable tests, because there are innumerable variables at play, many of which we don’t or can’t measure. Some of them we don’t even know exist, so we never ask about them to begin with. Visitors to a website can be distracted by bad site design or by their neighbor’s dog barking.
In business we assume that if a finding is statistically significant it is accurate. Unfortunately, it is often difficult to go much deeper than aggregated reports. Without being able to see the entire activity of a single person (which is becoming even less reliable with time), it is hard to understand why they did what they did. Without understanding that, we cannot intelligently optimize our efforts to help them.
The physicists observed the aggregated measurements of many neutrinos hitting a sensor. Physicist Joseph Lykken reviewed the study and said, “You have a proton beam … that makes the neutrinos, but you don’t know which proton made which neutrino. This makes it tough to claim nanosecond timing of the neutrinos.”
In business we might look at a problem like this and decide to use statistical modeling to find the truth. To the physicists who said the same thing Lykken responded, “Maybe so, but normally in experiments you use something well understood to measure something messy, not the other way around.”
This means that in business we have to use multiple sources (as opposed to a single laboratory experiment). We should triangulate our findings with surveys and focus groups. We should take advantage of social media to get a feel for how people interact with our brand and what the real triggers are in the purchase cycle. We should use multiple tools to confirm that there is real causation and not just correlation.
Measuring Messy with Messy
Here lies the fundamental problem. We are trying to measure something that is intrinsically messy with tools that are even messier. I have watched businesses do the equivalent of claiming nanosecond timing of neutrinos. They come to a decision based on statistically valid test results, estimating an earth-shaking payoff for changing something. After making the changes, however (eg. adopting some approach to social media, changing the checkout flow, etc.), the payoff never seems to materialize in the bottom line.
In many ways the physicists have it easier. People behave in a less reliable way than neutrinos. It’s not impossible to get actionable data when analyzing people’s behaviors. But we do have to remember that we’re dealing with people, and the numbers don’t always do a good job of representing what they are thinking or how we can best help them. The bigger the finding, the more cautious we have to be in confirming it.
Originally posted at http://www.cameronalverson.com/2011/10/neutrinos-light-speed-and-business.html