Join the DZone community and get the full member experience.Join For Free
Hortonworks Sandbox for HDP and HDF is your chance to get started on learning, developing, testing and trying out new features. Each download comes preconfigured with interactive tutorials, sample data and developments from the Apache community.
When watching the TV news, or reading newspaper commentary, I am frequently amazed at the attempts people make to interpret random noise.
For example, the latest tiny fluctuation in the share price of a major company is attributed to the CEO being ill. When the exchange rate goes up, the TV finance commentator confidently announces that it is a reaction to Chinese building contracts. No one ever says “The unemployment rate has dropped by 0.1% for no apparent reason.”
What is going on here is that the commentators are assuming we live in a noise-free world. They imagine that everything is explicable, you just have to find the explanation. However, the world is noisy — real data are subject to random fluctuations, and are often also measured inaccurately. So to interpret every little fluctuation is silly and misleading.
The finance news
Every night on the nightly TV news bulletins, a supposed expert will go through the changes in share prices, stock prices indexes, currency rates, and economic indicators, from the past 24 hours. Have these guys never heard of the efficient market hypothesis? The daily fluctuations in these time series are guaranteed to be close to white noise. So unless the change is much larger than normal, it is not worth reporting. (Or if it must be reported, than it should not be interpreted.)
A good rule-of-thumb would be that the change should not be interpreted unless it is at least in magnitude, where is the 99th percentile of all changes in that time series in the last 12 months. That way, we would only get attempts to explain the fluctuations 3–4 times per year.
Sadly, that’s unlikely to happen. Investors don’t like to think that their fortune is largely governed by randomness. I suspect that they get comfort in hearing bogus explanations of random fluctuations, because then they feel better about what is happening to their money. It also gives an illusion of potential control — if only I had known x, I could have made a different decision and made more money. People seem to like to think that the world is more controllable and less random than it really is.
Seasonally adjusted data
Seasonal adjustment of data usually assumes the following model
is the original data at time , is a smooth trend component, is a seasonal component and is the random error. (Sometimes an additive version is used instead.) There are some well-tested algorithms for estimating and from a set of data. The Australian Bureau of Statistics (ABS) primarily uses the X-12-ARIMA algorithm.
When the ABS releases an important time series, they will normally report both the trend value and the seasonally adjusted value . For example, here is the February 2014 release of the labour force participation rate. But the media tend to only report the seasonally adjusted value which is, of course, subject to much more noise than the trend estimate . Consequently, focusing on little fluctuations in is likely to be misleading. Unfortunately, the ABS encourages this mis-representation by focusing on the seasonally-adjusted value rather than the trend value in the media release. It is only those who bother to read the longer release who will get the more important information.
There are two simple solutions to this problem:
- Report the trend figure instead. It is far less volatile and more likely to reflect what is really happening with unemployment.
- Only report changes in seasonally adjusted data when they are significant. The ABS helpfully provides a 95% confidence interval for the change in , but that seems to be ignored.
However, that would mean that media outlets would have to be responsible, and not fill nightly news bulletins with meaningless interpretations of random fluctuations. It would also mean that politicians would have to be responsible, and not over-hype tiny increases or tiny decreases in the seasonally adjusted data. Unfortunately, that’s unlikely to happen any time soon.
Published at DZone with permission of Rob J Hyndman , DZone MVB. See the original article here.
Opinions expressed by DZone contributors are their own.