Over a million developers have joined DZone.

I predict a riot

DZone's Guide to

I predict a riot

· Big Data Zone ·
Free Resource

Learn how to operationalize machine learning and data science projects to monetize your AI initiatives. Download the Gartner report now.

Just as there are numerous organisations proclaiming to be able to predict how or when content will go viral, there appear to be an equal number of academics who believe they can predict the movement of masses of people.

Researchers from MIT for instance believe they can use Twitter to predict crowd behaviours.  They analysed tweets made about the 2013 coup in Egypt, and believe they could have predicted the unrest days in advance, purely by crunching the tweets.

They believe it is as simple as highlighting the tweets that signal the potential for unrest, such as public signals of intent or attempts to coordinate behaviour online.  It should be as simple as plucking out these event related tweets and determining any trends associated with them.

“The gathering of crowds into a single action can often be seen through trends appearing in this data far in advance,” the researcher says.

Now, of course, such analysis stands accused of simply being wise after the event, and few of these predictive tools have ever managed to actually predict things that have yet to happen.  The validity of the data mined via social media must make attributing tweets to a future course of action fraught with difficulty.  Not to mention the inherent sample bias of predicting actions based purely on those active on social media.

An example of the challenges faced comes from Duke University’s Ward Lab, who make semiannual predictions about world events.  Their effort in July of last year saw a mixed bag of results.  For instance, there was no mention of the current unrest in Ukraine.

The staff from the lab are mindful to point out that their predictions are designed more to test theories than gaze into the crystal ball.

“It’s a success only if it doesn’t come at the cost of predicting a lot of incidents that don’t occur,” says Michael D. Ward, the lab’s founder and chief investigator, who also runs the blog Predictive Heuristics. “But it suggests that we might be on the right track.”

Of course, this isn’t the only way to gain geopolitical insights.  WikiStrat is a consultancy that crowdsources geopolitical insight and expertise from its pool of experts.  This approach is more akin to the prediction markets that have been blossoming around the world.  SciCast for instance are a recent entry into this field, offering a range of science and technology related predictions, whilst CrowdMed offer a similar service for medical diagnosis.

I suspect this kind of approach is much more likely to yield results than scouring Twitter for glimpses of the future.

Original post

Bias comes in a variety of forms, all of them potentially damaging to the efficacy of your ML algorithm. Our Chief Data Scientist discusses the source of most headlines about AI failures here.


Opinions expressed by DZone contributors are their own.

{{ parent.title || parent.header.title}}

{{ parent.tldr }}

{{ parent.urlSource.name }}