DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
  1. DZone
  2. Data Engineering
  3. Big Data
  4. Better Predictions With Big Data

Better Predictions With Big Data

Things have been very unpredictable lately, to say the least. How can we fix this?

Adi Gaskell user avatar by
Adi Gaskell
·
Jan. 09, 17 · Opinion
Like (4)
Save
Tweet
Share
2.11K Views

Join the DZone community and get the full member experience.

Join For Free

Recent times have seen our predictive capabilities take a bit of a battering.  Numerous political polls have gotten events ranging from Brexit to the Trump election massively wrong, with senior political figures casting doubts on the ability of ‘experts’ as a result.

Alas, researchers from Columbia, Harvard and Princeton have recently devised a method that they believe will make us better able to make accurate predictions in areas from healthcare to politics.

The approach, which was documented in a recently published paper, aims to build upon previous work by the team that highlighted how certain variables, whilst appearing significant are not particularly useful for making predictions, whilst those that appear insignificant can be very important.

Finding the Key Variables

These early studies raised the question of just what makes a variable useful when forming predictions?  Traditional methods have tried to assign significance to a variable, before then putting them into models.

To provide a more robust approach, the researchers propose a new metric known as the influence score, which will be solely looking at the ability of the variable to predict outcomes.  It’s an approach that, when tested, was found to be reliable in distinguishing between noisy and predictive variables, thus improving the prediction rates quite significantly.  Indeed, in one test the prediction rates for breast cancer leapt from 70% to 92%.  It’s an approach the researchers are confident can be applied to various fields with similar outcomes.

“The practical implications are what drove the project, so they’re quite broad,” they say. “Essentially anytime you might be interested in predicting and identifying highly predictive variables, you might have something to gain by conducting variable selection through a statistic like the I-score, which is related to variable predictivity. That the I-score fares especially well in high dimensional data and with many complex interactions between variables is an extra boon for the researcher or policy expert interested in predicting something with large dimensional data.”

Would it make us any better at predicting election results?  Time will tell I suppose.

Big data

Published at DZone with permission of Adi Gaskell, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • Choosing the Best Cloud Provider for Hosting DevOps Tools
  • How Do the Docker Client and Docker Servers Work?
  • Simulate Network Latency and Packet Drop In Linux
  • Spring Boot Docker Best Practices

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: