DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
  1. DZone
  2. Data Engineering
  3. Data
  4. The Big Data Challenge

The Big Data Challenge

Akshat Thakar user avatar by
Akshat Thakar
·
Apr. 22, 15 · Interview
Like (0)
Save
Tweet
Share
2.98K Views

Join the DZone community and get the full member experience.

Join For Free

Big Data analytics comprises the below steps:

  1. Data Collection - collect un-structured and structured data from variety of conventional and non conventional sources including machine sensors.
  2. Data Storage - store data in robust, distributed, scalable storage  based on commodity hardware with replication copies.
  3. Descriptive Analytics - summarize data and develop data visualization.
  4. Predictive Analytics - develop model using available data using supervised learning algorithms.
  5. Prescriptive Analytics -  develop story for leveraging predictions.


We still have not discussed skeptical view which will help us in refining Big data steps. I will discuss few challenges in leveraging  true value of Big Data. Here are some scenarios and remedies. 

  1. Lack of identification of data sources or hidden data sources: there might be few data sources hidden in data collection step. Big data does not limit number of data sources and encourages to gather all data from all available sources. As a thumb rule all data need to be collected for solving Big data problem. With this understanding, we need to ensure adequate  security for All Data approach. Multiple teams can contribute in data collection. 
  2. Data Silos can be generated in Data Storage step with concern for data security, lack of unified data   service layer and   unified data modeling. We can address this challenge with unified data model with scope to define business entities, unified service layer and security implementation in form of authentication and authorization. There is emerging concept of Data Lake which needs data to be stored based on agreed schema between producer and consumer. 
  3. Analytics had been traditionally associated with smaller data sets and was performed in OLAP mode. It will be hard to replace existing Analytics/BI tools or enhance them unless we convince stake holders about advantages of Big Data with reference to analytics - real time analytics and larger data sets parallel processing capability. Algorithms are getting ported to Big Data based software packages which is very exciting. Big Data technologies will be leveraging existing analytics platforms - R, Python, SAS and provide unified platform for analytics. Big Data talent too has to merge with analytics skill to deliver descriptive, predictive and prescriptive analytics.

Conclusion

Big Data analytics is now more towards accurately defining data, uniform handling and developing data driven smart products.

Big data Analytics

Opinions expressed by DZone contributors are their own.

Popular on DZone

  • Why It Is Important To Have an Ownership as a DevOps Engineer
  • The Role of Data Governance in Data Strategy: Part II
  • What Was the Question Again, ChatGPT?
  • The Importance of Delegation in Management Teams

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com
  • +1 (919) 678-0300

Let's be friends: