DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

How does AI transform chaos engineering from an experiment into a critical capability? Learn how to effectively operationalize the chaos.

Data quality isn't just a technical issue: It impacts an organization's compliance, operational efficiency, and customer satisfaction.

Are you a front-end or full-stack developer frustrated by front-end distractions? Learn to move forward with tooling and clear boundaries.

Developer Experience: Demand to support engineering teams has risen, and there is a shift from traditional DevOps to workflow improvements.

Related

  • Control Your Services With OTEL, Jaeger, and Prometheus
  • Minimizing Latency in Kafka Streaming Applications That Use External API or Database Calls
  • Keeping Two Multi-Master Databases Aligned With a Vector Clock
  • Effortless Concurrency: Leveraging the Actor Model in Financial Transaction Systems

Trending

  • Supercharge Your Java Apps With AI: A Practical Tutorial
  • OTel Me Why: The Case for OpenTelemetry Beyond the Shine
  • Kung Fu Code: Master Shifu Teaches Strategy Pattern to Po – The Functional Way
  • Operationalizing Data Quality in Cloud ETL Workflows: Automated Validation and Anomaly Detection
  1. DZone
  2. Data Engineering
  3. Big Data
  4. HDP 3.1 Released! All The Kafka!

HDP 3.1 Released! All The Kafka!

A major upgrade to Hadoop distribution has been released. Read on to learn how to upgrade to it.

By 
Tim Spann user avatar
Tim Spann
DZone Core CORE ·
Dec. 18, 18 · Tutorial
Likes (7)
Comment
Save
Tweet
Share
17.2K Views

Join the DZone community and get the full member experience.

Join For Free

The upgrade documents to use can be found here.

I was able to quickly and easily upgrad from HDP 3.0.1 to HDP 3.1. This is the stable release you were looking for. This is the new Dockerized platform you need.

Image title

yum upgrade ambari-agent -y
yum upgrade ambari-server -y

New HDP Features

HDP 3.1, Ambari 2.7.3, and SmartSense 1.5.1

Image title

Apache Kafka

Kafka Streams Now Supported

Kafka Streams is now officially supported. Kafka Streams is fully integrated with platform services like Ranger and Schema Registry and supports secure/unsecure clusters.

Security

  • Simplified access control management with support for prefixed ACLs for bulk management of entities. Access control for topic creation for specific topics/topic prefix.
  • Hostname verification to prevent SSL configuration man in the middle attacks.
  • Improved Encryption support with faster TLS and CRC32C implementations. Over-the-wire encryption is faster.
  • Simpler security configuration with SSL trust stores update without broker restart and security for Zookeeper listeners can be configured before starting brokers.

Reliability

  • Quota limit notifications to distinguish network errors from quota limits reached.
  • Better broker resiliency by reducing memory footprint of message down conversion.
  • Replication protocol improvements for fixing log divergence during fast leader failover.

Performance

  • Prevent indefinite consumer block with new configuration options.
  • Windowed aggregation performance in Kafka streams vastly improved.

Apache Atlas

  • Support for icons in lineage graph by type of entity (for example, Apache Hive tables, Apache Kafka topics, etc.).
  • Filtering of lineage UI to exclude deleted entities and excluding process entities as an option.
  • Support for expanding lineage graph in UI to different number of hops (3, 6, 9, 12, etc.).
  • Various performance and stabilization fixes.

Apache Ranger

  • Support Apache Kafka 2.0 in Kafka Ranger Plugin.
  • Support 'DelegationToken' resource type with 'Describ' operation for Kafka.
  • Support for cluster resource type for Kafka.
  • Support for Create operation with Topic resource in Kafka.

Apache Hive

Features

  • Read and write Apache Kafka topics via Kafka Storage Handler.
  • Accelerating joins between Druid table and Hive table by runtime filtering of Druid tables using bloom filters constructed on Hive tables.

Performance

  • Exploiting constraints to generate efficient query plan.
  • Improvements in stat system to generate better query plans.
  • Improvements in Map join vectorization and filter expressions.

Stability

  • JDBCStorageHandler improvements for MySQL and Postgres. 

Ambari 2.7.3 and SmartSense 1.5.1

The latest maintenance release of our open source management tool for provisioning, managing, and monitoring HDP contains a number of fixes. For more details on the specific fixes, please see the Ambari Release Notes. We encourage current users of Ambari 2.7 to upgrade to 2.7.3 to take advantage of these fixes.

SmartSense 1.5.1 is the latest maintenance release focused on bug fixes and diagnostics capture improvements. For more information about the fixes introduced with this release, please see the SmartSense Release Notes.

Open JDK8 Support

Image title

Official Apache component versions for HDP 3.1.0:

  • Apache Accumulo 1.7.0
  • Apache Atlas 1.1.0
  • Apache Calcite 1.16.0
  • Apache DataFu 1.3.0
  • Apache Hadoop 3.1.1
  • Apache HBase 2.0.2
  • Apache Hive 3.1.0
  • Apache Kafka 2.0.0
  • Apache Knox 1.0.0
  • Apache Livy 0.5.0
  • Apache Oozie 4.3.1
  • Apache Phoenix 5.0.0
  • Apache Pig 0.16.0
  • Apache Ranger 1.2.0
  • Apache Spark 2.3.2
  • Apache Sqoop 1.4.7
  • Apache Storm 1.2.1
  • Apache TEZ 0.9.1
  • Apache Zeppelin 0.8.0
  • Apache ZooKeeper 3.4.6
  • Apache Superset 0.23.3
  •  Apache Druid 0.12.1

Image title

From improved UIs to faster performance to Dockerization to more features, you hav to try HDP 3.1 now.

Using Cloudbreak you can spin one up in your cloud of choice in minutes as a full web click experience. Start ingesting, running, querying and apply machine learning to data in real-time at massive scale. All of this is open source and available for you to use now. 

kafka Hierarchical Dirichlet process Database

Opinions expressed by DZone contributors are their own.

Related

  • Control Your Services With OTEL, Jaeger, and Prometheus
  • Minimizing Latency in Kafka Streaming Applications That Use External API or Database Calls
  • Keeping Two Multi-Master Databases Aligned With a Vector Clock
  • Effortless Concurrency: Leveraging the Actor Model in Financial Transaction Systems

Partner Resources

×

Comments

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • [email protected]

Let's be friends: