DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Refcards Trend Reports Events Over 2 million developers have joined DZone. Join Today! Thanks for visiting DZone today,
Edit Profile Manage Email Subscriptions Moderation Admin Console How to Post to DZone Article Submission Guidelines
View Profile
Sign Out
Refcards
Trend Reports
Events
Zones
Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones AWS Cloud
by AWS Developer Relations
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Partner Zones
AWS Cloud
by AWS Developer Relations
Building Scalable Real-Time Apps with AstraDB and Vaadin
Register Now

Trending

  • Observability Architecture: Financial Payments Introduction
  • Extending Java APIs: Add Missing Features Without the Hassle
  • Effective Java Collection Framework: Best Practices and Tips
  • Writing a Vector Database in a Week in Rust

Trending

  • Observability Architecture: Financial Payments Introduction
  • Extending Java APIs: Add Missing Features Without the Hassle
  • Effective Java Collection Framework: Best Practices and Tips
  • Writing a Vector Database in a Week in Rust
  1. DZone
  2. Coding
  3. Tools
  4. Installing and Running Kafka on an AWS Instance

Installing and Running Kafka on an AWS Instance

This tutorial will help you install and start the latest version of Kafka on the EC2 Linux instance, including starting a Zookeeper service.

Sahil Sawhney user avatar by
Sahil Sawhney
·
Updated May. 13, 17 · Tutorial
Like (4)
Save
Tweet
Share
40.31K Views

Join the DZone community and get the full member experience.

Join For Free

In this blog, we will install and start a single-node, latest and recommended version of Kafka 0.10.2.0 with the binary for Scala 2.12 on the EC2 Linux instance with centOS as its operating system. We will be using the t2.micro (free tier) instance which comes with 1 GB RAM and 8 GB SSD.

Prerequisites

1) Create an EC2 Instance

Steps for creating an AWS instance are clearly mentioned in the official AWS documentation; check here.

2) Install Java 8

Since we will be working with the Kafka binary for Scala 2.12, our instance must have Java 8. By default, the EC2 instances have Java 7. You may check and upgrade the Java version to 8 on your instance by following the steps here.

After installing Java 8, follow the steps mentioned below in sequence to start Kafka on your instance.

Step 1: Downloading and Extracting Kafka

Download kafka_2.12-0.10.2.0.tgz

wget http://mirror.fibergrid.in/apache/kafka/0.10.2.0/kafka_2.12-0.10.2.0.tgz

Extract the .tgz file

tar -xzf kafka_2.12-0.10.2.0.tgz

Since the zip is of no use now, we remove it:

rm kafka_2.12-0.10.2.0.tgz

Step 2: Starting Zookeeper

Since Kafka uses Zookeeper, we need to first start a Zookeeper server. We can use the convenience script packaged with Kafka to start a single-node Zookeeper instance or we can start Zookeeper on a standalone instance and specify its configurations in zookeeper.properties configuration file, we would be starting it using the convenience script that is packaged with Kafka. Since we have 1 GB RAM we would be setting KAFKA_HEAP_OPTS environment variable in our .bashrc to 50% of total RAM ie 500 MB in our case.

vi .bashrc

Insert the following environment variable.

export KAFKA_HEAP_OPTS="-Xmx500M -Xms500M"

After setting the variable, source your .baschrc.

source .bashrc

Start Zookeeper by the following command in background using nohup and divert its logs in zookeeper-logs file.

cd kafka_2.12-0.10.2.0
nohup bin/zookeeper-server-start.sh config/zookeeper.properties > ~/zookeeper-logs &

Then press ctrl+d to log out of the instance.

Ssh to your instance again and check the content of zookeeper-logs file. It must look like this:

zookeper_success

NOTE: In case the content of zookeeper-logs file is different, try freeing the RAM buffers/cache by following commands and re-run the cd and nohup command mentioned above (this may happen as t2.micro instance comes with quite less RAM, unlikely to happen with bigger instances).

sudo sh -c 'echo 1 >/proc/sys/vm/drop_caches'
sudo sh -c 'echo 2 >/proc/sys/vm/drop_caches'
sudo sh -c 'echo 3 >/proc/sys/vm/drop_caches'

Step 3: Starting Kafka

After successfully staring Zookeeper, it's now time to start Kafka via the following command:

cd kafka_2.12-0.10.2.0
nohup bin/kafka-server-start.sh config/server.properties > ~/kafka-logs &

Then press ctrl+d to log out of the instance.

Ssh to your instance again and check the content of Kafka-logs file. It must look like this:

kafka_success.png
This successfully starts Kafka on your EC2 instance. You may access your Kafka-server via Kafka-Scala or Kafka-Java API by making the required changes in the security groups. To stop Kafka and Zookeeper, enter the following commands:

bin/kafka-server-stop.sh
bin/zookeeper-server-stop.sh

Hope this blog helps you. Comments and suggestions are welcomed.

AWS kafka

Published at DZone with permission of Sahil Sawhney, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Trending

  • Observability Architecture: Financial Payments Introduction
  • Extending Java APIs: Add Missing Features Without the Hassle
  • Effective Java Collection Framework: Best Practices and Tips
  • Writing a Vector Database in a Week in Rust

Comments

Partner Resources

X

ABOUT US

  • About DZone
  • Send feedback
  • Careers
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 600 Park Offices Drive
  • Suite 300
  • Durham, NC 27709
  • support@dzone.com

Let's be friends: