Kafka running on windows

Hope you are clear about Apache Kafka introduction through my previous post. Here we will go through how we can install Apache Kafka on Windows.

STEP 1: Install JAVA 8 SDK

Make sure you installed JAVA 8 SDK on your system. You can use chocolatey ( https://chocolatey.org/ ) windows package manager for the same.

and make sure java SDK installed successfully.

STEP 2: Download and Install Apache Kafka Binaries

We will Apache Kafka binaries for installing Apache Kafka. Go to Apache Kafka official download page (https://kafka.apache.org/downloads) and download the binaries.

STEP 3: Create Data folder for Zookeeper and Apache Kafka

Create “data” folder and Kafka / Zookeeper directories inside data folder

STEP 4: Change the default configuration value

Update zookeeper data directory path in “config/zookeeper.Properties” configuration file.

Update Apache Kafka log file path in “config/server.properties” configuration file.

STEP 5: Start Zookeeper

Now time to start zookeeper from command prompt. Change your directory to bin\windows and execute zookeeper-server-start.bat command with config/zookeeper.Properties configuration file.

And make sure zookeeper started successfully

STEP 6: Start Apache Kafka

Finally time to start Apache Kafka from command prompt. Run kafka-server-start.bat command with kafka config/server.properties configuration file.

This will start our Apache Kafka successfully.

Conclusion

That is all for installing Apache Kafka on windows. Time to understand some of the Apache Kafka theory topics now. Please read Understanding Apache Kafka Topics and Partitions before we play around with Apache Kafka CLI commands.

Setting Up and Running Apache Kafka on Windows OS

In this article, we go through a step-by-step guide to installing and running Apache ZooKeeper and Apache Kafka on a Windows OS.

Join the DZone community and get the full member experience.

Introduction

In my last article, we covered setting up and using Hadoop. This article is all about configuring and starting an Apache Kafka server on a Windows OS. This guide will also provide instructions to set up Java and Apache ZooKeeper.

Apache Kafka is a fast and scalable messaging queue, capable of handling heavy loads in context of read and write, i.e. IO-related, stuff. You can find more about Kafka on http://kafka.apache.org/. Apache Kafka requires a running ZooKeeper instance, which is used for reliable distributed coordination. Please find more about ZooKeeper on https://zookeeper.apache.org/.

You can get help from this video for setting up Kafka on Windows.

Author’s GitHub:

I have created a bunch of Spark-Scala utilities at https://github.com/gopal-tiwari, might be helpful in some other cases.

Downloading the Required Files

For this tutorial, we are assuming that ZooKeeper and Kafka are unzipped in the C: drive, but you can unzip them in any location.

Here, we are using full-fledged ZooKeeper and not the one packaged with Kafka because it will be a single-node ZooKeeper instance. If you want, you can run Kafka with a packaged ZooKeeper located in a Kafka package inside the \kafka\bin\windows directory.

Installation

A. JDK Setup

1. Start the JRE installation and hit the “Change destination folder” checkbox, then click ‘Install.’

2. Change the installation directory to any path without spaces in the folder name. E.g. C:\Java\jre1.8.0_xx\. (By default it will be C:\Program Files\Java\jre1.8.0_xx), then click ‘Next.’

3. Now open the system environment variables dialogue by opening Control Panel -> System -> Advanced system settings -> Environment Variables.

4. Hit the New User Variable button in the User variables section, then type JAVA_HOME in Variable name and give your jre path in the Variable value. It should look like the below image:

(Java path and version may change according to the version of Kafka you are using)

6. Search for a Path variable in the “System Variable” section in the “Environment Variables” dialogue box you just opened.

7. Edit the path and type “;%JAVA_HOME%\bin” at the end of the text already written there, just like the image below:

8. To confirm the Java installation, just open cmd and type “java –version.” You should be able to see the version of Java you just installed.

If your command prompt somewhat looks like the image above, you are good to go. Otherwise, you need to recheck whether your setup version matches the correct OS architecture (x86, x64), or if the environment variables path is correct.

B. ZooKeeper Installation

1. Go to your ZooKeeper config directory. For me its C:\zookeeper-3.4.7\conf

2. Rename file “zoo_sample.cfg” to “zoo.cfg”

3. Open zoo.cfg in any text editor, like Notepad; I prefer Notepad++.

4. Find and edit dataDir=/tmp/zookeeper to :\zookeeper-3.4.7\data

5. Add an entry in the System Environment Variables as we did for Java.

a. Add ZOOKEEPER_HOME = C:\zookeeper-3.4.7 to the System Variables.

b. Edit the System Variable named “Path” and add ;%ZOOKEEPER_HOME%\bin;

6. You can change the default Zookeeper port in zoo.cfg file (Default port 2181).

7. Run ZooKeeper by opening a new cmd and type zkserver .

8. You will see the command prompt with some details, like the image below:

Congratulations, your ZooKeeper is up and running on port 2181!

C. Setting Up Kafka

1. Go to your Kafka config directory. For me its C:\kafka_2.11-0.9.0.0\config

2. Edit the file “server.properties.”

3. Find and edit the line log.dirs=/tmp/kafka-logs” to “log.dir= C:\kafka_2.11-0.9.0.0\kafka-logs .

4. If your ZooKeeper is running on some other machine or cluster you can edit “zookeeper.connect:2181” to your custom IP and port. For this demo, we are using the same machine so there’s no need to change. Also the Kafka port and broker.id are configurable in this file. Leave other settings as is.

5. Your Kafka will run on default port 9092 and connect to ZooKeeper’s default port, 2181.

D. Running a Kafka Server

Important: Please ensure that your ZooKeeper instance is up and running before starting a Kafka server.

1. Go to your Kafka installation directory: C:\kafka_2.11-0.9.0.0\

2. Open a command prompt here by pressing Shift + right click and choose the “Open command window here” option).

3. Now type .\bin\windows\kafka-server-start.bat .\config\server.properties and press Enter.

4. If everything went fine, your command prompt will look like this:

5. Now your Kafka Server is up and running, you can create topics to store messages. Also, we can produce or consume data from Java or Scala code or directly from the command prompt.

E. Creating Topics

1. Now create a topic with the name “test” and a replication factor of 1, as we have only one Kafka server running. If you have a cluster with more than one Kafka server running, you can increase the replication-factor accordingly, which will increase the data availability and act like a fault-tolerant system.

2. Open a new command prompt in the location C:\kafka_2.11-0.9.0.0\bin\windows.

3. Type the following command and hit Enter:

F. Creating a Producer and Consumer to Test Server

1. Open a new command prompt in the location C:\kafka_2.11-0.9.0.0\bin\windows

2. To start a producer type the following command:

3. Again open a new command prompt in the same location as C:\kafka_2.11-0.9.0.0\bin\windows

4. Now start a consumer by typing the following command:

Before kafka version 2.0 ( = 2.0):

5. Now you will have two command prompts, like the image below:

6. Now type anything in the producer command prompt and press Enter, and you should be able to see the message in the other consumer command prompt.

7. If you are able to push and see your messages on the consumer side, you are done with Kafka setup.

Some Other Useful Commands

1. List Topics: kafka-topics.bat —list —zookeeper localhost:2181
2. Describe Topic: kafka-topics.bat —describe —zookeeper localhost:2181 —topic [Topic Name]
3. Read messages from the beginning
   1. Before version kafka-console-consumer.bat —zookeeper localhost:2181 —topic [Topic Name] —from-beginning
   2. After version > 2.0: kafka-console-consumer.bat —bootstrap-server localhost:9092 —topic [Topic Name] —from-beginn
4. Delete Topic: kafka-run-class.bat kafka.admin.TopicCommand —delete —topic [topic_to_delete] —zookeeper localhost:2181

Opinions expressed by DZone contributors are their own.

Kafka running on windows

[2017-11-18 14:05:15,829] FATAL Shutdown broker because all log dirs in C:\kafka_2.11-1.0.0\kafka_2.11-1.0.0kafka-logs have failed (kafka.log.LogManager)

Also, Zookeeper has below error after I tried to bring up Kafka. Could you please help.

2017-11-18 14:05:22,116 [myid:] — INFO [SessionTracker:ZooKeeperServer@354] — Expiring session 0x1000170cf040000, timeout of 6000ms exceeded
2017-11-18 14:05:22,116 [myid:] — INFO [ProcessThread(sid cport:2181)::PrepRequestProcessor@487] — Processed session termination for sessionid: 0x1000170cf040000 Sign In· View Thread

Delete Topics

Adra@SRM

5-Mar-17 23:25

I installed kafka in windows from your instructions and is working well. I need the command line which deletes the topics in kafka from the command prompt which was already created. Sign In· View Thread

Error Broker-list

NecroJohnson

7-Sep-16 20:26

Hello When i am writing C:\kafka_2.11-0.10.0.1\bin\windows>kafka-console-consumer.bat —broker-list localhost:9092 —topic amazingtest its giving error broker-list not a recognised option Sign In· View Thread

I just made use of this

Sacha Barber

15-Mar-16 23:32

Just wanted to say thanks for this guide. I made use of this in a new article I just did : Apache Kafka 0.9 Scala Producer/Consumer With Some RxScala Pixie Dust Might be of interest to you Sign In· View Thread

hi

Member 11547475

23-Feb-16 23:43

hi thanks for your information. But when i try to run kafka with the command it says system cant find the path specified please help Sign In· View Thread

Re: hi

GopalTiwari

24-Feb-16 22:50

It seems like you have opened cmd in wrong folder, please navigate to kafka root folder in cmd. To run the command as it is you have to be in your kafka’s root directory. Sign In· View Thread

Re: hi

Member 11547475

25-Feb-16 3:07

Thanks for the reply my friend. Yes i have follow the tutorial but still i cant be able to run the kafka with the version you specify. Sign In· View Thread

Re: hi

GopalTiwari

25-Feb-16 5:30

Oh.. Can you please post the screenshot of your command prompt if possible. Sign In· View Thread

Re: hi

Member 11547475

26-Feb-16 9:49

thanks for your response. I will send the screenshot to you soon. Sign In· View Thread

Re: hi

Member 11547475

26-Feb-16 11:58

I could not find any option to send screenshot here. If you can tell me your email i can send to screenshot there. I have follow the direction you specify my zookeeper running well. But when i try to run kafka it says «system can not find the path specified» it that related to some classpath settings or something else Do i have to do the environment settings like you shown for zookeeper. Please let me know. Sign In· View Thread

Message Closed

GopalTiwari

27-Feb-16 9:05

Re: hi

Member 11547475

28-Feb-16 0:58

Thanks a lot for your prompt reply. I was able to find the solution and now kafka runs. It is just needs to make changes in batch file and also folder name needs to changes as well. «kafka_2.11-0.9.0.0» in place of this folder, i have replaced with «kafka» folder and seems working now. But really appreciate the efforts put forward by you. Sign In· View Thread

Re: hi

GopalTiwari

28-Feb-16 1:08

Sounds great. Happy kafka. Sign In· View Thread

Re: hi

Member 11547475

2-Mar-16 9:35

Thanks for your assistance. Now have some other issues i cant be able to run kafka-console-producer.bat —broker-list localhost:9092 —topic test If i try to run above command it says it could not find or load main class kakfa.producer.ConsoleProducer. Sign In· View Thread

Re: hi

Member 11547475

14-Mar-16 14:03

I have one question. If you have any idea about how to injest log into hadoop through kafka. If you have some idea, please share with me. Sign In· View Thread

Java edition?

Member 12332048

17-Feb-16 11:02

Your instructions say to install Java Server SE, which has the JDK, but your link just goes to the download page for Java SE. You are showing an installation wizard for Java, but the Java Server SE does not have this. The installation instructions for Java Server SE just say to unzip the files in the directory where you want to install. When running ‘java -version’ I get the following output, that is slightly different in the last line: java version «1.8.0_73» Java(TM) SE Runtime Environment (build 1.8.0_73-b02) Java HotSpot(TM) 64-Bit Server VM (build 25.73-b02, mixed mode) Note that the last line says ‘Server VM’, and yours says ‘Client VM’. Sign In· View Thread

kafka-python on Windows?

Member 12332048

16-Feb-16 14:09

I followed these instructions and kafka works wonderfully on Windows 7. But now I want to use it with kafka-python. Do you have any experience with this? I tried it, and it cannot even connect. Thanks in advance. Sign In· View Thread

This is great

Sacha Barber

11-Feb-16 20:01

I have previously use VirtualBox and Vagrant for giving me a Linux environment, but for testing stuff out I just prefer to use Windows, as that is where most of my dev tools are. Great stuff very useful. We seem to like the same sort of stuff, so I thought I would point you at some stuff that I wrote that you may like, small 3 part series on using Spark + Cassandra Introduction to Apache Spark Apache Spark/Cassandra 1 of 2 Apache Spark/Cassandra 2 of 2 You may find that of interest Sign In· View Thread

Re: This is great

GopalTiwari

27-Feb-16 7:07

Woh! You got me right man. I’m a windows lover & developing stuffs in spark, Cassandra, elasticsearch, kafka & kibana. I have gone through your articles, they are awesome. Sign In· View Thread

kafka in windows

bnagarajan

13-Jan-16 0:50

Thanks for the great explanation and well documented steps. Kafka is running perfectly without any single issue. Previously with old versions, I struggled a lot to setup kafka in windows. Today it was solved. Now I can do the testing in windows itself instead in VM. Sign In· View Thread

Re: kafka in windows

GopalTiwari

27-Feb-16 7:11

Its a good news for me. finally my article helped someone like me. a windows lover. Sign In· View Thread

Uber for cleaning

Member 12257127

11-Jan-16 2:56

The customer who needs a cleaning service registers his location and other details.Based on the location of the customer ,the app initiates the process. Sign In· View Thread

Last Visit: 31-Dec-99 18:00 Last Update: 18-Apr-21 18:53

Refresh

1

General News Suggestion Question Bug Answer Joke Praise Rant Admin

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.