Thoughts, Tips and Tricks on what I'm currently do for a living. Currently most of my spare time is spent on contributing to Akka.NET.

Friday, January 10, 2014

Kafka 0.8.0 on Windows

Getting Kafka 0.8.0 running on Windows isn’t straight forward if you follow the instructions. They are somewhat misleading, and the bat files are old. But with correct instructions and updated bat files it’s easy and can be done under 10 minutes. Some say you need Cygwin in order to run Kafka. This is not true. Only Server JRE is required.

Step 0. Prerequisite – Java SE Server JRE

You need Java SE Server JRE in order to run Kafka. If you have JDK installed, you already have Server JRE installed.

  1. Download Java SE Server JRE
    http://www.oracle.com/technetwork/java/javase/downloads/index.html
    For me Chrome changed the extension. If that happens change it back to .tar.gz in order to unpack it.
  2. Unpack it to a folder, for example c:\JreServer
    Update the system environment variable PATH to include C:\JreServer\jre\bin (Control Panel and search for environment variable).

Step 1. Download Kafka

  1. Download the binaries for Kafka from http://kafka.apache.org/downloads.html
  2. Unzip to a folder, for example c:\kafka
Update the bat files

Unfortunately the bat files for Kafka 0.8.0 are full of errors, so in order to start Zookeeper and Kafka they must be replaced.

  1. Download updated windows bat files from https://github.com/HCanber/kafka/releases
  2. Unzip and copy them into c:\kafka\bin\windows (overwrite the files already there)
Update Config

The config files need to be updated.

  1. Open config\server.properties and locate log.dirs=/tmp/kafka-logs. If you keep the default it will result in an error later on. Set it to a full path without . or .. in it and with forward slashes. Example: 
    log.dirs=c:/kafka/kafka-logs
  2. This step is optional but you might want to set the data directory for Zookeeper as well. Open config\zookeeper.properties and locate dataDir=/tmp/zookeeper. Example: 
    dataDir=c:/kafka/zookeeper-data

Step 2. Start the server

Note! Before executing the commands below you need to change directory to the folder where you unzipped Kafka: 

cd c:\kafka
  1. Start Zookeeper in a Command Prompt or PowerShell Console. 
    .\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
  2. Start Kafka in another Command Prompt or PowerShell Console. 
    .\bin\windows\kafka-server-start.bat .\config\server.properties

Step 3. Create a topic

  1. Create a topic. 
    .\bin\windows\kafka-create-topic.bat --zookeeper localhost:2181 --replica 1 --partition 1 --topic test
  2. List topics. 
    .\bin\windows\kafka-list-topic.bat --zookeeper localhost:2181

Step 4. Send some messages

  1. Start Console Producer 
    .\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test
  2. Write some messages 
    This is a message 
This is another message

Step 5. Start a consumer

  1. Start Console Consumer 
    .\bin\windows\kafka-console-consumer.bat --zookeeper localhost:2181 --topic test --from-beginning

 

Building from sources

See Building Kafka 0.8.0 from sources on Windows

Posted by at
Labels:

8 comments:

  1. hbagchiApril 11, 2014 at 5:26 AM

    In Kafka 0.8.1, "log.dirs" should be set with backward slash(\) and not forward slash(/)

    ReplyDelete
  2. MosheApril 28, 2014 at 2:34 PM

    I've used double back slashes inorder for the configuration to work.

    ReplyDelete
  3. AnonymousMay 21, 2014 at 5:16 PM

    Thank you SO MUCH for posting these docs! We have a shop where some people run Macs and some run Windows, and for us Windows users, this was SUPER-HELPFUL! It even pointed out some inconsistencies in our local setups, which we got to resolve :)

    Thanks again!

    ReplyDelete
  4. AnonymousJuly 5, 2014 at 12:12 AM

    Thanks for your helpful manual! It's the only one that actually works. With kafka 0.8.1.1 you only need some minor changes:

    - use kafka-topics.bat instead of
    kafka-create/list-topic.bat
    - step 3 must be:.\bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test and then .\bin\windows\kafka-topics.bat --list --zookeeper localhost:2181

    ReplyDelete
  5. BlogsJuly 24, 2014 at 7:47 AM

    At Step 3. some changes are needed in the script as below

    In kafka-create-topic
    %~dp0kafka-run-class.bat kafka.admin.TopicCommand --replication-factor 1 --create %*

    ReplyDelete
  6. BlogsJuly 24, 2014 at 7:48 AM

    In kafka-create-topic some changes are needed as below

    %~dp0kafka-run-class.bat kafka.admin.TopicCommand --create --replication-factor 1 %*

    ReplyDelete
  7. AnonymousOctober 10, 2014 at 11:32 AM

    Thank you. This is quite helpful for me.

    Why don't you make pull request to Kafka master? It is so helpful, please consider.

    Thanks.

    ReplyDelete
  8. UnknownJanuary 7, 2016 at 11:48 AM

    You can find the step by step tutorial here. www.codeproject.com/Articles/1068998/Running-Apache-Kafka-on-Windows-without-Cygwin

    ReplyDelete

Subscribe to: Post Comments (Atom)