Thoughts, Tips and Tricks on what I'm currently do for a living. Currently most of my spare time is spent on contributing to Akka.NET.

Friday, January 10, 2014

Kafka 0.8.0 on Windows

Getting Kafka 0.8.0 running on Windows isn’t straight forward if you follow the instructions. They are somewhat misleading, and the bat files are old. But with correct instructions and updated bat files it’s easy and can be done under 10 minutes. Some say you need Cygwin in order to run Kafka. This is not true. Only Server JRE is required.

Step 0. Prerequisite – Java SE Server JRE

You need Java SE Server JRE in order to run Kafka. If you have JDK installed, you already have Server JRE installed.

  1. Download Java SE Server JRE
    http://www.oracle.com/technetwork/java/javase/downloads/index.html
    For me Chrome changed the extension. If that happens change it back to .tar.gz in order to unpack it.
  2. Unpack it to a folder, for example c:\JreServer
    Update the system environment variable PATH to include C:\JreServer\jre\bin (Control Panel and search for environment variable).

Step 1. Download Kafka

  1. Download the binaries for Kafka from http://kafka.apache.org/downloads.html
  2. Unzip to a folder, for example c:\kafka
Update the bat files

Unfortunately the bat files for Kafka 0.8.0 are full of errors, so in order to start Zookeeper and Kafka they must be replaced.

  1. Download updated windows bat files from https://github.com/HCanber/kafka/releases
  2. Unzip and copy them into c:\kafka\bin\windows (overwrite the files already there)
Update Config

The config files need to be updated.

  1. Open config\server.properties and locate log.dirs=/tmp/kafka-logs. If you keep the default it will result in an error later on. Set it to a full path without . or .. in it and with forward slashes. Example:
    log.dirs=c:/kafka/kafka-logs
  2. This step is optional but you might want to set the data directory for Zookeeper as well. Open config\zookeeper.properties and locate dataDir=/tmp/zookeeper. Example:
    dataDir=c:/kafka/zookeeper-data

Step 2. Start the server

Note! Before executing the commands below you need to change directory to the folder where you unzipped Kafka:

cd c:\kafka
  1. Start Zookeeper in a Command Prompt or PowerShell Console.
    .\bin\windows\zookeeper-server-start.bat .\config\zookeeper.properties
  2. Start Kafka in another Command Prompt or PowerShell Console.
    .\bin\windows\kafka-server-start.bat .\config\server.properties

Step 3. Create a topic

  1. Create a topic.
    .\bin\windows\kafka-create-topic.bat --zookeeper localhost:2181 --replica 1 --partition 1 --topic test
  2. List topics.
    .\bin\windows\kafka-list-topic.bat --zookeeper localhost:2181

Step 4. Send some messages

  1. Start Console Producer
    .\bin\windows\kafka-console-producer.bat --broker-list localhost:9092 --topic test
  2. Write some messages
    This is a message 
    This is another message

Step 5. Start a consumer

  1. Start Console Consumer
    .\bin\windows\kafka-console-consumer.bat --zookeeper localhost:2181 --topic test --from-beginning

 

Building from sources

See Building Kafka 0.8.0 from sources on Windows

8 comments:

  1. In Kafka 0.8.1, "log.dirs" should be set with backward slash(\) and not forward slash(/)

    ReplyDelete
  2. I've used double back slashes inorder for the configuration to work.

    ReplyDelete
  3. Thank you SO MUCH for posting these docs! We have a shop where some people run Macs and some run Windows, and for us Windows users, this was SUPER-HELPFUL! It even pointed out some inconsistencies in our local setups, which we got to resolve :)

    Thanks again!

    ReplyDelete
  4. Thanks for your helpful manual! It's the only one that actually works. With kafka 0.8.1.1 you only need some minor changes:

    - use kafka-topics.bat instead of
    kafka-create/list-topic.bat
    - step 3 must be:.\bin\windows\kafka-topics.bat --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test and then .\bin\windows\kafka-topics.bat --list --zookeeper localhost:2181

    ReplyDelete
  5. At Step 3. some changes are needed in the script as below

    In kafka-create-topic
    %~dp0kafka-run-class.bat kafka.admin.TopicCommand --replication-factor 1 --create %*

    ReplyDelete
  6. In kafka-create-topic some changes are needed as below

    %~dp0kafka-run-class.bat kafka.admin.TopicCommand --create --replication-factor 1 %*

    ReplyDelete
  7. Thank you. This is quite helpful for me.

    Why don't you make pull request to Kafka master? It is so helpful, please consider.

    Thanks.

    ReplyDelete
  8. You can find the step by step tutorial here. www.codeproject.com/Articles/1068998/Running-Apache-Kafka-on-Windows-without-Cygwin

    ReplyDelete