Shell Source Connector

The Shell source connector runs shell commands periodically and produces the results of these to Kafka. The output of the command is read line by line with each line being ingested into Kafka as a separate message

Quick Start - Reading a file.


  • Confluent Platform <> is installed and services are running. This quick start assumes that you are using cp-demo <>, but standalone installations are also supported.

In this demo we’ll use the Shell Source connector to read lines from a file into Kafka.

Create the Kafka topic

Before we can read the file we need to create a topic to hold the file data. From inside a cp-demo broker container (docker-compose exec kafka1 bash):

  1. Create topics:

    kafka-topics --zookeeper zookeeper:2181 --topic --create --replication-factor 1 --partitions 1

Create a test file and a script to read it

From inside a cp-demo Connect container (docker-compose exec connect bash):

  1. Create a test file:

    cat /proc/cpuinfo > /etc/kafka/secrets/cpuinfo
  2. Create a test script:

    echo "cd /etc/kafka/secrets" > /etc/kafka/secrets/
    echo "if [ -f cpuinfo ]; then" >> /etc/kafka/secrets/
    echo "cat cpuinfo" >> /etc/kafka/secrets/
    echo "mv cpuinfo cpuinfo.processed" >> /etc/kafka/secrets/
    echo "fi" >> /etc/kafka/secrets/
    chmod +x /etc/kafka/secrets/

Load the Shell Source Connector

Now we submit the Shell Source connector to the cp-demo connect instance this will load our test file into Kafka:

  1. From outside the container in the cp-demo root directory:

    curl -X POST -H "Content-Type: application/json" \
    --cert scripts/security/connect.certificate.pem \
    --key scripts/security/connect.key \
    --tlsv1.2 \
    --cacert scripts/security/snakeoil-ca-1.crt \
    --data '{ \
    "name": "shell-source", \
    "config": {"connector.class":"", \
            "tasks.max":"1", \
            "shell.command":"sh /etc/kafka/secrets/", \
            "": "3000", \
            "topic":"" \
            }}' \

Confirm the results

Confirm messages on the topic from inside a cp-demo Broker container (docker-compose exec kafka1 bash):

kafka-console-consumer --bootstrap-server localhost:10091 --topic --from-beginning


The special strings ${key}, ${topic} and ${value} can be used in the shell.commandproperty to inject message data into the command run.