Confluent Kafka Sink connector for MySQL

##Download Confluent Community version.
https://www.confluent.io/confirmation/

## Install Confluent Kafka Local. (Untar setup file)

tar -xvf confluent-community-5.3.1–2.12.tar

##Setup Conflunet Cli.

curl -L https://cnfl.io/cli | sh -s — -b ~/confluent-5.3.1/bin

##Include Confluent bin into PATH.

export PATH=${PATH}:~/confluent-5.3.1/bin

## Start Conflunet

confluent local start

##Download MySQL Jar file place under

cp mysql-connector-java-5.1.48.jar ~/confluent-5.3.1/share/java/kafka-connect-jdbc
cp mysql-connector-java-5.1.48-bin.jar ~/confluent-5.3.1/share/java/kafka-connect-jdbc

## Create sink MySQL property file for Confluent connector

sink-quickstart-mysql.properties and place it under ~/confluent-5.3.1/etc/kafka-connect-jdbc/

name=test_mysql_sink
connector.class=io.confluent.connect.jdbc.JdbcSinkConnector
tasks.max=1
topics=orders
connection.url=jdbc:mysql://localhost:3306/demo?user=mysqluser&password=pw
auto.create=true
value.converter.schema.registry.url=http://localhost:8081
key.converter.schema.registry.url=http://localhost:8081
key.converter=io.confluent.connect.avro.AvroConverter
value.converter=io.confluent.connect.avro.AvroConverter

## Load MySQL Connector : test_mysql_sink

confluent local load test_mysql_sink -- -d sink-quickstart-mysql.properties

##Check status of Connector.

confluent local status test_mysql_sink

##Now Produce Avro Messages:

Note: Change topic name at line #33 to orders from Test_MovieData

## Logon to MySQL and verify that the Order table is created under demo schema and records are loaded.

if you get Java Error while start confluent then install java 1.8

The local commands are intended for a single-node development environment
only, NOT for production usage. https://docs.confluent.io/current/cli/index.html
Current Java version ‘12’ is unsupported at this time. Confluent CLI will exit.ERROR: The Confluent CLI requires Java version 1.8 or 1.11.
See https://docs.confluent.io/current/installation/versions-interoperability.html .
If you have multiple versions of Java installed, you may need to set JAVA_HOME to the version you want Confluent to use.

##Setup JAVA_HOME

export JAVA_HOME=$(/usr/libexec/java_home -v 1.8)

Reference:

https://www.ibm.com/support/knowledgecenter/en/SSPT3X_4.2.5/com.ibm.swg.im.infosphere.biginsights.admin.doc/doc/admin_kafka_jdbc_sink.html

Confluent Kafka Python.

https://www.confluent.io/blog/simplest-useful-kafka-connect-data-pipeline-world-thereabouts-part-1/

--

--

Lead Data Engineer@Nisum Technologies

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store