CloudKarafka + Spark Setup:

Nirav Langaliya
2 min readApr 28, 2020

cloudkarafka provides a free shared plan for development and testing. you can create an account on cloudkafka and start your development or testing quickly.

Configure Kafka instance ( For Additional Information, Please refer )

Once it is created, you can have connection credentials.

check the version of Scala


Invoke the Spark Shell and provide the correct package.

spark-shell --packages org.apache.spark:spark-sql-kafka-0-10_2.11:2.4.5,org.apache.spark:spark-avro_2.11:2.4.5

Read Kafka Topic via Spark
Publish Data to Kafka

Once you publish data on the Kafka topic, Kafka consumer at spark will start display data on the console.