Skip to content
This repository has been archived by the owner on Jan 9, 2020. It is now read-only.

Latest commit

 

History

History
42 lines (31 loc) · 1.76 KB

README.md

File metadata and controls

42 lines (31 loc) · 1.76 KB
Raw

AvroRepoKafkaProducerTest

Basic kafka producer to send Avro Messages with an Avro schema repository.

Working mode:

  1. Producer construct the avro message
  2. Encode the message using a test sqchema included in folder resources
  3. Fill the content of the message with some basic information.
  4. Encode the message (using com.linkedin.camus.etl.kafka.coders.KafkaAvroMessageEncoder class)
    • KafkaAvroMessageEncoder communicates with an avro schemas repository.
    • Avro repo send an ID to our test program, the message is stored in Kafka with the ID in the Header and the bytes payload.

Avro Schema Repo: https://github.com/mvalleavila/avro-1.7.4-schema-repo

  • Stores the schemas, if an schema is not stored sends a new ID, if the schema exists sen the previously assigned ID.
  • Then the consumer "asks" for the schema with a determinated ID.

Camus: https://github.com/mvalleavila/camus

  • Camus is a connection between Kafka and HDFS. So is a Kafka Consumer, in this example a camus class it's usedto encode the message, and "talk" with the Avro Schema Repository

=================================

Compilation

mvn clean package

==================================

Use

  • Edit resources/config.properties with the rights IP and Ports where you are running kafka brokers and the schema repo.
     metadata.broker.list=hadoop-manager:9092   
     etl.schema.registry.url=http://hadoop-manager:2876/schema-repo

  • To select what Avro message will be sent edit avro.schema.file=resources/testSchema1.avsc, three different files are included for three tests.

  • Execute run.sh
    The ID of the message received is printed in console.
    You can test sending the same message and check the same ID is being received.