Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Kinesis source doesn't pass the given polling retrieval config to underlying KCL #5269

Open
RashmiRam opened this issue Dec 17, 2024 · 1 comment · May be fixed by #5272
Open

[BUG] Kinesis source doesn't pass the given polling retrieval config to underlying KCL #5269

RashmiRam opened this issue Dec 17, 2024 · 1 comment · May be fixed by #5272
Labels
bug Something isn't working

Comments

@RashmiRam
Copy link

Describe the bug
Documentation of Data prepper says that consumer_strategy can be given either as polling or fan-out in Kinesis source config. If polling is given, polling config must be given too. Even after polling config is given and polling consumer_strategy is set in config, Kinesis Fanout Publisher is only initialised.

To Reproduce
Steps to reproduce the behavior:

  source:
    kinesis:
      streams:
        - stream_name: "test_stream"
          initial_position: LATEST
          compression: gzip
      codec:
        ndjson:
      consumer_strategy: "polling"
      records_to_accumulate: 100
      polling:
        max_polling_records: 10000
        idle_time_between_reads: 1s

Expected behavior
Kinesis source to make use of polling retrieval strategy .

Additional context
Passing of newly created retrieval config in KCL scheduler instantiation here causes this issue.

@RashmiRam
Copy link
Author

I will raise a PR for the same shortly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
2 participants