Kafka based persistent stream provider

[yoniabrahamy]

Hello all,
This is not really an issue, I just wanted to write in this forum that I have implemented a persistent stream provider based on Kafka. You can check it out here: https://github.com/gigya/Orleans.KafkaStreamProvider

I would love to get some feedbacks about it, we are already using it in a system on production environment and it seems to be working pretty great.

If you have any questions I'll try to write some answers in the following week (although I'm on a Christmas holiday, so I will do my best 😅).

Happy holidays!

[jthelin]

Very nice! Thanks for posting this @yoniabrahamy

I will leave @gabikliot, @jason-bragg and other Orleans streaming experts to provide more detailed comments.

[gabikliot]

Looks great @yoniabrahamy !
This is basically an implementation of what was discussed in #756 and #990.

Do you guys prefer to keep it in your Gigya repository, or maybe move to https://github.com/OrleansContrib? This is where we plan to have all the non-core Orleans contbution projects. It will help others discover them. Totally up to you.
I think we can ask @richorama to open a repo there for you, with full admin rights for you (and only you and whom ever you choose to add) on that repo.
Richard, is it indeed possible? If people decide to move their repo to OrleansContrib, they of course need the full control of who can be accept PRs and be admin, for their repo.

[gabikliot]

Looking through the code, I noticed you are offloading a lot to Thread Pool - going via Task.Run. Any particular reason to do that? Look here for cases when one may want to do this, but I am not sure yours is the case.
Specifically, here Task.Run is totaly redundant.

Also, why do you need ClientGenerator and ClientGenerator/Serialization?

[randa1]

@gabikliot This is actually sort of a middle-ground solution before the implementation of #756. It doesn't allow full historic rewinding, it just retains messages in the queue cache (the TimedQueueCache class) up to a certain time guarantee (we use it to save 3 hours of messages in the cache). Once the messages expire from the cache we commit their offsets to Kafka, meaning that next time the Silo starts up the messages from the 3 hours time window will be re-loaded into the cache.

Once we see for a while how this solution fares in production we'll decide whether we want to implement the full solution (#756).

[gabikliot]

Thanks for clarifying @randa1.
@jason-bragg will be very interested to hear that.

[richorama]

hey @yoniabrahamy let me know if you'd like to add it to the contrib org, and I'll add you as an admin.

[yoniabrahamy]

@gabikliot about the whole Thread Pool offloading. Basically I sent every Kafka-bound operation to the default scheduler in order to not cause pressure on the Orleans Scheduler with these operations. I took the idea from the implementation of AzureStreamProvider and we also saw that not doing as such causes issues such as Orleans responses not getting in time.
About the line of code you referenced, that indeed does not have to go through the default scheduler since the fetch task already does. That needs to be fixed.

[gabikliot]

The discussion about Thread Pool offloading moved to: gigya/Orleans.KafkaStreamProvider#16

[randa1]

@richorama Regarding the OrleansContrib question, let's do it. Other than @yoniabrahamy, please also add @armona and myself as admins, as we'll be more actively maintaining it in the future (we're all members of the same team here at Gigya).

Sorry for the late answer but I had to verify with other people in the company that it's fine.

[richorama]

done!

[sergeybykov]

I found this present under my Christmas tree. :-) This is great!

[jason-bragg]

@yoniabrahamy
This is very cool. I've been focused on porting an EventHubStreamProvider from 343 industries, but I'm excited to see Kafka one as well.

The cache mechanism introduced in the Kafka stream provider provides rewind able streams for a period of time, which is something we needed to do with EventHub as well. The issue we ran into when developing this was that messages kept for an extended period of time all end up in Gen2 causing a serious garbage collection hit when running under load. We worked around this by keeping the cached messages in a combination of large blocks of tightly packed data structures and blocks of raw data, both of which were pooled.

If you guys encounter similar problems under load you may want to check out PooledQueueCache #1245. This cache is the cache we developed for 343 that addresses the GC hit. The port is still incomplete, but most of the guts are there.

[jason-bragg]

@yoniabrahamy and @randa1
Regarding rewinding and recovery of stream processing, you may want to check out PR "Stream Recovery Tests #1289". It covers some basic recovery scenarios using implicit streams over a persistent stream provider. This includes delivery retry logic, and stream rewinding. The test harness was intended to be general enough to be run against any persistent stream provider, so you may want to consider running the suite against your provider at some point. The test runner may need some tweaking, but it should be in better shape after using it with the event hub stream provider - coming soon.

[randa1]

@jason-bragg, I'll take a look. We want to adopt your PooledQueueCache when we have the time, so we're keeping an eye on it too. Thanks!

[amccool]

side question (related to @gabikliot earlier response): if OrleansContrib is the go to place for "add-ons" to Orleans, do we have a build process, and generating nugets?

[sergeybykov]

I don't think @richorama has set anything up there. I'm not sure if we can leverage the .NET Foundation resources for that or not. I need to ask them.

[richorama]

I use travis to build, and manually do the nugets.

There are a number of free build systems that could be used, or we could stand one up, or use the .NET one, but it's kind of down to each individual repo to choose the most appropriate, and set them up.

Happy to take a suggestion, but reluctant to have a policy.

Sent from my iPhone

On 25 Jan 2016, at 21:10, Sergey Bykov [email protected] wrote:

I don't think @richorama has set anything up there. I'm not sure if we can leverage the .NET Foundation resources for that or not. I need to ask them.

—
Reply to this email directly or view it on GitHub.

[amccool]

re: build and nuget creation ... this probably needs to move to a new issue

[martinwoodward]

Moving discussion of builds to #1334 as recommended by @amccool

[DixonDs]

I am just wondering. What happened with the decision to move this implementation into OrleansContrib?