Nats

[gedw99]

I got this working with nats server running in process.

https://github.com/synadia-io/rethink_connectivity/blob/main/20-embedding-nats-server/main.go

@destel i would be happy to show the usage with example code .

This allow concurrency to be in app, across a machine or across a cluster . No code changes needed with any of those topologies.

it’s possible that it’s interesting to any developers that are looking to use the patterns of rill and also scale it

[destel]

Do you mean something like hazelcast in java?
I mean having a Map operation that transforms a channel, and that transformation is distributed between multiple machines using NATS.

[gedw99]

yes thats exactly right and thanks for taking an interest.

What I want to see is if my golang compute code can be wrapped with nats.go and rill so that it scales:

• for a single process with many go routines
• for a single computer
• for a cluster of computers.

The tenant being that you dont have to change your code, but just the way you connect to nats server.

Regarding a cluster of computers, NATS has mirroring for when you have a global super cluster. The writes go to the Origin ( or owner of the stream) and the Reads are from any NATS. NATS will replicate the data stream underneath based on how you setup mirroring.

[destel]

I've thought about it and there are some open questions.

Take for example a piece of code

out := Map(in, 5, func(id int) (*User, error)) {
    row, err := db.QueryRowContext(ctx, "SELECT * FROM users ...")
    // scan row, handle errors etc.    
})

There are some problems mostly related to the fact that the function passed to the Map is anonymous:

• How to pass the function (body of Map) to a remote host?
• The function depends on external variables. db and ctx in this case
• Even if we find a way to pass it to a remote host, that would require special security measures.
• Function arguments must be serializable into a JSON or some other format
• In this particular example the bottleneck is a database, so there's no benefit in distributing the work across a cluster

What I think might potentially work is some specialized DistributedMap function. Instead of an anonymous functions, it uses specially registered ones:

func NatsMap[A,B any](in <- chan Try[A], topic string, n int) <-chan.Try[B] {
    // take items from 'in'
    // serialize
    // send request to NATS topic
    // receive response
    // unserialize it into a value of type B
    // write to the output chan
}


func RegisterNatsMapHandler[A,B any](topic string, handler func(A) (B, error)) {
   // do nats Subscribe, etc.   
}

Open questions here:

• Work duplication. As far as I remember nats sends request to all subscribed nodes nodes, but only one result is used.
• How to handle errors, especially timeout errors. Retry or not?
• Delivery guarantees. If we choose to retry errors that will result in "at least once" semantics which may not be desirable.
• types A and B should still be serializable, but that's probably not an issue

To summarize, I think the second approach with explicit handler registration is doable. It could be a separate package. Still, there are a lot of edge cases that should be worked through very carefully.

What do you think? While this approach diverges from the original goal of seamless scaling without code changes, it might offer a more practical path forward given the complexities involved. I'm open to discussing this further and exploring other ideas you might have to achieve your scaling goals.

[gedw99]

I really appreciate your examination of this.

I hit the same questions, and also think the 2nd option with explicit is needed.

To help that approach , I am using a reflector that can asks NATS for what streams are there and code gen golang from it . Same as the old trick with generating golang code off a database.
I feel this will help devs a robust way to use Ril and nats together.

It’s a nice thing that I have found d helped me in so many ways .

https://github.com/shono-io/nats-shell

if you would like to work together on ril and nats i will be happy to contribute . I feel like it’s an exploration of what can be, as these things can end up at different results and i feel that’s a good way to look at this. I agree with you that it might not scale out but only up but let’s just see 👍

It woukd be best , since you know Ril, that you lead the proposal. I will endeavour to help on the nats side .

I am in telegram too btw.

[gedw99]

We need to add the NATS double ack. So all calls to nats are idempotent over time and space.

Just writing it here so we remember.

don’t want those pesky race conditions :)

[destel]

I've given this more thought over the weekend. Here's where I stand:

1. I have an example of integration with the stdlib SQL package here: [link]. However, I haven't included it in the library yet because I'm still considering the best way to structure it (main package, subpackage, separate GitHub repo, etc.).

2. I have plans for other integrations as well, including Redis and NSQ. Your idea of a NATS integration is also intriguing.

3. The main challenge is where to host these integrations. Adding them directly to Rill would introduce unnecessary dependencies even for those users who don't need them.

4. Currently, the most viable solution seems to be creating separate GitHub repositories for each integration. This would keep Rill simple while still offering extended functionality for those who need it.

5. I am trying to be very careful about API designs. I need some time to digest all this information and thoughts before making a decision on how to proceed.

Your ideas and input are valuable, and I appreciate the discussion. While I don't have an immediate solution, I'm actively considering how to structure it all in a way that would minimize the risk of breaking API changes in the future.

For now, I'll keep this issue open as a reference point for future decisions on integrations. Feel free to continue the discussion or share any additional thoughts you might have.

[gedw99]

I do t have much to add to that for now.

You might find this interesting though.

Nats and benthos is a common way to transform the data as it travels through the streams.

https://m.youtube.com/watch?v=EA2Pco3EvpU