Topic prefix stripping in server-to-server connections

[smurfix]

Hi,

While FlashMQ supports prefixing topics on messages it sends and receives, it doesn't seem to be able to remove a prefix from incoming or outgoing topics, at least as far as flashmq.conf manpage is concerned.

This would afford transparently tunneling arbitrary messages through a shared server.

Mosquitto can do it ;-)

[halfgaar]

I can see why you would want it. I'd have to spend some time looking at how complex it would be to make, and what the performance implications are (I tend to avoid adding overly heavy features).

[smurfix]

I hear you. A "strip the first N components of the incoming / outgoing message" feature would work for most if not all users, if that's easier than a comparison; I didn't yet encounter any such link that needed more complex arrangements.

In any case, on the positive side, at least this feature has no performance impact if it's not used.

[marcelrv]

This topic mapping would indeed be very appreciated.
@smurfix you indicate topic prefixing when sending, how did you accomplish that as that would already do most of the job in my case, but did not recognise how to do that from the conf file nor from the bridge code

[smurfix]

I currently accomplish topic prefixing with RabbitMQ, but I'd rather switch to something faster + more extensible (FlashMQ has this nice plugin architecture …) if at all possible.

[halfgaar]

I did some checking for feasibility. One question pops to mind: if there is authentication in place, do you expect that to happen before or after the mangling? Let's say we're talking about messages on topic a/b/c:

• In incoming publish: do you want the 'write' permission to be checked on prefix/a/b/c or a/b/c?
• In delivering a publish to a subscriber, same question: do you want that 'read' permission to be checked on prefix/a/b/c or a/b/c?

There are some architectural difficulties if you would want the ACL check to happen with the prefix. Logically, I'd say without the prefix makes most sense.

[smurfix]

As long as you're consistent whether the checks are done on the paths as seen by the client or as seen by the server (= every other client that doesn't have mangling applied to it) I can write an ACL to match.

Thus, my answer is "whatever happens to be easier to implement".

[wiebeytec]

OK, then I'll give it a go. Prefix stripping on receive is easy. Adding a prefix on send is a bit more difficult, but if we can do it at the very last step, meaning after authentication on the original topic string, it's probably doable.

Do note that it will incur some extra overhead, because there are certain optimizations it can't do then.

Probably the next feature you want is multiple bridge connections with load balancing. It's on my list...

[halfgaar]

I created a branch with a test version: https://github.com/halfgaar/FlashMQ/tree/PREVIEW-bridge-prefixes. If compiling is too hard, I can probably give you a .deb package.

The bridge config sections supports a local_prefix and remote_prefix option. It should act the same as Mosquitto's, in terms of how it subscribes and what it adds and removes.

The prefix options are defined per bridge, not per pattern. There are some conceptual problems with having them per pattern. I'm not sure how Mosquitto does it...

The ACL checking turned out to be the following:

• Remote prefix is removed and local prefixes is added before ACL write checks. So, the ACL write checking is done after the mangling.
• The read ACL checks (delivery to subscribers) is done before mangling. So, the ACL is done on the path as it appears locally. Then, after ACL, the local prefix is removed and the remote prefix added.

[halfgaar]

Actually, I was a bit premature. There are various issues with that branch still.

[smurfix]

Thanks. Looking forward to testing it once you've got the issues ironed out.

Building a .deb is not a problem.

[smurfix]

OK, then I'll give it a go.

👍🏼

Probably the next feature you want is multiple bridge connections with load balancing. It's on my list...

Heh. Not really. My volume is high but not that high.

What I do want is redundancy, i.e. connect my MQTT servers in an arbitrary mesh and lose no messages (*) if any one of them hangs / fails / whatever. I'm currently using my own protocol on top of MQTT for this and I'd love to be able to bypass the whole thing.

(*) except exact duplicates of course; when a sensor tells me five times that it's 18° out there I don't care how many of those arrive – as long as the number is greater than zero.

[halfgaar]

As an update; I was doing some prep work for this change, which kind of opened up can of worms of things that needed fixing. So, I'm working on a new release, coming in a few days I hope, and then I can continue with this.

[smurfix]

Thanks for the update. Not a problem.

[halfgaar]

I think I'm close to giving you a .deb to test. For what OS+version do you want it?

[smurfix]

Cool.
Standard Debian stable should be fine.

[halfgaar]

You can download a test version for Bookwork here. It's a release build, so optimized and no symbols. At this point, I don't expect test scenarios I need debuggers for, just whether it behaves correctly. I can always make a debug build if required.

Install with dpkg -i.

This comment still applies as to how it works.

The more scenarios tested, the better. Like bridges with persistent sessions (non-clean start and non-0 expiry), severed connections, queued messages, using aliases, etc.

[halfgaar]

I'm eager to any test results, both positive and negative alike :)

[smurfix]

Yeah and I'm eager for some "free" time in which to test the setup I have in mind …

I'll try to shovel some hours free next week.

[halfgaar]

Has the shoveling worked ;)?

[smurfix]

On 28.05.24 08:27, Wiebe Cazemier wrote: Has the shoveling worked ;)?

Still shoveling, unfortunately. Best laid plans and all that *grumble*.

…

-- -- regards -- -- Matthias Urlichs

[halfgaar]

I created new test builds, based off 1.15.2:

• https://www.flashmq.org/wp-content/uploads/testing/flashmq_1.15.2-1719506565+jammy_amd64.deb
• https://www.flashmq.org/wp-content/uploads/testing/flashmq_1.15.2-1719592043+bookworm_amd64.deb

There were some architectural changes in master, so this branch was adapted to that.

[smurfix]

Could you rebase this to the subscription-id branch? That'd let me set up a mostly-integrated test environment which I hopefully can (lightly) torture over the holidays.

[halfgaar]

It has some merge conflicts, and more with the subscription IDs. I can, but I'd rather first finish the subscription IDs and have that in master.

[smurfix]

Thanks. No hurry.

[halfgaar]

I tried to rebase, but now the tests fail. I don't know why yet.

I also saw some likely mistakes. So, this still needs some work.

[halfgaar]

I have a new test branch:

https://github.com/halfgaar/FlashMQ/tree/WIP-bridge-prefixes

The prefix is not applied automatically to the bridge's subscribe and publish directives. It felt too magic to do so, plus it would mean the prefixes are applied to all subscribe and publish directives, which is not good.

Let me know how it works out.

[halfgaar]

I rebased the test branch on master.

[halfgaar]

@smurfix and @Evil2000 have you been able to test it?

[smurfix]

Looks good to me.

One obvious enhancement would be multiple and/or unidirectional rulesets (i.e. mangle outgoing Local/ to Remote/Foo/ but leave incoming messages as they are) but the current version is sufficient for my immediate usecase.

NB what happens when I mangle remote/bar/ to (empty), subscribe to remote/bar/#, and a message with topic remote/bar/ arrives? Empty topics are illegal in MQTT but zero-length topic substrings are not, AFAIK.

Documentation

I would reword the "The prefixes aren't added" sentence to "Prefixes are not applied to …" because the prefix doesn't just add, it also takes away.

What happens to messages that don't fit the to-be-stripped prefix? are they left alone, or not transmitted/received at all, or is the respective "added" prefix added anyway?

Minor typo: "but you'll have to think about what hat results in"

[Evil2000]

@smurfix and @Evil2000 have you been able to test it?

Sorry for my late reply.
Currently I have my own code in production on an embedded system (Victron CerboGX which uses flashmq), but with MQTT 3.1.1 only. So, I had no time to test with this branch.

[smurfix]

@Evil2000 Welcome to the club, that's one of my usecases too.

I have written a setup script that teaches the Venus stuff to run on any plain old Linux laptop or whatever. I "only" need to find some time to update that to the latest Venus release …

[Evil2000]

@Evil2000 Welcome to the club, that's one of my usecases too.

I have written a setup script that teaches the Venus stuff to run on any plain old Linux laptop or whatever. I "only" need to find some time to update that to the latest Venus release …

@smurfix so you use the venus stuff on your own ("old laptop") hardware and connect the multiplus etc. directly to this device and save some money. This is smart. ;-)
Did you publish your scripts somewhere?

[smurfix]

Well for "production" it's a plain Raspberry Pi 3, running Debian instead of Venus OS because that way I can use my own kernel (Venus unhelpfully omits some kernel modules), and my own backup scripts and whatever else I need on the system (like a reasonably-up-to-date Python …), but yes.

I wrote about it on the Victron discussion forum some time ago.

https://github.com/M-o-a-T/venusian – though as I said it does need an update; be my guest if you want to take a stab at it, PRs are more than welcome. Note that the current version predates Victron's switch to FlashMQ.

We should probably take further discussion of this to my repo. ;-)

[halfgaar]

@smurfix I can probably add a directionality parameter to it. Your other feedback is good too, I'll get to it.

@Evil2000 just for reference about your PR: my branch is more complex because there are various things to keep into account, like queued messages, and ACL checks of those, but also the packet caching mechanism. It's a bit obscure how that works, and if not done carefully, other clients will be given changed topic string too. So, I think it's worthwhile to check if my branch works for you, because it should be safer in production.

I'd like to get all the feedback processed quickly, before Venus bumps its FlashMQ version. If we can get this into the next release, Venus 3.60 will have it too.

[smurfix]

I can probably add a directionality parameter to it

You may, but without separate-per-direction and/or multiple prefix sets that's not interesting for my specific usecase.

If we can get this into the next release, Venus 3.60 will have it too.

That would be superb, as a Venusian system might then be able to use their FlashMQ (and its Victron plugin) as-is, instead of running a second process and playing games with port redirection.