Dealing with forwarded data and workers that move home

[michielbdejong]

How about this setup:

• each system in the club has its own credentials check to decide if an incoming write should be accepted or not
• a write can be accepted if the credentials are believed to be specific to the worker (person who in the real world actually worked the hours)
• a write can also be accepted if the credentials are believed to be specific to a system that is authoritative for the worker (could be a client/user-agent or a server)
• each system in the club forwards each newly accepted write to each other system
• these other systems then do their own credentials check, based on their own config
• the access control list at each system therefore acts as a routing table, so we can route message forwarding in a tree
• if a message is repeated, the second one is ignored

That's what I'm implementing now in pondersource/prejournal#134 .
It means for instance that if the link from timeld.org to time.pondersource.com is down, updates will still reach time.pondersource.com via timesheet.dev3.evoludata.com, but these messages will be discarded unless the access control list is actively updated.

When a worker wants to start using a different time tracker app, it can be that their data will enter the federation at a different gateway than before. If that is before, then the user needs to be added to the access control list at the new gateway, and at the old gateway, the entry needs to be updated to indicate that data for this worker will from now on come via that neighbour system.

Example:

• 1 worker, "Alice"
• 2 apps, "App1" and "App2"
• 3 club-member systems: "S1", "S2" and "S3"
• S1 periodically polls the API of App1
• S2 periodically receives Webhook updates from App2

Initial situation:
Alice enters her hours in App1
S1 periodically polls the API of App1
S1 forwards this data to S2 and to S3

S1 accepts data about Alice from App1
S2 accepts data about Alice from S1
S3 accepts data about Alice from S1

Now Alice switches to using App2. Changes to be made:

• S1 will now accept data about Alice from S2 instead of from App1
• S2 will now accept data about Alice from App2 instead of from S1
• S3 will now accept data about Alice from S2 instead of from S1
  

[kroky]

I agree. Tiki will do the same - send to all registered club members and be able to receive from all of them provided they authenticate.

[michielbdejong]

where "provided they authenticate" is specific to which worker the data is about, right? If your server receives an API call from our server using the credentials you gave us, it would accept timesheet entries if they are about Ismoil and me, but not if they are about George and Angus, correct?

[kroky]

No, that's not correct. It will accept entries according to permissions your user has and currently it can insert entries for any other user. I could change permissions to allow users to insert entries only for themselves but I deliberately wanted to give broader permissions for initial integration and we tighten them up if needed.

[mcalligator]

I think this might be overlooking a couple of things - here's my reasoning:

• When Alice switches to using App2 (integrated with federated system club member S2), S2 is authoritative for those entries in the eyes of S1 and S3: it's as if she entered them into S2 directly - unless they are stored with their external App2 identifiers, in which case S2 (as well as the other club members) should treat them as read-only.
• App2 may not necessarily support or provide individual user credentials for Alice, so there's no guarantee that S2 will receive those when it polls App2. Hence S2 will need to represent Alice as an identity with credentials for which it is authoritative - both internally and when forwarding data to S1 and S3. If it already knows about Alice from having previously received timesheet data about her from S1, then it will already have an internal identity for her (mapped to the identity S1 uses to represent her), and hence it needn't create fresh credentials. When Alice makes the switch, S2 won't need to do credential-checking for Alice, since it will already trust the data it receives from App2 (by virtue of the integration).
• Going further, I don't think it's necessary for each worker to have a "home system" in the federation in the way you've described: the federation involves trusting the other club members to be forwarding reliable data from authenticated and authorised users. So in theory, Alice could enter time in all three external Apps, each integrated with a different club member, and her timesheet data would be shared among them all, with each mapping the identity accompanying incoming data to the identity they've stored for her, and hence matching the data to that worker.

We therefore need to distinguish between a worker's user identity for the federated system authoritative for their data, and the credentials used to forward those data to other club members. At issue is what level of access control each club member expects recipients of forwarded data to enforce over those data. The implicit assumption expressed earlier in the thread is that the receiving system won't necessarily be able to support the same access control, and thus the sending system needs to restrict what data it sends. The further implicit assumption is that the data sent should be restricted to those that the user identity initiating the transfer has access to. However, this also assumes that the sending system is only forwarding data on command, rather than automatically - which would require using system credentials for the receiving system, rather than user credentials. For example, when S1 forwards to S2, it might use credentials that S2 has provided specifically for the purposes of system-to-system data transfer.

[michielbdejong]

OK, very interesting discussion!

Let's label a set of credentials as a "user". So this can either be a natural person, or a neighbouring node in the network.

Let's label the natural persons whose hours we track as "workers".

Let's label what I described and implemented in PreJournal / time-pondersource-com as "hard-wired call tree": each worker has a home node, and from there, the data about that worker always propagates along the same hard-wired tree. It enters the home node via the worker's own user. When it's forwarded from node X to node Y, node X has a user at node Y, and node Y is hardwired so that node X is the only user that can tell node Y about that specific worker.

Let's label the relaxation that Victor implemented ("every user can always edit every worker's entries in any node") as "last-write-wins".

One problem I see with this architecture is that if you edit the same entry in two places, you risk creating a storm of updates that chase each other. In practice this would be unlikely unless the network is recovering from a network split. If we go for this relaxed security model then we should add some sort of mechanism to avoid update storms, probably? - CC @gsvarovsky :)

I understand what Angus wrote by itself, but I have a hard time linking a conclusion to it - @Angus-McAllister are you proposing the same "last-write-wins" architecture as Victor, or a subtle variation on it?

[mcalligator]

OK, happy to clarify what I had in mind (and apologies that it's not sufficiently evident in my post above!) My post in the thread for PreJournal Issue 134 does elaborate on this, but to bring it out from that product-specific context, let me summarise the thinking here:

• Time-tracking data is logged in any system, whether that is part of the federated 'club', or external to it (obviously).
• If logged in an external system, that system must be integrated in some way with at least one club member to enable data to enter the federation (also obvious).
• Each worker logging time has an identity with a set of credentials in at least one federated system (whether they enter their time directly in that federated system or indirectly in an external one integrated with it).
• Each worker also has a mapped equivalent identity (but not necessarily associated credentials) in other club member systems (implying that identity data are replicated between federated systems along with timesheet data).
• Each club member system forwards data to other club members using a separate set of system credentials issued by the target system that give the sender permission to write those data.
• This enables any given club member to receive timesheet data about any individual worker from any other system in the club.
• As you have identified, this also makes conflicting updates possible if an previous entry arrives later than a more recent one.
• Whilst the ultimate solution would be treat the federation as a set of CRDT domains, the more loosely-coupled solution is either to prevent conflicts (as you have done by hard-linking each external system to a specific club member instead) or handle them with appropriate logic. The backstop method is last-write wins, as you've suggested, but a better approach is for federated systems to assume a sufficiently accurate shared clock and capture the timestamps of when each entry was made into the original time-tracking tool (which timeld admittedly does not currently do). This assumes that clock skew < time gap between successive operations on one entry, which is reasonable for this domain.

Hope that clears things up!

[kroky]

I'd just comment that Tiki works mostly the way Angus describes with last-write wins logic in action. I don't mind we integrate the timestamp record checking but it might be a v2 upgrade maybe?

Regarding credentials, note that Tiki associates a set of credentials to a particular user inside Tiki which means that incoming data via these credentials are all tied to the permissions this particular user has. In our case user equals the worker in terms of entity - we use the same system users for credentials and workers but currently we don't impose a restriction for one user to create an entry to another worker (user). If we agree that this is not undesired, I can limit the UI to disallow it but keep importing work like that, so club members can use one set of credentials and create entries for different workers.

Also, currently, we don't do any mapping of worker identities - we should create each worker as a user in the Tiki instance but that is a soft requirement, i.e. data is imported even if user identity doesn't exist locally but if you want to give that user an account to review and edit their data in that instance, admin can create a user for them.

[michielbdejong]

I think we can conclude that PreJournal and Tiki work differently, and make that a part of our research report.
Then that triggers some potential follow-up research topics for the 1 October proposal:

1. preventing malicious edits, for instance by explicitly linking/merging identities when 1 user uses 2 apps or when an internal node in the network acts as a trusted forwarder, and being strict in access control otherwise
2. preventing message storms, for instance by keeping an edit log and "rolling back" if an older message comes in with a delay.