Guidance on global Cosmos token cache

[mvromer]

I have an ASP.NET Core web app that's deployed to multiple Azure regions using what Microsoft calls the geode pattern. The app uses MIW for authentication, and I think I've reached a point where I need to think about how each of the regional copies of my app can have access to the same "view" of a user's account info/tokens that get serialized and cached by MIW/MSAL. I was hoping to get some guidance on what extension points in MIW to look at to achieve this given some of the considerations below.

Background

Let's say I've deployed to two regions (East US and North Europe). Originally, each region contained the following:

• An Azure App Service running my ASP.NET Core web app
• An Azure Redis cache acting as the in-region distributed cache for my app. I originally configured MIW to use this as its distributed token cache.
• Miscellaneous services needed to run the app (Key Vault, Azure Storage, etc.)

At the front of it all I use Azure Front Door to route a user's request to the nearest geode based on latency.

The main problem with this approach is that, in a given session, account lookup and token acquisition could fail because it's possible (and, in my initial experiments, very likely) a user's requests start getting routed to different regions. For example, a user initially authenticates in East US, and then a subsequent request is routed to North Europe, at which point the app (MIW specifically) in that region fails to find account information for the user because it's not in Redis cache located in the North Europe region, only the East US region.

Possible solution (?)

Given I'm deploying to Azure and already make use of Cosmos DB, I think the most promising idea I found is to use Cosmos instead of Redis as my token cache store. In particular, if I configure my Cosmos DB to have session consistency and ensure that I can flow the Cosmos session token correctly on a per-account basis when reading from/writing to the token cache, then (in theory) my app should be able to authenticate a user in one region and lookup their account/tokens in another region. At least, that's my understanding around Cosmos' guarantees around session consistency.

One constraint is that I cannot use Microsoft's CosmosCache distributed cache implementation because after looking at its implementation, it doesn't look like there's any way to get it to set the Cosmos session token when making calls to Cosmos. So I will have to provide my own wrapper around Cosmos, which I'm not opposed to. But ideally I would be able to plug in that custom cache store into MIW's existing token caching framework.

Questions

I'm looking to build a proof of concept around this above idea, and I had some questions specifically around what parts of MIW I could take advantage of here.

First, would my best play be to implement my own IMsalTokenCacheProvider that wraps my custom Cosmos cache implementation? I'm guessing, if so, I would want to derive from MsalAbstractTokenCacheProvider to take advantage of things like data protection?

Second, is there a recommended way of flowing additional information (especially that derived from the request context) to/from a token cache provider? To get the read-your-writes consistency guarantees of Cosmos DB, I would need to pull the Cosmos session token in the response I get after writing a user's account/token info to Cosmos and store it on a per-account basis. Then, on subsequent requests I would need to pass in that same session token when reading that user's account/token info from Cosmos.

My gut says that it might be best to set the session token in a cookie so that subsequent requests can flow it back to my web app. Given that, I feel like I would need to somehow set and/or read the cookies as part of writing/reading from the token cache. Again, my gut says that if I'm writing my own custom IMsalTokenCacheProvider that I might just inject IHttpContextAccessor into it and use it to see if there's a valid Cosmos session token available. That said, I don't fully have in my head all the different places where a token cache provider might get called or if there is a better way of flowing extra information to/from a token cache provider.

I don't know if there are other considerations I need to keep in mind. That said, any guidance from the MIW team would be greatly appreciated. Thank you!

[monko9j1]

Wondering if you ever came to a conclusion to this as I have the same question.

[mvromer]

In my case I wrote my own token cache provider that derives from MsalAbstractTokenCacheProvider and a custom cache implementation that is loosely based on IDistributedCache but 1) uses a multi-region Cosmos with session consistency for persistence and 2) passes back Cosmos session tokens on writes and lets me pass in those tokens on reads.

Inside my MSAL token provider, I inject an IHttpContextAccessor and my custom cache implementation (call it ICache for now). When writing, I just write to my ICache. If the write was successful, the result will have the session token for the cache entry. I then use the IHttpContextAccessor to see if a valid context is present, and if so, write the session token to a cookie.

When reading, I'll check to see if a valid HTTP context is present and, if so, try to read a session token from the request cookies. If a session token is present, I'll pass that along with the cache key to my ICache for reading. Otherwise, the cache will try to read without a session token, at which point you're subject to Cosmos' eventual consistency guarantees.

I've had this implementation running for almost a year now in our multi-region Azure deployment. Since implementing it, we haven't noticed any problems with looking up MSAL tokens across regions. That said, it depends on running within the context of a valid HTTP request to be able to set/retrieve the Cosmos session tokens, so if you use things like background/hosted services, this might not be appropriate for you unless you figure out another way of flowing the session token to your token cache provider.