Feature: Cache last read

[jhmartin]

S3 outages can happen, so consider caching the last successful read from S3 and utilizing it if the S3 call fails.

As an example, the instance-metadata service on one of my nodes started returning expired credentials thus causing the s3 call to fail, and it would be unfortunate if that prevented all future access to the node.

[alexrudd]

Hey, this is a feature I've been wanting to add. I should have some time next week to take a look at this and you're other two issues.

Thanks for the contributions!

[alexrudd]

Well I didn't have time to implement this in the end, but I have been thinking about how this might work. I see two problems for which caching could be beneficial:

a) Reducing the delay when establishing multiple ssh tunnels concurrently or in quick successions. There's no need to be checking s3 too frequently if a recent cache exists.
b) Offering a temporary fallback for when calls to s3 fail.

Any cache would have to be mounted into the key-manager container and thus would exist and potentially be editable on the host (is that a problem?).

When using the cache as a fallback, should there be a limit to how long it can be used in this way (s3 should never be down for significant amounts of time)?

Is there a nice portable way of giving feedback to the user regarding key-manager's operation (which key did it authenticate you with, did it use the cache, is it unable to connect to s3)? MOTD would be an ideal way of doing this (and it looks quite easy on CoreOS) but I'm unfamiliar with how this is down on other platforms.

I'll try and find some more time to think about and look into this. Feel free to share your own thoughts.

Thanks,
Alex