[reaper] re-write lambda #4135
[reaper] re-write lambda #4135
Conversation
twrichards
force-pushed
the
reaper/lambda-rewrite
branch
2 times, most recently
from
9c280ab to
dbb948f
Compare
twrichards
force-pushed
the
reaper/lambda-rewrite
branch
from
5182e16 to
5651b94
Compare
…next batch (for both soft and hard delete)
twrichards
force-pushed
the
reaper/lambda-rewrite
branch
10 times, most recently
from
2b398c9 to
8bc12e4
Compare
twrichards
force-pushed
the
reaper/lambda-rewrite
branch
6 times, most recently
from
c9f2de4 to
8154301
Compare
twrichards
force-pushed
the
reaper/lambda-rewrite
branch
2 times, most recently
from
8bdad62 to
62aebfa
Compare
twrichards
force-pushed
the
reaper/lambda-rewrite
branch
from
62aebfa to
ef84a18
Compare
| # run: | | ||
| # npm install | ||
| # npm run riffraff-artefact | ||
| - name: Reaper |
| export const AWS_REGION = 'eu-west-1'; | ||
| export const STACK = 'media-service'; | ||
| export const REAPER_APP_NAME = 'grid-reaper'; | ||
| export const ENV_VAR_BATCH_SIZE = 'BATCH_SIZE'; |
There was a problem hiding this comment.
this is just a tiny bit confusing because I am assuming the variable name is ENV_VAR_BATCH_SIZE and the value is a number (string but number) but it sounds like the variable name is "BATCH_SIZE"? I could be reading wrong. But maybe clearer if we just use the string in place?
There was a problem hiding this comment.
this is a variable representing the name of the environment variable, and it's in the constants.ts so it can be referred to when it's set in the CDK stack, and then when its read in the lambda code. Having a constant for this avoids there ever being a discrepancy when using that value.
There was a problem hiding this comment.
ok, I think I have a mild preference for using the string in place but I will leave it up to you, it's really not important :)
|
|
||
| const recordDate = new Date(); | ||
|
|
||
| const IDsToSoftDelete = await fetch( |
There was a problem hiding this comment.
| const IDsToSoftDelete = await fetch( | |
| const idsToSoftDelete = await fetch( |
| body: JSON.stringify(IDsToHardDelete), | ||
| }, | ||
| ).then(parseResponseAs<HardDeletedStatuses>()); | ||
| // FIXME check for false values in the response |
There was a problem hiding this comment.
I think we still want to do this, or else change the 'Hard delete succeeded.' message?
| .then((result) => { | ||
| const value = result.Parameter?.Value; | ||
| if (!value) { | ||
| throw Error(`Could not retrieve parameter value for '${Name}'`); |
There was a problem hiding this comment.
is there a useful message in result we could log here?
|
Closing in favour of a consolidated approach entirely within |
Re-write the existing lambda, primarily to coordinate (and log) calls to the endpoints added in #4134. It's designed to run on a schedule (currently every 15mins, regardless of stage).
Each invocation it attempts to...
is:reapable(see Add a new filter showing reapable images #3926)is:reapableAND have been 'soft deleted' for 2 weeks or more...logging the above and storing a permanent record to dedicated (& stage specific) S3 bucket.
* batch size is different per stage (proportional to env's ingenstion rate) but is 1000 for PROD (since this is the maximum batch for the bulk delete APIs etc. used under the hood).