[FEATURE] Integration with AWS Step Functions

[mcuervoe]

Is your feature request related to a problem? Please describe.
When AWS Lambda is used in AWS Step Functions, AWS Step Functions uses the field "errorType" to implement
catch and retry logic. When the Lambda is implemented in Java, Javascript, or Python, this field is
generated when the code raises an exception.

In the case of Java, the "errorType" takes the name of the exception class.

See these documents for details:

• https://docs.aws.amazon.com/step-functions/latest/dg/tutorial-handling-error-conditions.html
• https://docs.aws.amazon.com/lambda/latest/dg/java-exceptions.html
• https://aws.amazon.com/getting-started/hands-on/handle-serverless-application-errors-step-functions-lambda/

Unfortunately, when the lambda function is implemented with holy-lambda, I have not seen a way to make this mechanism works:

• If the code throws a java exception, holy lambda wraps the exception, and the input that arrives at the Step Functions doesn't have the field "errorType" populated. This makes the code treated with "Lambda.Unknown".

{
"error": "Lambda.Unknown",
"cause": "The cause could not be determined because Lambda did not return an error type. Returned payload: {\"statusCode\":500,\"headers\":{\"content-type\":\"application/json\"},\"body\":\"{\\\"via\\\":[{\\\"type\\\":\\\"eu.electronicid.evidences.exceptions.InvalidEvidenceFileNameException\\\",\\\"message\\\":\\\"key is not valid file name, key=120290614_769f878d-696a-419a-b66c-91e642f7d0e3.zip\\\",\\\"at\\\":[\\\"eu.electronicid.evidence_archiving.logging$exception_type__GT_exception\\\",\\\"invokeStatic\\\",\\\"logging.clj\\\",15]}],\\\"trace\\\":[[\\\"eu.electronicid.evidence_archiving.logging$exception_type__GT_exception\\\",\\\"invokeStatic\\\",\\\"logging.clj\\\",15],[\\\"eu.electronicid.evidence_archiving.logging$fatal\\\",\\\"invokeStatic\\\",\\\"logging.clj\\\",30],[\\\"eu.electronicid.evidence_archiving.core$validate_request\\\",\\\"invokeStatic\\\",\\\"core.clj\\\",41],[\\\"eu.electronicid.evidence_archiving.core$EvidenceArchivingFunction\\\",\\\"invokeStatic\\\",\\\"core.clj\\\",69],[\\\"eu.electronicid.evidence_archiving.core$EvidenceArchivingFunction\\\",\\\"invoke\\\",\\\"core.clj\\\",69],[\\\"clojure.lang.Var\\\",\\\"invoke\\\",\\\"Var.java\\\",384],[\\\"fierycod.holy_lambda.custom_runtime$next_iter\\\",\\\"invokeStatic\\\",\\\"custom_runtime.clj\\\",80],[\\\"fierycod.holy_lambda.custom_runtime$next_iter\\\",\\\"invoke\\\",\\\"custom_runtime.clj\\\",60],[\\\"clojure.lang.Var\\\",\\\"invoke\\\",\\\"Var.java\\\",399],[\\\"eu.electronicid.evidence_archiving.core$_main\\\",\\\"invokeStatic\\\",\\\"core.clj\\\",97],[\\\"eu.electronicid.evidence_archiving.core$_main\\\",\\\"doInvoke\\\",\\\"core.clj\\\",97],[\\\"clojure.lang.RestFn\\\",\\\"applyTo\\\",\\\"RestFn.java\\\",137],[\\\"eu.electronicid.evidence_archiving.core\\\",\\\"main\\\",null,-1]],\\\"cause\\\":\\\"key is not valid file name, key=120290614_769f878d-696a-419a-b66c-91e642f7d0e3.zip\\\"}\"}"
}

Since all the exceptions are treated the same, there is no way to create a different workflow branch based on the error returned.

• If the code tries to create an output with the field "errorType", something like this:

(try
    (validate-request mode request)
    {:statusCode 200
     :headers    {"content-type" "application/json"}
     :body       {"message": "ok"}}
    (catch Exception e
        {:statusCode   500
         :errorType    (get (ex-data e) :error-type (type e))
         :errorMessage (.getMessage e)
         :headers      {}
         :body         (json/generate-string (ex-data e))})

In this case, the output gets wrapped in a field "output". AWS Step Functions receives something like:

{
  "output": "{\"statusCode\":500,\"errorType\":\"invalid-file-name\",\"errorMessage\":\"key is not valid file name, key=120290614_769f878d-696a-419a-b66c-91e642f7d0e3.zip\",\"headers\":{},\"body\":\"{\\\"error-type\\\":\\\"invalid-file-name\\\",\\\"key\\\":\\\"120290614_769f878d-696a-419a-b66c-91e642f7d0e3.zip\\\",\\\"mode\\\":\\\"archive\\\",\\\"env\\\":\\\"test\\\",\\\"id\\\":\\\"arn:aws:states:eu-west-1:593702161747:execution:EvidenceArchivingStepsTest:cabf4fe4-5b25-5df5-9502-4cc2d95a210c_b79f44d7-a610-ec98-f47c-3c79eddc9c0b\\\",\\\"bucket\\\":\\\"eid-evidence-test-input\\\",\\\"message\\\":\\\"key is not valid file name, key=120290614_769f878d-696a-419a-b66c-91e642f7d0e3.zip\\\"}\"}",
  "outputDetails": {
    "truncated": false
  }
}

In this case, AWS Steps Functions does not even consider this as an error (and therefore, the catch and retry clauses of the workflow can't be used)

Describe the solution you'd like
I think it is needed a way to provide a errorType to AWS Step Functions when some error happens processing the lambda

Describe alternatives you've considered
I see there are two possible alternatives:

1. When the code throws an exception, ensure that "errorType" gets populated.

• Pros:
  • Consistency with Java, Javascript, Python implementations
• Cons:
  • We may break code that is based on previous behavior.
  • Creating custom java exception classes is necessary for a fine-tune workflow behavior mapping.

2. Provide a mechanism to return "erroType" from a map

• Pros:
  • No need to create custom java exception classes, so this is a much more idiomatic solution

Additional context
Add any other context or screenshots about the feature request here.

[FieryCod]

Do you maybe have a simple repro with step functions? I'm happy to tune HL to make it work. What I can see is you're getting the error in the form of:

(defn- send-runtime-error
  [runtime iid ^Exception err disable-analytics?]
  (u/println-err! (u/->str "[holy-lambda] Runtime error:\n" (pr-str (Throwable->map err))))
  (let [response (u/http "POST" (url runtime iid "/error")
                         {:statusCode 500
                          :headers    {"content-type" "application/json"}
                          :body       (Throwable->map err)}
                         disable-analytics?)]
    (when-not (response :success?)
      (u/println-err! (u/->str "[holy-lambda] Runtime error sent failed.\n" (str (response :body))))
      (System/exit 1))))

I have to check the custom runtime design documents.

[mcuervoe]

I do not have this simple repo, but I will create one. I will post it here when I have it ready.

[FieryCod]

Thank you! Just simple "hello world" on AWS SAM + Step Functions exposing this issue would be enough 😀 Nothing too fancy please.

[mcuervoe]

Hi,
I have created this repo:

https://github.com/mcuervoe/holy-lambda-steps

It is an implementation in Clojure/HL of the example in python that you can find in https://aws.amazon.com/getting-started/hands-on/handle-serverless-application-errors-step-functions-lambda/

You can follow the instruction in this article to invoke the step function with different inputs to generate different exceptions (Step 4, Test your Error Handling Workflow).

If you have any issues with the deployment of the code, let me know

Miguel

[FieryCod]

@mcuervoe please check the latest HL release (0.6.7). This should fix your issue :)

FYI: HL now supports both Clojure ExceptionInfo and Java classes. (842b5cf)

{:errorMessage (.getMessage err)
 :errorType    (or
                 ;; You can throw an `(ex-info "SomeMessage {:type "CustomErrorType"})` to receive "error" "CustomErrorType"
                 (:type (ex-data err))
                 ;; As a fallback the ErrorName is inferred from the class
                 (.getName (.getClass ^Class err)))
 :stackTrace   (mapv str (.getStackTrace err))}

Please let me know if this solves your issue. Btw, thank you for the repro. Really well done and easy to follow!

[mcuervoe]

Hi @FieryCod ,

I tested the change, and it works beautifully. I updated the repo https://github.com/mcuervoe/holy-lambda-steps so that it accepts new status codes in the input to try the throw of ExceptionInfo with the field :type to determine the error type:

• 429 -> com.company.lambda_steps.exceptions.TooManyRequestsException
• 430 -> ExceptionInfo with :type equals to too-many-requests
• 503 -> com.company.lambda_steps.exceptions.ServerUnavailableException
• 504 -> ExceptionInfo with :type equals to too-many-requests
• 300 -> ExceptionInfo without :type
• 200 -> No exception
• everything else -> RuntimeException

I think it can be used as an example of the integration of HL and Step Functions

I really like the new functionality of allowing ex-info to drive the workflow when errors occur.

Thanks,
Miguel