Milk, cookies, segfaults!

A CloudFormation recipe for scheduling Lambdas with 1-minute frequency

Featured

Cloud Infrastructure

A CloudFormation recipe for scheduling Lambdas with 1-minute frequency

Posted by Valeriu Paloş on .

Among the plethora of tools which Amazon has given us there’s AWS CloudFormation. IMO, this proves that AWS is (still) in a class of it’s own. It’s the embodiment of the “infrastructure as code” concept. And, it’s battle-tested, it works! So well, in fact, that they ElasticBeanstalk right on-top of it.

Disclaimer:

I have not used this in production yet, I’m currently running a longer test to gain some confidence. Therefore I do not recommend you rely on this for critical stuff. Test first!

Automation not supported

However, once you gain some experience with this beast, you start realizing that it also has faults. In fact, they tend to pile up; that’s one reason we now have some promissing tools like Terraform to play with. One such issue is the fact that CloudFormation doesn’t support the creation of scheduled (recurring) Lambda functions. And that’s a big problem, because you’re forced to do it by hand, by creating a scheduled event source using the AWS CLI, APIs or the Web Console.

Striving for 1-minute rate

Moreover, even if you do schedule a Lambda function by hand, the fastest rate at which you can invoke it is 5 minutes, no less.

Recently, I needed an automatically created Lambda function to be periodically invoked at 1-minute intervals. No more, no less. And, eventually, I found a way.

Strategy

The trick is to use a custom metric called Tick (published by our lambda) which can be either 0(zero) or 1(one). We set two CloudWatch alarms which trigger on each state, respectively: TickState0 triggers when Tick == 0 for 1+ minutes and TickState1 triggers when Tick == 1 for 1+ minutes.

{
  "TickState0": {
    "Type": "AWS::CloudWatch::Alarm",
    "Properties": {
      "AlarmName": "TickState0",
      "Namespace": "Tick",
      "MetricName": "Tick",
      "Statistic": "Average",
      "EvaluationPeriods": 1,
      "Period": 60,
      "Threshold": 0,
      "ComparisonOperator": "LessThanOrEqualToThreshold",
      "AlarmActions": [ { "Ref": "TickTopic" } ]
    }
  },

  "TickState1": {
    "Type": "AWS::CloudWatch::Alarm",
    "Properties": {
      "AlarmName": "TickState1",
      "Namespace": "Tick",
      "MetricName": "Tick",
      "Statistic": "Average",
      "EvaluationPeriods": 1,
      "Period": 60,
      "Threshold": 1,
      "ComparisonOperator": "GreaterThanOrEqualToThreshold",
      "AlarmActions": [ { "Ref": "TickTopic" } ]
    }
  }
}
CloudWatch Alarms

Two alarms are set to trigger on distinct values of the `Tick` metric; both will notify the same SNS Topic.

Each Alarm triggers the Lambda (via an SNS topic) and the Lambda toggles the Tick metric (which will trigger the opposite alarm after ~1 minute).

# Get message body (stack update or alarm).
message = json.loads(event['Records'][0]['Sns']['Message'])

# Detect source event.
source = message.get('RequestType', 'Alarm')

# Confirm stack deletion.
if source == 'Delete':
    return send(message, context, SUCCESS)

# Set/toggle state.
if source == 'Alarm':
    state = int(not float(message['Trigger']['Threshold']))
else :
    state = 0

# Set metric to state.
cw = boto3.client('cloudwatch')
cw.put_metric_data(
    Namespace = 'Tick',
    MetricData = [ {
        'MetricName': 'Tick',
        'Value': state
    } ]
)
AWS Lambda excerpt

Reads the current threshold from the invoking Alarm, and sets the opposite state (i.e. toggles the metric).

What’s left is to use a CF custom resource to ensure the Lambda is called on stack operations (create/update/delete) so the whole setup is self-managed.

{
  "StartTick": {
    "Type": "Custom::StartTick",
    "DependsOn": [ "TickState0", "TickState1" ],
    "Properties": {
      "ServiceToken": { "Ref" : "TickTopic" }
    }
  }
}
Custom resource

Ensures the Lambda function will be triggered on stack update operations.

Test Drive!

  • Clone the code (see below) and use it to launch a CF stack. You’ll have to upload the lambda function (the tick.zip file) on S3 somewhere (and update it’s URI in the CF JSON file).
  • After it started, you should observe in the CloudWatch -> Logs section that a log message is published every minute by the Lambda function (see screenshot below).
The Lambda function outputs the timestamp on each call.
Invocation traces

The Lambda function outputs the timestamp on each call.

Some considerations

  • Please note that the actual Lambda has additional code and is prepared to be invoked by either the CloudWatch Alarms or the custom resource events (i.e. stack updates).
  • Expecting a shorter trigger period using this method is not realistic, since AWS CloudWatch will always aggregate metrics inside a 1-minute time period.
  • In practice, however, you will observe that this actually tends to be triggered faster than on 1-minute intervals; this is influenced by the different check-times of each CloudWatch alarm (check times differ).

Please see the GitHub repository for the full code!

That’s it, enjoy! :)

user

Valeriu Paloş

By trade, a programmer of things. Having a passion for distributed systems, parallelization, infrastructures, algorithms, languages and, well... so much more!