EC2 Spot Interruptions – AWS Fault Injection Simulator

Abstract

AWS Fault Injection Simulator now supports Spot Interruptions, now you can trigger the interruption of an Amazon EC2 Spot Instance using AWS Fault Injection Simulator (FIS).
With FIS, you can test the resiliency of your workload a…


This content originally appeared on DEV Community and was authored by 🚀 Vu Dao 🚀

Abstract

  • AWS Fault Injection Simulator now supports Spot Interruptions, now you can trigger the interruption of an Amazon EC2 Spot Instance using AWS Fault Injection Simulator (FIS).
  • With FIS, you can test the resiliency of your workload and validate that your application is reacting to the interruption notices that EC2 sends before terminating your instances.
  • This blog guide you step-by-step to create FIS Experiment templates using AWS CDK

Table Of Contents

🚀 Overview of EC2 spot instance

  • Amazon EC2 Spot Instances reduce the cost up to 90% but can be interrupted or reclaimed at any time with warning in 2 mins.
  • We can use aws-node-termination-handler to ensures that the Kubernetes control plane responds appropriately to events that can cause your EC2 instance to become unavailable

🚀 Simulate Spot Interruptions architect

  • Starting the FIS experiment which sends send-spot-instance-interruptions event.
  • Use cloudwatch event rule to catch EC2 Spot Instance Interruption Warning event and then trigger lambda function for sending slack notifications.
  • aws-node-termination-handler kubernetes DaemonSet also takes action when catching the event

Now we start creating CDK stacks

🚀 Create Lambda function - send slack

  • Lambda handler parses the event to send slack message which contains event detail-type, instance ID and action

    app.py 

    import requests
from datetime import datetime
import json

def send_slack(msg):
    """ Send payload to slack """
    webhook_url = "https://hooks.slack.com/services/******"
    footer_icon = 'https://cdkworkshop.com/images/new-cdk-logo.png'
    color = '#36C5F0'
    level = ':white_check_mark: INFO :white_check_mark:'
    curr_time = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
    payload = {"username": "Test",
            "attachments": [{
                                "pretext": level,
                                "color": color,
                                "text": f"{msg}",
                                "footer": f"{curr_time}",
                                "footer_icon": footer_icon}]}
    requests.post(webhook_url, data=json.dumps(payload), headers={'Content-Type': 'application/json'})

def handler(event, context):
    detail_type = event.get('detail-type', '')
    instance_id = event['detail']['instance-id']
    action = event['detail']['instance-action']
    message = f'{detail_type}\nresource: {instance_id}, action: *{action}*'
    send_slack(message)

  • Lambda stack

    lambda.ts 

    const send_slack = new lambda.Function(this, 'slackLambda', {
            description: 'Send Event message to slack',
            runtime: lambda.Runtime.PYTHON_3_8,
            code: lambda.Code.fromAsset('lambda-code/app.zip'),
            handler: 'app.handler',
            functionName: 'send-slack-spot-event'
        });

🚀 Create event rule of spot interruption

  • The event listens to EC2 Spot Instance Interruption Warning to trigger the above lambda function

    event.ts 

            const spot_event = new event.Rule(this, 'SpotEventRule', {
            description: 'Spot termination event rule',
            ruleName: 'spot-event',
            eventPattern: {
                source: ['aws.ec2'],
                detailType: ['EC2 Spot Instance Interruption Warning'],
                detail: {
                    'instance-action': ['terminate']
                }
            }
        });

        spot_event.addTarget(new event_target.LambdaFunction(send_slack));

🚀 Create FIS service role

  • IAM role for AWS FIS permissions to handle the target resources here is EC2 instance

    fis_role.ts 

            const fis_role = new iam.Role(this, 'FisRole', {
            roleName: 'spot-fis-test',
            assumedBy: new iam.ServicePrincipal('fis.amazonaws.com')
        });

        const ec2_policy_sts = new iam.PolicyStatement({
            sid: 'SpotFisTest',
            effect: iam.Effect.ALLOW,
            actions: [
                'ec2:DescribeInstances',
                'ec2:StopInstances',
                'ec2:SendSpotInstanceInterruptions'
            ],
            resources: ['arn:aws:ec2:ap-northeast-1:*:instance/*'],
            conditions: {
                'StringEquals': {'aws:RequestedRegion': props?.env?.region}
            }
        });

        fis_role.addToPolicy(ec2_policy_sts);

🚀 Create FIS Experiment Template

  • The experiment template includes:

    • Action: send-spot-instance-interruptions, parameter: durationBeforeInterruption PT2M
    • Targets:
    • Resource type: aws:ec2:spot-instance
    • Resource filters: State.Name=running
    • Selection mode: COUNT(1)

  • Stack

    fis.ts 

            const target: fis.CfnExperimentTemplate.ExperimentTemplateTargetProperty = {
            resourceType: 'aws:ec2:spot-instance',
            resourceTags: {'eks:nodegroup-name': 'eks-airflow-nodegroup-pet'},
            selectionMode: 'COUNT(1)',
            filters: [{
                path: 'State.Name',
                values: ['running']
            }]
        };

        const action: fis.CfnExperimentTemplate.ExperimentTemplateActionProperty = {
            actionId: 'aws:ec2:send-spot-instance-interruptions',
            parameters: {'durationBeforeInterruption': 'PT2M'},
            targets: {'SpotInstances': 'spot-fis-target'}
        };

        const fis_exp = new fis.CfnExperimentTemplate(this, 'FisExperiment', {
            description: 'Spot Interruption Simulate',
            roleArn: fis_role.roleArn,
            tags: {
                'Name': 'spot-interrupt-test',
                'cdk': 'fis-stack'
            },
            stopConditions: [
                {source: 'none'}
            ],
            targets: {'spot-fis-target': target},
            actions: {'send-spot-instance-interruptions': action}
        });

🚀 Start experiment template

  • Start

  • Complete

  • Slack notify the event and aws-node-termination-handler action either

🚀 Conclution

  • This kind of FIS experiment help us to test the scenario of spot interruption to check aws-node-termination-handler and fault tolerance of application
  • We should also know about FIS pricing. The AWS FIS price is $0.10 per action-minute.
.ltag__user__id__512906 .follow-action-button { background-color: #000000 !important; color: #62df88 !important; border-color: #000000 !important; }
vumdao image

🚀 Vu Dao 🚀 Follow

Awesome Devops || AWS SA || CloudOpz

GitHub logo vumdao / vumdao


This content originally appeared on DEV Community and was authored by 🚀 Vu Dao 🚀


Print Share Comment Cite Upload Translate Updates
APA

🚀 Vu Dao 🚀 | Sciencx (2021-10-24T15:39:45+00:00) EC2 Spot Interruptions – AWS Fault Injection Simulator. Retrieved from https://www.scien.cx/2021/10/24/ec2-spot-interruptions-aws-fault-injection-simulator/

MLA
" » EC2 Spot Interruptions – AWS Fault Injection Simulator." 🚀 Vu Dao 🚀 | Sciencx - Sunday October 24, 2021, https://www.scien.cx/2021/10/24/ec2-spot-interruptions-aws-fault-injection-simulator/
HARVARD
🚀 Vu Dao 🚀 | Sciencx Sunday October 24, 2021 » EC2 Spot Interruptions – AWS Fault Injection Simulator., viewed ,<https://www.scien.cx/2021/10/24/ec2-spot-interruptions-aws-fault-injection-simulator/>
VANCOUVER
🚀 Vu Dao 🚀 | Sciencx - » EC2 Spot Interruptions – AWS Fault Injection Simulator. [Internet]. [Accessed ]. Available from: https://www.scien.cx/2021/10/24/ec2-spot-interruptions-aws-fault-injection-simulator/
CHICAGO
" » EC2 Spot Interruptions – AWS Fault Injection Simulator." 🚀 Vu Dao 🚀 | Sciencx - Accessed . https://www.scien.cx/2021/10/24/ec2-spot-interruptions-aws-fault-injection-simulator/
IEEE
" » EC2 Spot Interruptions – AWS Fault Injection Simulator." 🚀 Vu Dao 🚀 | Sciencx [Online]. Available: https://www.scien.cx/2021/10/24/ec2-spot-interruptions-aws-fault-injection-simulator/. [Accessed: ]
rf:citation
» EC2 Spot Interruptions – AWS Fault Injection Simulator | 🚀 Vu Dao 🚀 | Sciencx | https://www.scien.cx/2021/10/24/ec2-spot-interruptions-aws-fault-injection-simulator/ |

Please log in to upload a file.



There are no updates yet.
Click the Upload button above to add an update.

You must be logged in to translate posts. Please log in or register.

Related Posts