How to save money on AWS Transfer Family(SFTP) Server- Non-Production use

Use Case:-

Our company has an SFTP server in the lower environment, but we don’t have regular traffic on it. Our QA team only uses it once every two weeks during production release testing. If you create an AWS SFTP server, you will be charged $0.30 p…


This content originally appeared on DEV Community and was authored by Rakesh Sanghvi

Use Case:-

Our company has an SFTP server in the lower environment, but we don't have regular traffic on it. Our QA team only uses it once every two weeks during production release testing. If you create an AWS SFTP server, you will be charged $0.30 per hour, and AWS will continue to charge you until you delete the server. When attempting to stop the SFTP server, you will receive the following message.

alert

Our expenditure on SFTP servers in lower environments is nearly $750 per month, despite our infrequent usage throughout the entire month. Can we tear up/down during off hours or can we create SFTP sever on Demand? Here is what I came up with a solution.

The Architecture:

Architecture

To make it simple I created a simple Step Function, we can extend this functionality if we want to call Step Function from API Gateway or create an EventBridge schedule.

Create SFTP serve step function:-

step function

Delete SFTP server step function:-

delete step function

Steps:-

  1. Configured EventBridge Scheduler.

  2. Create a Step Function State Machine using AWS ASL language.

  3. Add user information necessary for SFTP user creation, including the user's ID, S3 bucket path, and SSH public key, to the static data stored in DynamoDB. Additionally, once the SFTP server is successfully created, store its ID in this table. This id will use to delete the SFTP server.

You can download the source code from here, which has been implemented using AWS CDK and SAM, whichever approach you prefer.

CDK Deploy:-

1) cd cdk
2) npm install
3) cdk deploy --all -a "npx ts-node bin/app.ts" --profile <your profile name>

SAM Deploy:-

1) cd sam
2) sam deploy --stack-name SFTPAppStack --capabilities    CAPABILITY_NAMED_IAM --guided --profile <profile name>

Populate Static Data in DynamoDB:-

1) cd dynamodb
2) ./dynamodb.sh

You will see the below records in the DynamoDB table.

DynamoDB

you can use this link to generate your private key and public key. Modify your public key in the DynamoDB script.

Populate Dummy Data in S3:-

1) cd s3
2) ./s3.sh

Understand State Machine:-
I am using the AWS SDK integration API to create an SFTP server with the help of the endpoint "arn:aws:states:::aws-sdk:transfer:createServer". There are various parameters available for creating the server, but it's important to ensure the security aspect is verified if you plan to create an SFTP server in your account.

 "CreateServer": {
      "Type": "Task",
      "Next": "Save ServerId",
      "Parameters": {
        "Domain": "S3",
        "EndpointType": "PUBLIC",
        "IdentityProviderType": "SERVICE_MANAGED",
        "LoggingRole": "${SFTPBaseRole}",
        "Protocols": [
          "SFTP"
        ],
        "SecurityPolicyName": "TransferSecurityPolicy-2018-11"
      },
      "Resource": "arn:aws:states:::aws-sdk:transfer:createServer"
    },

Above execution will return sever-id which I am saving into the DynamoDB table. This server-id will be used to delete the SFTP server. Here is the code snippet to save server-id into the DynamoDB table using SDK endpoint "arn:aws:states:::dynamodb:putItem"

"Save ServerId": {
      "Type": "Task",
      "Resource": "arn:aws:states:::dynamodb:putItem",
      "Parameters": {
        "TableName": "${SFTPTable}",
        "Item": {
          "PK": {
            "S": "SFTP"
          },
          "SK": {
            "S": "ServerId"
          },
          "ServerId": {
            "S.$": "$.ServerId"
          }
        }
      },
      "Next": "Get SFTP User",
      "ResultPath": null
    }

Now I am fetching all SFTP users from the DynamoDB table. This users list we can iterate using Map and pass it to endpoint "arn:aws:states:::awssdk:transfer:createUser".

"Get SFTP User": {
      "Type": "Task",
      "Parameters": {
        "TableName": "${SFTPTable}",
        "KeyConditionExpression": "PK = :v1 AND begins_with(SK,:v2)",
        "ExpressionAttributeValues": {
          ":v1": {
            "S": "SFTP"
          },
          ":v2": {
            "S": "USER"
          }
        }
      },
      "Resource": "arn:aws:states:::aws-sdk:dynamodb:query",
      "Next": "Iterate User",
      "ResultPath": "$.context"

Iterate all users list using the map.

    "Iterate User": {
      "Type": "Map",
      "ItemProcessor": {
        "ProcessorConfig": {
          "Mode": "INLINE"
        },
        "StartAt": "Create SFTP User",
        "States": {
          "Create SFTP User": {
            "Type": "Task",
            "End": true,
            "Parameters": {
              "ServerId.$": "$.ItemSelector.ServerId",
              "HomeDirectory.$": "$.ItemSelector.Value.S3Path.S",
              "HomeDirectoryType": "PATH",
              "Role": "${SFTPBaseRole}",
           "SshPublicKeyBody.$":"$.ItemSelector.Value.SSHPublicKey.S",
           "UserName.$":   "States.ArrayGetItem(States.StringSplit($.ItemSelector.Value.SK.S, '#'), 1)"
            },
            "Resource": "arn:aws:states:::aws-sdk:transfer:createUser"
          }
        }
      },
      "End": true,
      "ItemsPath": "$.context.Items",
      "ItemSelector": {
        "ItemSelector": {
          "ServerId.$": "$.ServerId",
          "Value.$": "$$.Map.Item.Value"
        }
      }
    }

The $$.Map.Item.Value.Value context object contains the value of each data item. You can use $$.Map.Item.Value if you want to add/modify JSON elements in iteration. You can see I am adding the new attribute "ServerId.$": "$.ServerId"

Login to SFTP Server:-

You should be able to log in using the following hostname. I am using the us-west-2 region. Modify according to your region. Use any user you have created with your private key.

.server.transfer.us-west-2.amazonaws.com

Delete SFTP Server:-

Getting ServerId from DynamoDB and passing this value to endpoint "arn:aws:states:::aws-sdk:transfer:deleteServer"

{
    "Comment": "A description of my state machine",
    "StartAt": "Get SFTP ServerId",
    "States": {
      "Get SFTP ServerId": {
        "Type": "Task",
        "Resource": "arn:aws:states:::dynamodb:getItem",
        "Parameters": {
          "TableName": "${SFTPTable}",
          "Key": {
            "PK": {
              "S": "SFTP"
            },
            "SK": {
              "S": "ServerId"
            }
          }
        },
        "Next": "DeleteServer",
        "OutputPath": "$.Item"
      },
      "DeleteServer": {
        "Type": "Task",
        "End": true,
        "Parameters": {
          "ServerId.$": "$.ServerId.S"
        },
        "Resource": "arn:aws:states:::aws-sdk:transfer:deleteServer"
      }
    }
  }

Cleanup:-

sam delete --stack-name SFTPStack --profile <your profile name>
                               OR
cd cdk
cdk destroy --profile <your profile name>

Wrap-Up:-

By creating or deleting a server during off hours, we can observe how much money can be saved. At my company, we have made the decision to generate an SFTP server on an as-needed basis. I'm confident that our costs will be less than $10 per month, in contrast to the $750 expense for a non-production environment. As I mentioned earlier, this post is not about how to create a secured SFTP server, please ensure the security aspect if you are creating an SFTP server in your account. Hope you like this post and helps you to reduce your AWS $$$ bill.


This content originally appeared on DEV Community and was authored by Rakesh Sanghvi


Print Share Comment Cite Upload Translate Updates
APA

Rakesh Sanghvi | Sciencx (2023-04-21T22:24:39+00:00) How to save money on AWS Transfer Family(SFTP) Server- Non-Production use. Retrieved from https://www.scien.cx/2023/04/21/how-to-save-money-on-aws-transfer-familysftp-server-non-production-use/

MLA
" » How to save money on AWS Transfer Family(SFTP) Server- Non-Production use." Rakesh Sanghvi | Sciencx - Friday April 21, 2023, https://www.scien.cx/2023/04/21/how-to-save-money-on-aws-transfer-familysftp-server-non-production-use/
HARVARD
Rakesh Sanghvi | Sciencx Friday April 21, 2023 » How to save money on AWS Transfer Family(SFTP) Server- Non-Production use., viewed ,<https://www.scien.cx/2023/04/21/how-to-save-money-on-aws-transfer-familysftp-server-non-production-use/>
VANCOUVER
Rakesh Sanghvi | Sciencx - » How to save money on AWS Transfer Family(SFTP) Server- Non-Production use. [Internet]. [Accessed ]. Available from: https://www.scien.cx/2023/04/21/how-to-save-money-on-aws-transfer-familysftp-server-non-production-use/
CHICAGO
" » How to save money on AWS Transfer Family(SFTP) Server- Non-Production use." Rakesh Sanghvi | Sciencx - Accessed . https://www.scien.cx/2023/04/21/how-to-save-money-on-aws-transfer-familysftp-server-non-production-use/
IEEE
" » How to save money on AWS Transfer Family(SFTP) Server- Non-Production use." Rakesh Sanghvi | Sciencx [Online]. Available: https://www.scien.cx/2023/04/21/how-to-save-money-on-aws-transfer-familysftp-server-non-production-use/. [Accessed: ]
rf:citation
» How to save money on AWS Transfer Family(SFTP) Server- Non-Production use | Rakesh Sanghvi | Sciencx | https://www.scien.cx/2023/04/21/how-to-save-money-on-aws-transfer-familysftp-server-non-production-use/ |

Please log in to upload a file.




There are no updates yet.
Click the Upload button above to add an update.

You must be logged in to translate posts. Please log in or register.