Deployment of Defence

We will follow the below sections to setup CloudWatch and Serverless defence against SSH Bruteforce attacks

Installing CloudWatch Agent on SSH Instance

We will need to create an IAM Role for the CloudWatch agent running in EC2 instance to be able to push logs to CloudWatch. For this we will create a role with the policy CloudWatchAgentServerPolicy and will have to attach it to the Instance.

Once we have attached the role to the instance, we can download and install the agent from the official repo. Log in into the machine and run the below commands,

wget https://s3.amazonaws.com/amazoncloudwatch-agent/ubuntu/amd64/latest/amazon-cloudwatch-agent.deb

Once the file download is complete, we can install the debian package,

sudo dpkg -i amazon-cloudwatch-agent.deb

Once the installation is complete, we will need to create the configuration for CloudWatch Agent to run with. 

{
  "agent": {
    "run_as_user": "root"
  },
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/log/auth.log",
            "log_group_name": "ssh_logs",
            "log_stream_name": "{instance_id}"
          }
        ]
      }
    }
  }
}

Copy and save the above configuration to a file config.json in your Linux machine. Configuration can also be generated interactively using the config-wizard command - 

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-config-wizard

Essentially in the configuration we ask the CloudWatch agent to look for /var/log/auth.log where the ssh logs are stored and push them to CloudWatch under the log group ssh_logs with the stream name same as the instance ID.

To start the CloudWatch Agent run the below command,

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a fetch-config -m ec2 -c file:configuration-file-path -s

Update the configuration-file-path field with the absolute path to your config.json file

-a is for choosing an Action through which we do fetch-config to select our config file

-m is for mode

-s is to start the agent

We can verify that the logs are being pushed to CloudWatch by heading to CloudWatch console - Log Groups section

Create CloudWatch Metric and Alarm

Now that we have the SSH logs in CloudWatch, we can create a metric to filter failed attempts and use an alarm to notify us about a possible attack. To create a metric, we need to understand the pattern in which SSH logs are created.

A sample failed attempt log would look like below,

Feb 24 06:58:52 ip-172-31-40-226 sshd[3340]: Invalid user mno from 127.0.0.1 port 38168

To filter the logs that fall under the above pattern, we will use this filter,

[Mon, day, timestamp, ip, id, status="Invalid", user, username, from, srcip, ...]

To create the metric, we will use the below command,

aws logs put-metric-filter \
--log-group-name ssh_logs \
--filter-name FailedSSHAttempts \
--filter-pattern '[Mon, day, timestamp, ip, id, status="Invalid", user, username, from, srcip, ...]' \
--metric-transformations \
metricName=failed_ssh_attempts,metricNamespace=LogMetrics,metricValue=1,defaultValue=0

The command will create a metric with the name failed_ssh_attempts that we can further use it for creating an alarm. Before creating an Alarm we will need an Simple Notification Service (SNS) topic created which would notified when there is an alarm. To create a SNS topic,

aws sns create-topic --name bruteforce-trigger-lambda

The above command will create a SNS topic bruteforce-trigger-lambda and return the Arn for the resource. We will use that Arn to create the alarm using the below command,

aws cloudwatch put-metric-alarm \
--alarm-name SSHBruteForceAlarm \
--metric-name failed_ssh_attempts \
--namespace LogMetrics \
--statistic Sum \
--period 300 \
--threshold 5 \
--comparison-operator GreaterThanOrEqualToThreshold \
--evaluation-periods 1 \
--alarm-actions <ARN_OF_SNS_TOPIC>

The command will create an alarm using the metric that we had created before and if there are more than or equal to 5 events present during a time period of 5 minutes (300 seconds), then the alarm will trigger the SNS topic bruteforce-trigger-lambda that we created.

Serverless Defence for identifying the IP and blocking it

We will deploy two serverless applications on Lambda. The first serverless function will have HTTP endpoints that would block a given IP and also show the blocking history for the Access Control List (ACL). All this data gets stored in DynamoDB to maintain history.

cd into the serverless-fn-blockip directory. The following parameters are configurable before deployment,

  • region: AWS Region to deploy in. ACL must be in the same region
  • accessToken: Access token used to authorize requests to block IPs
  • aclID: ACL that will be used for blocking
  • stateTableName: DynamoDB table that will be created to maintain current blocking state
  • historyTableName: DynamoDB table that will be created to maintain action history
  • ruleValidity: Time (in minutes) after which the IP is unblocked

Configure the above fields in config.js file,

module.exports = {
    region: "<AWS_REGION>",
    accessToken: "<SOME_RANDOM_STRING>",
    aclId: "<VPC_ACL_ID>",
    stateTableName: "ip_block_state",
    historyTableName: "ip_block_history",
    ruleValidity: 10
}

Make sure to modify at least the region, aclId and accessToken based on your requirements before deployment.

To deploy the function we use Serverless framework which in turn would create a CloudFormation stack to get the required resources deployed. cd into the directory and run the below commands,

npm install
#Create DynamoDB tables
node initDb.js
serverless deploy

The deploy command would return the endpoints that we require once the stack is created. Note down the endpoints, for use in future.

For the next Serverless function, cd into serverless-fn-trigger directory. The following parameters are configurable before deployment,

  • region: AWS Region to deploy in. Same as the previous function
  • logGroup: Name of the log group (in our case ssh_logs)
  • blockIPLambdaUrl: blockIP function's HTTP Endpoint returned in previous step
  • blockIPLambdaAccessToken: Access token used to authorize requests to block IPs
  • attackThreshold: Minimum number of events from a single IP to block the IP
  • startTimeOffset: Time offset from current time after which the requests are considered in minutes
module.exports = {
	region: "<AWS_REGION>",
    logGroup: "<LOG_GROUP_NAME>",
    blockIPLambdaUrl: "<BLOCK_IP_ENDPOINT>",
    blockIPLambdaAccessToken: "<BLOCK_IP_ENDPOINT_ACCESS_TOKEN>"
    attackThreshold: 3,
    startTimeOffset: 5,
}

Make sure to modify at least the region, logGroup, blockIPLambdaUrl and blockIPLambdaAccessToken before deployment

In the serverless.yml configuration file, we will have to add the SNS Topic ARN and name as the trigger for the lambda function that is to be deployed. This will connect our previous infrastructure (CloudWatch Logs, Metric and Alarms) with the Lambda functions.

Update the values <AWS_ARN_SNS_TOPIC> and <SNS_TOPIC_NAME> under the functions section of the config file,

functions:  
  vpchandler:
    handler: handler.vpchandler
    events:
     - sns:
        arn: <AWS_ARN_SNS_TOPIC>
        topicName: <SNS_TOPIC_NAME>

Once done, to deploy the serverless application, run the below commands,

npm install
serverless deploy