Blog
Anand Tiwari
Anand Tiwari
InfoSecurity Enthusiast, Author of ArcherySec tool.

Deploy ArcherySec as a Serverless on AWS using Zappa


ArcherySec is handy tool to feed vulnerability data from DevSecOps pipeline. Vulnerability management is a key part of your DevSecOps process. In our last blog post, we saw how to integrate ArcherySec in our CI/CD pipeline. In this blog post we’ll see how ArcherySec can be deployed as a serverless.

We often heard about “serverless” in computer world. Serverless is a cloud computing performance model where the cloud provider dynamically controls the allocation and provisioning of servers. A serverless application works in stateless compute containers that are event-prompted, temporary, and completely handled by the cloud provider. The pricing in the serverless is based on the count of executions rather than pre-purchased compute capacity.

Think how if your vulnerability management application is serverless and event-driven. We need ArcherySec API when we are feeding data from our DevSecOps pipeline or when we are analyzing vulnerabilities. Hosting ArcherySec application on an EC2 Instance or in containers like ECS make us pay for the idle time too.

In this article we will guide you to deploy a ArcherySec application as server-less using Zappa. Before start deploying ArcherySec application lets understand how Zappa works.

Zappa

Zappa is an open source project that helps you in building and deploying server-less, event-driven Python applications. It helps in deploying and building any WSGI compatible applications that are developed in python. It uses AWS Lambda and AWS API Gateway, and Amazon S3.

When we talk about application hosted on server-less, basically it’s not really serverless, there still is a webserver in a cloud that serving your web responses. The difference is the server life spent is measured in milliseconds. The entire life cycle of a server is within a single HTTP request.

In a traditional client server communication the client makes a request to the web server, calls the application HTML, CSS, etc and code sends back a response to the browser. The server should be up and running 24/7 just waiting for requests and responses.

Source: https://www.codeproject.com/KB/aspnet/HttpZipFolder/HttpZipFolder_Fig2.png


With AWS Lambda + AWS API Gateway when the HTTP request comes in through the AWS API Gateway, then it creates the instance of the AWS Lambda function which process the request returns through the AWS API Gateway and then the server is wiped out. So you have used a server in a matter of milliseconds and a major advantage of this approach is that you don’t have to worry about keeping your server up and running 24/7 and you only pay aws instance for a millisecond.

Source: https://medium.com/38th-street-studios/the-move-to-serverless-digital-ocean-heroku-aws-oh-my-ea5224a8d4bb


AWS Lambda

AWS Lambda lets you run code without provisioning or managing servers. You pay only for the compute time you consume, there is no charge when your code is not running.

So AWS Lambda function is perfect solution for us. It scales automatically and we pay only when we consume it. If our code is not running then it’s not chargeable. Also its running with zero administration.

Configure AWS Credentials

Before start using AWS, we have to make sure we have a valid AWS account and have the AWS access-key which we are going to use in our environment.

Get the access key from IAM

  • Go to IAM console and navigate to users.
  • Go to Security Credentials
  • Create access key and note down the value of “Access key ID” & “Secret access key”.


  • Create a folder called aws at the root level.

mkdir .aws

###~/.aws/credentials

[default]
aws_access_key_id = XXXXXXXXXXXXXXXXXXXXXXXXXXXX
aws_secret_access_key = XXXXXXXXXXXXXXXXXXXXXXXXXXXX

Permissions

Make sure you have provided correct permissions to the user. We need following permission to run Zappa.

  • AWSLambda
  • AmazonS3
  • AmazonAPIGateway
  • AWSDeepRacerCloudFormationAccessPolicy
  • AmazonEC2

Roles

We need to create role and provide the appropriate permissions. Zapp configuration file will use and deploy applications on Lambda and API gateway.

  • Go to Roles and create new role.
  • Attached the following policies
    • AWSLambda
    • AmazonAPIGatewayPushToCloudWatchLogs
    • AWSLambdaVPCAccessExecutionRole
  • Move to Trust Relationship and Edit Trust Relationship.
  • Copy and paste the below code.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Principal": {
        "Service": [
          "events.amazonaws.com",
          "lambda.amazonaws.com",
          "apigateway.amazonaws.com"
        ]
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

We need to trust the amazonaws events, lambda and apigateway end points.

  • Note down Role Name and Role ARN as we need in Zappa configuration file.

Deploy ArcherySec Application Using Zappa

Clone ArcherySec project folder in our local system.

$ git clone https://github.com/archerysec/archerysec.git
$ cd archerysec

Install ArcherySec required library


$ pip install -r requirements.txt

Next install zappa


$ pip install zappa

After installing Zappa, let’s configure Zappa

$ zappa init

which will ask us for the following:

  • Name of environment - default ‘dev’
  • S3 bucket for deployments. If the bucket does not exist, zappa will create it for us. Zappa uses this bucket to hold the zappa package temporarily while it is being transferred to AWS lambda, which is then deleted after deployment.
  • Project’s settings provide as archerysecurity.settings.base

Some additional parameter required by Zappa configuration file.

  • Edit the zappa_settings.json file
  • Provide the aws_region: us-east-1
  • Manage_roles: false
  • role_name:
  • role_arn:

Zappa configuration file


{
    "dev": {
	"aws_region": "us-east-1",
        "django_settings": "archerysecurity.settings.base",
        "profile_name": "default",
        "project_name": "archerysec-aws",
        "runtime": "python2.7",
        "s3_bucket": "zappa-test",
	"manage_roles": false,
	"role_name": "archerysectest",
	"role_arn": "arn:aws:iam::868167788748:role/archerysectest",
    }
}

Now let us deploy the app

$ zappa deploy dev


Now you need to add your AWS API Gateway host as allowed host in application settings file and disable debugging by marking it as False.

  • Edit settings file: archerysecurity/settings/base.py
  • DEBUG = False
  • Replace * with your host in Allowed host ALLOWED_HOSTS = ['your_api_gateway_host']


  • Add below lines in setting file archerysecurity/settings/base.py

FORCE_SCRIPT_NAME = "/dev/"

LOGIN_URL = FORCE_SCRIPT_NAME + 'login/'

  • Update code by running command zappa update dev

Awsome!! we have successfully deployed ArcherySec tool on AWS Lambda function.

Application can be accessible on API Gateway URL: https://yoururl.execute-api.us-east-1.amazonaws.com/dev/login/?next=/dev/


Noticed that the application is not loading static files. We need to use AWS S3 bucket to upload static files and load statics file using django django-s3-storage.

Static file on AWS S3 Bucket

Before setting up django we need to configure S3 bucket.

To serve the static file we need to enable CORS for the S3 bucket, which enables browsers to get resources from different urls.

  • Go to S3 bucket and then Permissions.
  • Click on CORS Configuration.
  • Paste the below lines.
<CORSConfiguration>
        <CORSRule>
            <AllowedOrigin>*</AllowedOrigin>
            <AllowedMethod>GET</AllowedMethod>
            <MaxAgeSeconds>3000</MaxAgeSeconds>
            <AllowedHeader>Authorization</AllowedHeader>
        </CORSRule>
</CORSConfiguration>

Configure ArcherySec for S3

  • Install django-s3-storage

$ pip install django-s3-storage

  • and also add it in the requirements.txt file.
  • Now update the settings.py file to add django_s3_storage to INSTALLED_APPS

INSTALLED_APPS = (
          ...,
          'django_s3_storage',
     )
  • and also add these lines at the bottom

S3_BUCKET = "itestzappa-test"

STATICFILES_STORAGE = "django_s3_storage.storage.StaticS3Storage"

AWS_S3_BUCKET_NAME_STATIC = S3_BUCKET

STATIC_URL = "https://%s.s3.amazonaws.com/" % S3_BUCKET

  • Now load the static files using below command

$ python manage.py collectstatic --noinput

It’s take a while to upload static files on AWS s3 bucket.


Once the static files are uploaded on s3 bucket you can update using Zappa command. ` zappa update dev `

and after updating zappa, let us check by refreshing the page.


Database Connection

Django supports both MySQL and PostgreSQL. Hence you can integrated ArcherySec database either of any one databases.

Setup PostgreSQL on AWS RDS

  • Go to AWS RDS
  • Create new database
  • Select PostgreSQL


  • Check on “Only enable options eligible for RDS Free Usage Tier”
  • Provide “DB instance identifier”, “Master username”, “Master password”


  • Setup “Network & Security” which should be same of ArcherySec Lambda function


  • Add database details in ArcherySec settings file once database has been created.
  • Edit file archerysecurity/settings/base.py

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'archerysecdbname',
        'USER': 'administrator',
        'PASSWORD': 'this_is_not_a_good_password',
        'HOST': 'archerysec-db.cr3erer3sjdlk.us-east-1.rds.amazonaws.com',
        'PORT': '5432',
    }

}

Zappa Settings

Now we add the VPC configuration to our Zappa settings file so that the lambda functions can connect to the database.

{
    "dev": {
        "django_settings": "archerysecurity.settings.base", 
        "s3_bucket": "archerysectest-code",
        "aws_region": "us-east-1",
        "vpc_config" : {
           "SubnetIds": [ "subnet-f3446aba","subnet-c5b8c79e" ], // use the private subnet
           "SecurityGroupIds": [ "sg-9a9a1dfc" ]

       }

   }
}

Migrate database

Create your migrations and push the updated code. ` zappa update dev `

Invoke the zappa manage command: zappa manage dev migrate


  • after updating zappa, let us check by refreshing the page.
  • Create new user using Signup form https://your_apigateway_end.execute-api.us-east-1.amazonaws.com/dev/webscanners/signup/


  • Login into account.


Restrict ArcherySec signup page from publicly.

Currently ArcherySec signup page accessible publicly and we need to restrict signup page from publicly.

  • Edit file webscanners/web_views.py
  • Search def signup function and comment @public decorator


  • Edit file archeryapi/views.py
  • Search def class CreateUsers and comment @public decorator


  • Edit file archerysecurity/settings/base.py
  • Search STRONGHOLD_PUBLIC_URLS
  • Comment r'^/api/createuser/$',


  • Update code by running command zappa update dev

Congratulations!! Now ArcherySec has been deployed as serverless, you can check lambda logs using zappa tail

References

comments powered by Disqus

Explore →