Previous DevOps blog posts have covered the following use cases for infrastructure and application deployment automation:

• Deploy to Production Using AWS CodeBuild and the AWS Developer Tools Suite: Deploying a simple Java application in an in-place deployment model using AWS CodeCommit, AWS CodeBuild, and AWS CodeDeploy orchestrated by AWS CodePipeline.
• Performing Blue/Green Deployments with AWS CodeDeploy and Auto Scaling Groups: Extending the CI/CD model by using the CodeDeploy blue/green deployment feature to create a production environment and make it easier to roll back to the previous environment if problems arise.
• Streamline AMI Maintenance and Patching Using Amazon EC2 Systems Manager | Automation: Using EC2 Systems Manager and automation to patch, update agents, or bake applications into an Amazon Machine Image (AMI) and avoid the time and effort associated with manual image updates.

An AMI provides the information required to launch an instance, which is a virtual server in the cloud. You can use one AMI to launch as many instances as you need. It is security best practice to customize and harden your base AMI with required operating system updates and, if you are using AWS native services for continuous security monitoring and operations, you are strongly encouraged to bake into the base AMI agents such as those for Amazon EC2 Systems Manager (SSM), Amazon Inspector, CodeDeploy, and CloudWatch Logs. A customized and hardened AMI is often referred to as a “golden AMI.” The use of golden AMIs to create EC2 instances in your AWS environment allows for fast and stable application deployment and scaling, secure application stack upgrades, and versioning.

In this post, using the DevOps automation capabilities of Systems Manager, AWS developer tools (CodePipeLine, CodeDeploy, CodeCommit, CodeBuild), I will show you how to use AWS CodePipeline to orchestrate the end-to-end blue/green deployments of a golden AMI and application code. Systems Manager Automation is a powerful security feature for enterprises that want to mature their DevSecOps practices.

Here are the high-level phases and primary services covered in this use case:

You can access the source code for the sample used in this post here: https://github.com/awslabs/automating-governance-sample/tree/master/Bluegreen-AMI-Application-Deployment-blog.

This sample will create a pipeline in AWS CodePipeline with the building blocks to support the blue/green deployments of infrastructure and application. The sample includes a custom Lambda step in the pipeline to execute Systems Manager Automation to build a golden AMI and update the Auto Scaling group with the golden AMI ID for every rollout of new application code. This guarantees that every new application deployment is on a fully patched and customized AMI in a continuous integration and deployment model. This enables the automation of hardened AMI deployment with every new version of application deployment.

We will build and run this sample in three parts.

Part 1: Setting up the AWS developer tools and deploying a base web application

Part 1 of the AWS CloudFormation template creates the initial Java-based web application environment in a VPC. It also creates all the required components of Systems Manager Automation, CodeCommit, CodeBuild, and CodeDeploy to support the blue/green deployments of the infrastructure and application resulting from ongoing code releases.

Part 1 of the AWS CloudFormation stack creates these resources:

• A Java-based web application running on EC2 instances loaded with CodeDeploy agents in an Auto Scaling group behind an Elastic Load Balancing load balancer.
• A Systems Manager Automation document that patches the supplied base AMI and creates the golden AMI.
• A CodeCommit repository to securely store code and files for your application.
• A CodeBuild project with configuration details about how AWS CodeBuild builds your source code.
• A CodeDeploy deployment group with Auto Scaling group details about the web application EC2 instances.
• A CodeDeploy application with a deployment group configured with the Automatically copy Auto Scaling group setting.
• The following Lambda functions:
  1. A function to get the Amazon-provided source AMI ID based on region and architecture.
  2. A function to update the Systems Manager parameter with the golden AMI ID.
  3. A function to update the CodeDeploy deployment group with required blue/green configurations. (Currently AWS CloudFormation does not support creating a deployment group with blue/green deployment configurations.)

After Part 1 of the AWS CloudFormation stack creation is complete, go to the Outputs tab and click the Elastic Load Balancing link. You will see the following home page for the base web application:

Make sure you have all the outputs from the Part 1 stack handy. You need to supply them as parameters in Part 3 of the stack.

Part 2: Setting up your CodeCommit repository

In this part, you will commit and push your sample application code into the CodeCommit repository created in Part 1. To access the initial git commands to clone the empty repository to your local machine, click Connect to go to the AWS CodeCommit console. Make sure you have the IAM permissions required to access AWS CodeCommit from command line interface (CLI).

After you’ve cloned the repository locally, download the sample application files from the part2 folder of the Git repository and place the files directly into your local repository. Do not include the aws-codedeploy-sample-tomcat folder. Go to the local directory and type the following commands to commit and push the files to the CodeCommit repository:

git add .
git commit -a -m "add all files from the AWS Java Tomcat CodeDeploy application"
git push

After all the files are pushed successfully, the repository should look like this:

Part 3: Setting up CodePipeline to enable blue/green deployments     

Part 3 of the AWS CloudFormation template creates the pipeline in AWS CodePipeline and all the required components.

a) Source: The pipeline is triggered by any change to the CodeCommit repository.

b) BuildGoldenAMI: This Lambda step executes the Systems Manager Automation document to build the golden AMI. After the golden AMI is successfully created, a new launch configuration with the new AMI details will be updated into the Auto Scaling group of the application deployment group. You can watch the progress of the automation in the EC2 console from the Systems Manager –> Automations menu.

c) Build: This step uses the application build spec file to build the application build artifact. Here are the CodeBuild execution steps and their status:

d) Deploy: This step clones the Auto Scaling group, launches the new instances with the new AMI, deploys the application changes, reroutes the traffic from the elastic load balancer to the new instances and terminates the old Auto Scaling group. You can see the execution steps and their status in the CodeDeploy console.

After the CodePipeline execution is complete, you can access the application by clicking the Elastic Load Balancing link. You can find it in the output of Part 1 of the AWS CloudFormation template. Any consecutive commits to the application code in the CodeCommit repository trigger the pipelines and deploy the infrastructure and code with an updated AMI and code.

If you have feedback about this post, add it to the Comments section below. If you have questions about implementing the example used in this post, open a thread on the Developer Tools forum.

About the author

Ramesh Adabala is a Solutions Architect in Southeast Enterprise Solution Architecture team at Amazon Web Services.

Sam Dengler is a Solutions Architect at Amazon Web Services

Summary

Amazon CloudWatch Events now supports AWS CodeCommit Repository State Changes event types for activities like pushing new code to a repository. Using these new event types, customers can build Amazon CloudWatch Event rules to match AWS CodeCommit events and route them to one or more targets like an Amazon SNS Topic, AWS Step Functions state machine, or AWS Lambda function to trigger automated workflows to process repository changes.

In this blog, I will provide three examples for using AWS Lambda and JGit to build cost-effective serverless solutions to securely process AWS CodeCommit repository state changes:

• Replicate CodeCommit Repository
• Enforce Git Commit Message Policy
• Backup Git Archive to Amazon S3

Source code and Amazon CloudFormation templates for the examples are located in the following GitHub repository: https://github.com/awslabs/serverless-codecommit-examples.

AWS CodeCommit CloudWatch Events

Amazon CloudWatch Events delivers a near real-time stream of system events that describe changes in AWS resources. Below is an example Amazon CloudWatch Event for one of the new AWS CodeCommit Repository State Changes event types, referenceUpdated. Any change to the repository will trigger a referenceUpdated event, however triggers for particular branches can be filtered using the referenceType and referenceName fields in the event details.

{
    "version": "0",
    "id": "01234567-0123-0123-0123-012345678901",
    "detail-type": "CodeCommit Repository State Change",
    "source": "aws.codecommit",
    "account": "123456789012",
    "time": "2017-06-12T10:23:43Z",
    "region": "us-east-1",
    "resources": [
        "arn:aws:codecommit:us-east-1:123456789012:myRepo"
    ],
    "detail": {
        "event": "referenceUpdated",
        "repositoryName": "myRepo",
        "referenceType": "head",
        "referenceName": "myBranch",
        "commitId": "3e5983EXAMPLE",
        "oldCommitId": "1a7813EXAMPLE"
    }
}

We will use the Amazon CloudWatch Event’s fields to create a pattern to match the events for which we will trigger a target, in this case an AWS Lambda function.

Accessing AWS CodeCommit using HTTPS URLs

The HTTPS URL method for accessing an AWS CodeCommit repository is particularly suited to a serverless solution because an Amazon Lambda execution container already provides temporary AWS IAM key credentials associated to the function’s AWS IAM Execution Role. The function’s Execution Role is associated to one or more AWS IAM Policies, in which you specify permissions allowing the function to access AWS resources. For each function, we will limit the AWS IAM polices to only the AWS CodeCommit repository, Amazon S3 bucket, or Amazon SNS topics, following the AWS best practice to grant least privileged access.

For example, the AWS IAM Policy snippet below restricts the Amazon Lambda function to pull from the source repository and push to the target repository.

Policies:
  - Version: '2012-10-17'
    Statement:
      - Effect: Allow
        Resource: !Sub 'arn:aws:codecommit:${AWS::Region}:${AWS::AccountId}:${SourceRepositoryName}'
        Action:
          - 'codecommit:GetRepository'
          - 'codecommit:GitPull'
      - Effect: Allow
        Resource: !Sub 'arn:aws:codecommit:${TargetRepositoryRegion}:${AWS::AccountId}:${TargetRepositoryName}'
        Action:
          - 'codecommit:GetRepository'
          - 'codecommit:GitPush'

When using the HTTPS URL access method, a credential helper is configured for the Git client, which executes the “aws codecommit credential-helper” command to provide a SigV4 compatible user name and password using AWS IAM credentials (see more). When using JGit as the Git client, a CredentialsProvider can be supplied to Git commands to achieve the same result.

The Spring Cloud Config project provides an implementation of the JGit CredentialsProvider for AwsCodeCommit (source), which conveniently uses the AWS DefaultAWSCredentialsProviderChain to discover AWS credentials in the standard priority order supported by Amazon Lambda. The AwsCodeCommit.calculateCodeCommitPassword method is particularly interesting to review as an example of SigV4 transformation logic.

Cloning a repository is repeated across examples, and the functionality has been delegated to a supporting CloneCommandBuilder Class below.

public class CloneCommandBuilder {

    private File directory;

    public CloneCommandBuilder() throws IOException {
        directory = Files.createTempDirectory(null).toFile();
    }

    public CloneCommand buildCloneCommand(String sourceUrl) {
        return buildCloneCommand(sourceUrl, new AwsCodeCommitCredentialProvider());
    }

    public CloneCommand buildCloneCommand(String sourceUrl,
            AwsCodeCommitCredentialProvider credentialsProvider) {

        return new CloneCommand().setDirectory(directory)
                .setURI(sourceUrl)
                .setCredentialsProvider(credentialsProvider)
                .setBare(true);
    }
}

Next we’ll look at some examples to process the repository events using Amazon Lambda.

Example 1: Replicate CodeCommit Repository

Customers often need to replicate commits from one repository to another to support disaster recovery or cross region CI/CD pipelines. In this example, the Amazon Lambda function will clone a repository from the source and push to the target. This example is intended to update an existing target repository, which should not be empty before configuring replication prior to configuring replication.

Please note, Amazon Lambda functions are limited to 1.5GB memory, 512MB ephemeral disk capacity (“/tmp” space), and a 5 minute execution time. If your repository is unable to be processed within these limits, please see the Replicating and Automating Sync-Ups for a Repository with AWS CodeCommit blog article for an alternative approach to replicate repositories using an Amazon EC2 instance.

Let’s take a look at some code!

public class ReplicateRepositoryHandler
        implements RequestHandler<CodeCommitEvent, HandlerResponse> {

    private static Logger logger = Logger.getLogger(ReplicateRepositoryHandler.class);
    private final String targetUrl;
    private final AwsCodeCommitCredentialProvider credentialsProvider;

    public ReplicateRepositoryHandler() {
        String targetName = System.getenv("TARGET_REPO_NAME");
        String targetRegion = System.getenv("TARGET_REPO_REGION");

        CodeCommitMetadata target = new CodeCommitMetadata(targetName, targetRegion);
        targetUrl = target.getCloneUrlHttp();
        credentialsProvider = new AwsCodeCommitCredentialProvider();
    }

    // ...

On instantiation, the Amazon Lambda function discovers the target AWS CodeCommit repository HTTPS URL by querying the repository metadata using the target repository name and region. This discovery process is repeated across the examples, and the code has been delegated to the CodeCommitMetadata class below.

public class CodeCommitMetadata {

    private RepositoryMetadata repositoryMetadata;

    public CodeCommitMetadata(String repoName, String repoRegion) {
        AWSCodeCommitClientBuilder builder = AWSCodeCommitClientBuilder.standard();
        AWSCodeCommit client = builder.withRegion(repoRegion).build();

        GetRepositoryRequest request = new GetRepositoryRequest();
        request.withRepositoryName(repoName);

        GetRepositoryResult result = client.getRepository(request);
        repositoryMetadata = result.getRepositoryMetadata();
    }

    public String getCloneUrlHttp() {
        return repositoryMetadata.getCloneUrlHttp();
    }
}

When the Amazon Lambda function is triggered by the AWS CloudWatch Event, the source repository name and region in the event are used to discover the source repository HTTPS URL. We use JGit to clone the source repository from this URL into a local repository stored in a temporary directory in the Amazon Lambda execution container.

public HandlerResponse handleRequest(CodeCommitEvent event, Context context) {
    try {
        String sourceName = event.getDetail().getRepositoryName();
        String sourceRegion = event.getRegion();
   
        // clone source repository
        CodeCommitMetadata source = new CodeCommitMetadata(sourceName, sourceRegion);
        String sourceUrl = source.getCloneUrlHttp();
        Git git = new CloneCommandBuilder().buildCloneCommand(sourceUrl).call();

        // ...

Once we’ve cloned the local repository to the Amazon Lambda execution container, the last step is to set the target AWS CodeCommit repository as a new remote location and push the local references using the reference specification “+refs/*:refs/*”.

// push target repository
git.push().setCredentialsProvider(credentialsProvider)
          .setRemote(targetUrl)
          .setRefSpecs(new RefSpec("+refs/*:refs/*"))
          .call();
// ...

In the next example, we’ll review how we can build a Lambda function to enforce commit message policies.

Example 2: Enforce Git Commit Message Policy

Some customers choose to enforce policies on a Git repository to maintain code quality. In this example, we use the same tools described above to clone a repository and validate the commit messages from the Git log using a regular expression.

public class PolicyEnforcerHandler implements RequestHandler<CodeCommitEvent, HandlerResponse> {

    private static Logger logger = LoggerFactory.getLogger(ArchiveRepositoryHandler.class);

    private final String mainBranch;
    private final String snsTopicArn;
    private final Pattern pattern;
    private final AmazonSNS snsClient;

    public PolicyEnforcerHandler() {
        mainBranch = System.getenv("MAIN_BRANCH_NAME");
        snsTopicArn = System.getenv("SNS_TOPIC_ARN");

        String messageRegex = System.getenv("MESSAGE_REGEX");
        pattern = Pattern.compile(messageRegex);

        String snsRegion = snsTopicArn.split(":")[3];
        snsClient = AmazonSNSClientBuilder.standard().withRegion(snsRegion).build();
    }

    // ...

On instantiation, the Amazon Lambda function compiles the regular expression for message validation and creates an Amazon SNS client to send notifications.

@Override
public HandlerResponse handleRequest(CodeCommitEvent event, Context context) {
    String sourceName = event.getDetail().getRepositoryName();
    String sourceRegion = event.getRegion();
    String commitId = event.getDetail().getCommitId();
    String oldCommitId = event.getDetail().getOldCommitId();

    try {
        // clone source repository
        CodeCommitMetadata source = new CodeCommitMetadata(sourceName, sourceRegion);
        String sourceUrl = source.getCloneUrlHttp();
        Git git = new CloneCommandBuilder().buildCloneCommand(sourceUrl).call();

        // ...

When the Amazon Lambda function is triggered by the AWS CloudWatch Event, the process to clone the repository is the same, discovering the AWS CodeCommit HTTPS URL from the AWS CloudWatch Event and cloning a bare Git repository to the Amazon Lambda execution container.

// use the OldCommitId, or default to the main branch
String toGitReference = Optional.ofNullable(oldCommitId).orElse(mainBranch);
Repository repository = git.getRepository();
ObjectId to = repository.resolve(toGitReference);
ObjectId from = repository.resolve(commitId);

// ...

JGit RevWalk is used to determine the range of commits over which to validate the message policy. When commits are added to an existing branch, AWS CodeCommit will emit an referenceUpdated event, which includes commitId and oldCommitId fields that establish the range of commits.

When commits are added to a new branch, AWS CodeCommit will emit a referenceCreated event, which includes a commitId but not the oldCommitId. In this case, we will use the main branch name to determine the common ancestry of the commit chains, called the merge base, in order to establish the range of commits.

// create a RevWalk and set the range of commits
try (RevWalk walk = new RevWalk(repository)) {
    walk.markStart(walk.parseCommit(from));
    walk.markUninteresting(walk.parseCommit(to));

    // iterate the list of commits and validate each message
    for (RevCommit commit : walk) {
        Matcher matcher = pattern.matcher(commit.getShortMessage());

        // publish a message to the topic if the message does not match
        if (!matcher.find()) {
            String message = buildMessage(commit);
            logger.info(message);
            snsClient.publish(snsTopicArn, message);
        }
    }

    walk.dispose();
}

// ...

Once the range has been established, we iterate the list of commit messages, testing each against the message policy regular expression. If the message does not match the regular expression, then it is out of compliance from the policy, and a message is published to the Amazon SNS topic for notification.

In the next example, I’ll review how to backup an archive of the files in a Git repository.

Example 3: Backup Git Archive to Amazon S3

The previous examples have focused on the bare Git repository objects, however there are some use cases for processing the files in the Git repository at a particular reference. In this example, I’ll build a Lambda function to create a zip of the files in the repository and store it in Amazon S3 as a backup.

public class ArchiveRepositoryHandler
        implements RequestHandler<CodeCommitEvent, HandlerResponse> {

    private static Logger logger = LoggerFactory.getLogger(ArchiveRepositoryHandler.class);

    private final String ZIP_FORMAT = "zip"
    private final String targetS3Bucket;
    private final AmazonS3 s3Client;

    public ArchiveRepositoryHandler() {
        targetS3Bucket = System.getenv("TARGET_S3_BUCKET");
        s3Client = AmazonS3ClientBuilder.defaultClient();
        ArchiveCommand.registerFormat(ZIP_FORMAT, new ZipFormat());
    }

    // ...

On instantiation, the Amazon Lambda function creates an Amazon S3 client and registers the ZipFormat with JGit.

@Override
public HandlerResponse handleRequest(CodeCommitEvent event, Context context) {
    String sourceName = event.getDetail().getRepositoryName();
    String sourceRegion = event.getRegion();
    String commitId = event.getDetail().getCommitId();

    try {
        // clone source repository
        CodeCommitMetadata source = new CodeCommitMetadata(sourceName, sourceRegion);
        String sourceUrl = source.getCloneUrlHttp();
        Git git = new CloneCommandBuilder().buildCloneCommand(sourceUrl).call();

        // ...

When the Amazon Lambda function is triggered by the AWS CloudWatch Event, the process to clone the repository is the same, discovering the AWS CodeCommit HTTPS URL from the AWS CloudWatch Event and cloning a bare Git repository to the Amazon Lambda execution container.

// create and upload archive for commitId
File file = Files.createTempFile(null, null).toFile();
try (OutputStream out = new FileOutputStream(file)) {
    ObjectId objectId = git.getRepository().resolve(commitId);
    git.archive().setTree(objectId)
                 .setFormat(ZIP_FORMAT)
                 .setOutputStream(out)
                 .call();

    String key = sourceName + "." + commitId + "." + ZIP_FORMAT;
    s3Client.putObject(targetS3Bucket, key, file);
}

// ...

Once the repository has been cloned, we use a JGit ArchiveCommand to create a zip artifact representing the working files of repository at the commit triggering the event. The generated zip artifact is then uploaded to Amazon S3 using the repository name and commit shortId as the key.

Conclusion

AWS CloudWatch Event’s support for AWS CodeCommit Repository State Changes event types opens possibilities to build event-driven source code workflow automation using the same AWS CloudWatch Events service that acts as an event bus across many AWS services. Combining this new capability with Amazon Lambda, the JGit client, and AWS IAM policy controls provides builders with a set of tools to build serverless solutions that securely access AWS resources, scale on demand, and are cost effective.

In this blog, I’ve demonstrated three example solutions built using these tools, however AWS CodeCommit’s integration with AWS CloudWatch Events allows you to integrate with other AWS CloudWatch Events targets, like Amazon SQS or AWS Step Functions.

I encourage you to visit the GitHub repository (https://github.com/awslabs/serverless-codecommit-examples), which has instructions to launch these examples in your own AWS account. Please share your ideas and questions in the comments below, or submit pull requests and issues to the GitHub repository!

In June 2017, AWS CodeBuild announced you can now specify an alternate build specification file name or location in an AWS CodeBuild project.

In this post, I’ll show you how to use different build specification files in the same repository to create different builds. You’ll find the source code for this post in our GitHub repo.

Requirements

The AWS CLI must be installed and configured.

Solution Overview

I have created a C program (cbsamplelib.c) that will be used to create a shared library and another utility program (cbsampleutil.c) to use that library. I’ll use a Makefile to compile these files.

I need to put this sample application in RPM and DEB packages so end users can easily deploy them. I have created a build specification file for RPM. It will use make to compile this code and the RPM specification file (cbsample.rpmspec) configured in the build specification to create the RPM package. Similarly, I have created a build specification file for DEB. It will create the DEB package based on the control specification file (cbsample.control) configured in this build specification.

(more…)

For their continuous integration and continuous deployment (CI/CD) pipeline path, many companies use tools like Jenkins, Chef, and AWS CloudFormation. Usually, the process is managed by two or more teams. One team is responsible for designing and developing an application, CloudFormation templates, and so on. The other team is generally responsible for integration and deployment.

One of the challenges that a CI/CD team has is to validate the CloudFormation templates provided by the development team. Validation provides early warning about any incorrect syntax and ensures that the development team follows company policies in terms of security and the resources created by CloudFormation templates.

In this post, I focus on the validation of AWS CloudFormation templates for syntax as well as in the context of business rules.

(more…)

In CodePipeline Update – Build Continuous Delivery Workflows for CloudFormation Stacks, Jeff Barr discusses infrastructure as code and how to use AWS CodePipeline for continuous delivery. In this blog post, I discuss the continuous delivery of nested CloudFormation stacks using AWS CodePipeline, with AWS CodeCommit as the source repository and AWS CodeBuild as a build and testing tool. I deploy the stacks using CloudFormation change sets following a manual approval process.

Here’s how to do it:

In AWS CodePipeline, create a pipeline with four stages:

• Source (AWS CodeCommit)
• Build and Test (AWS CodeBuild and AWS CloudFormation)
• Staging (AWS CloudFormation and manual approval)
• Production (AWS CloudFormation and manual approval)

(more…)

Written by AWS Solutions Architects Jason Barto and Heitor Lessa

 
In Part 1 of this post, we described how AWS CodeBuild, AWS CodeCommit, and HashiCorp Packer can be used to build an Amazon Machine Image (AMI) from the latest version of Amazon Linux. In this post, we show how to use AWS CodePipeline, AWS CloudFormation, and Amazon CloudWatch Events to continuously ship new AMIs. We use Ansible by Red Hat to harden the OS on the AMIs through a well-known set of security controls outlined by the Center for Internet Security in its CIS Amazon Linux Benchmark.

You’ll find the source code for this post in our GitHub repo.

At the end of this post, we will have the following architecture:

Requirements

 
To follow along, you will need Git and a text editor. Make sure Git is configured to work with AWS CodeCommit, as described in Part 1.

Technologies

 
In addition to the services and products used in Part 1 of this post, we also use these AWS services and third-party software:

AWS CloudFormation gives developers and systems administrators an easy way to create and manage a collection of related AWS resources, provisioning and updating them in an orderly and predictable fashion.

Amazon CloudWatch Events enables you to react selectively to events in the cloud and in your applications. Specifically, you can create CloudWatch Events rules that match event patterns, and take actions in response to those patterns.

AWS CodePipeline is a continuous integration and continuous delivery service for fast and reliable application and infrastructure updates. AWS CodePipeline builds, tests, and deploys your code every time there is a code change, based on release process models you define.

Amazon SNS is a fast, flexible, fully managed push notification service that lets you send individual messages or to fan out messages to large numbers of recipients. Amazon SNS makes it simple and cost-effective to send push notifications to mobile device users or email recipients. The service can even send messages to other distributed services.

Ansible is a simple IT automation system that handles configuration management, application deployment, cloud provisioning, ad-hoc task-execution, and multinode orchestration.

Getting Started

 
We use CloudFormation to bootstrap the following infrastructure:

Region	AMI Builder Launch Template
N. Virginia (us-east-1)
Ireland (eu-west-1)

After launching the CloudFormation template linked here, we will have a pipeline in the AWS CodePipeline console. (Failed at this stage simply means we don’t have any data in our newly created AWS CodeCommit Git repository.)

Next, we will clone the newly created AWS CodeCommit repository.

If this is your first time connecting to a AWS CodeCommit repository, please see instructions in our documentation on Setup steps for HTTPS Connections to AWS CodeCommit Repositories.

To clone the AWS CodeCommit repository (console)

1. From the AWS Management Console, open the AWS CloudFormation console.
2. Choose the AMI-Builder-Blogpost stack, and then choose Output.
3. Make a note of the Git repository URL.
4. Use git to clone the repository.

For example: git clone https://git-codecommit.eu-west-1.amazonaws.com/v1/repos/AMI-Builder_repo

To clone the AWS CodeCommit repository (CLI)

# Retrieve CodeCommit repo URL
git_repo=$(aws cloudformation describe-stacks --query 'Stacks[0].Outputs[?OutputKey==`GitRepository`].OutputValue' --output text --stack-name "AMI-Builder-Blogpost")

# Clone repository locally
git clone ${git_repo}

Bootstrap the Repo with the AMI Builder Structure

 
Now that our infrastructure is ready, download all the files and templates required to build the AMI.

• Download ami-builder-packer ZIP.
• Extract and copy the contents to the Git repo.

Your local Git repo should have the following structure:

.
├── ami_builder_event.json
├── ansible
├── buildspec.yml
├── cloudformation
├── packer_cis.json

Next, push these changes to AWS CodeCommit, and then let AWS CodePipeline orchestrate the creation of the AMI:

git add .
git commit -m "My first AMI"
git push origin master

AWS CodeBuild Implementation Details

 
While we wait for the AMI to be created, let’s see what’s changed in our AWS CodeBuild buildspec.yml file:

...
phases:
  ...
  build:
    commands:
      ...
      - ./packer build -color=false packer_cis.json | tee build.log
  post_build:
    commands:
      - egrep "${AWS_REGION}\:\sami\-" build.log | cut -d' ' -f2 > ami_id.txt
      # Packer doesn't return non-zero status; we must do that if Packer build failed
      - test -s ami_id.txt || exit 1
      - sed -i.bak "s/<<AMI-ID>>/$(cat ami_id.txt)/g" ami_builder_event.json
      - aws events put-events --entries file://ami_builder_event.json
      ...
artifacts:
  files:
    - ami_builder_event.json
    - build.log
  discard-paths: yes

In the build phase, we capture Packer output into a file named build.log. In the post_build phase, we take the following actions:

1. Look up the AMI ID created by Packer and save its findings to a temporary file (ami_id.txt).
2. Forcefully make AWS CodeBuild to fail if the AMI ID (ami_id.txt) is not found. This is required because Packer doesn’t fail if something goes wrong during the AMI creation process. We have to tell AWS CodeBuild to stop by informing it that an error occurred.
3. If an AMI ID is found, we update the ami_builder_event.json file and then notify CloudWatch Events that the AMI creation process is complete.
4. CloudWatch Events publishes a message to an SNS topic. Anyone subscribed to the topic will be notified in email that an AMI has been created.

Lastly, the new artifacts phase instructs AWS CodeBuild to upload files built during the build process (ami_builder_event.json and build.log) to the S3 bucket specified in the Outputs section of the CloudFormation template. These artifacts can then be used as an input artifact in any later stage in AWS CodePipeline.

For information about customizing the artifacts sequence of the buildspec.yml, see the Build Specification Reference for AWS CodeBuild.

CloudWatch Events Implementation Details

 
CloudWatch Events allow you to extend the AMI builder to not only send email after the AMI has been created, but to hook up any of the supported targets to react to the AMI builder event. This event publication means you can decouple from Packer actions you might take after AMI completion and plug in other actions, as you see fit.

For more information about targets in CloudWatch Events, see the CloudWatch Events API Reference.

In this case, CloudWatch Events should receive the following event, match it with a rule we created through CloudFormation, and publish a message to SNS so that you can receive an email.

Example CloudWatch custom event

[
        {
            "Source": "com.ami.builder",
            "DetailType": "AmiBuilder",
            "Detail": "{ \"AmiStatus\": \"Created\"}",
            "Resources": [ "ami-12cd5guf" ]
        }
]

Cloudwatch Events rule

{
  "detail-type": [
    "AmiBuilder"
  ],
  "source": [
    "com.ami.builder"
  ],
  "detail": {
    "AmiStatus": [
      "Created"
    ]
  }
}

Example SNS message sent in email

{
    "version": "0",
    "id": "f8bdede0-b9d7...",
    "detail-type": "AmiBuilder",
    "source": "com.ami.builder",
    "account": "<<aws_account_number>>",
    "time": "2017-04-28T17:56:40Z",
    "region": "eu-west-1",
    "resources": ["ami-112cd5guf "],
    "detail": {
        "AmiStatus": "Created"
    }
}

Packer Implementation Details

 
In addition to the build specification file, there are differences between the current version of the HashiCorp Packer template (packer_cis.json) and the one used in Part 1.

Variables

  "variables": {
    "vpc": "{{env `BUILD_VPC_ID`}}",
    "subnet": "{{env `BUILD_SUBNET_ID`}}",
         “ami_name”: “Prod-CIS-Latest-AMZN-{{isotime \”02-Jan-06 03_04_05\”}}”
  },

• ami_name: Prefixes a name used by Packer to tag resources during the Builders sequence.
• vpc and subnet: Environment variables defined by the CloudFormation stack parameters.

We no longer assume a default VPC is present and instead use the VPC and subnet specified in the CloudFormation parameters. CloudFormation configures the AWS CodeBuild project to use these values as environment variables. They are made available throughout the build process.

That allows for more flexibility should you need to change which VPC and subnet will be used by Packer to launch temporary resources.

Builders

  "builders": [{
    ...
    "ami_name": “{{user `ami_name`| clean_ami_name}}”,
    "tags": {
      "Name": “{{user `ami_name`}}”,
    },
    "run_tags": {
      "Name": “{{user `ami_name`}}",
    },
    "run_volume_tags": {
      "Name": “{{user `ami_name`}}",
    },
    "snapshot_tags": {
      "Name": “{{user `ami_name`}}",
    },
    ...
    "vpc_id": "{{user `vpc` }}",
    "subnet_id": "{{user `subnet` }}"
  }],

We now have new properties (*_tag) and a new function (clean_ami_name) and launch temporary resources in a VPC and subnet specified in the environment variables. AMI names can only contain a certain set of ASCII characters. If the input in project deviates from the expected characters (for example, includes whitespace or slashes), Packer’s clean_ami_name function will fix it.

For more information, see functions on the HashiCorp Packer website.

Provisioners

  "provisioners": [
    {
        "type": "shell",
        "inline": [
            "sudo pip install ansible"
        ]
    }, 
    {
        "type": "ansible-local",
        "playbook_file": "ansible/playbook.yaml",
        "role_paths": [
            "ansible/roles/common"
        ],
        "playbook_dir": "ansible",
        "galaxy_file": "ansible/requirements.yaml"
    },
    {
      "type": "shell",
      "inline": [
        "rm .ssh/authorized_keys ; sudo rm /root/.ssh/authorized_keys"
      ]
    }

We used shell provisioner to apply OS patches in Part 1. Now, we use shell to install Ansible on the target machine and ansible-local to import, install, and execute Ansible roles to make our target machine conform to our standards.

Packer uses shell to remove temporary keys before it creates an AMI from the target and temporary EC2 instance.

Ansible Implementation Details

 
Ansible provides OS patching through a custom Common role that can be easily customized for other tasks.

CIS Benchmark and Cloudwatch Logs are implemented through two Ansible third-party roles that are defined in ansible/requirements.yaml as seen in the Packer template.

The Ansible provisioner uses Ansible Galaxy to download these roles onto the target machine and execute them as instructed by ansible/playbook.yaml.

For information about how these components are organized, see the Playbook Roles and Include Statements in the Ansible documentation.

The following Ansible playbook (ansible</playbook.yaml) controls the execution order and custom properties:

---
- hosts: localhost
  connection: local
  gather_facts: true    # gather OS info that is made available for tasks/roles
  become: yes           # majority of CIS tasks require root
  vars:
    # CIS Controls whitepaper:  http://bit.ly/2mGAmUc
    # AWS CIS Whitepaper:       http://bit.ly/2m2Ovrh
    cis_level_1_exclusions:
    # 3.4.2 and 3.4.3 effectively blocks access to all ports to the machine
    ## This can break automation; ignoring it as there are stronger mechanisms than that
      - 3.4.2 
      - 3.4.3
    # CloudWatch Logs will be used instead of Rsyslog/Syslog-ng
    ## Same would be true if any other software doesn't support Rsyslog/Syslog-ng mechanisms
      - 4.2.1.4
      - 4.2.2.4
      - 4.2.2.5
    # Autofs is not installed in newer versions, let's ignore
      - 1.1.19
    # Cloudwatch Logs role configuration
    logs:
      - file: /var/log/messages
        group_name: "system_logs"
  roles:
    - common
    - anthcourtney.cis-amazon-linux
    - dharrisio.aws-cloudwatch-logs-agent

Both third-party Ansible roles can be easily configured through variables (vars). We use Ansible playbook variables to exclude CIS controls that don’t apply to our case and to instruct the CloudWatch Logs agent to stream the /var/log/messages log file to CloudWatch Logs.

If you need to add more OS or application logs, you can easily duplicate the playbook and make changes. The CloudWatch Logs agent will ship configured log messages to CloudWatch Logs.

For more information about parameters you can use to further customize third-party roles, download Ansible roles for the Cloudwatch Logs Agent and CIS Amazon Linux from the Galaxy website.

Committing Changes

 
Now that Ansible and CloudWatch Events are configured as a part of the build process, commiting any changes to the AWS CodeComit Git Repository will triger a new AMI build process that can be followed through the AWS CodePipeline console.

When the build is complete, an email will be sent to the email address you provided as a part of the CloudFormation stack deployment. The email serves as notification that an AMI has been built and is ready for use.

Summary

 
We used AWS CodeCommit, AWS CodePipeline, AWS CodeBuild, Packer, and Ansible to build a pipeline that continuously builds new, hardened CIS AMIs. We used Amazon SNS so that email addresses subscribed to a SNS topic are notified upon completion of the AMI build.

By treating our AMI creation process as code, we can iterate and track changes over time. In this way, it’s no different from a software development workflow. With that in mind, software patches, OS configuration, and logs that need to be shipped to a central location are only a git commit away.

Next Steps

 
Here are some ideas to extend this AMI builder:

• Hook up a Lambda function in Cloudwatch Events to update EC2 Auto Scaling configuration upon completion of the AMI build.
• Use AWS CodePipeline parallel steps to build multiple Packer images.
• Add a commit ID as a tag for the AMI you created.
• Create a scheduled Lambda function through Cloudwatch Events to clean up old AMIs based on timestamp (name or additional tag).
• Implement Windows support for the AMI builder.
• Create a cross-account or cross-region AMI build.

Cloudwatch Events allow the AMI builder to decouple AMI configuration and creation so that you can easily add your own logic using targets (AWS Lambda, Amazon SQS, Amazon SNS) to add events or recycle EC2 instances with the new AMI.

If you have questions or other feedback, feel free to leave it in the comments or contribute to the AMI Builder repo on GitHub.

AWS Service Catalog enables organizations to create and manage catalogs of IT services that are approved for use on AWS. These IT services can include everything from virtual machine images, servers, software, and databases to complete multitier application architectures. You can use AWS Service Catalog to centrally manage commonly deployed IT services. It also helps you achieve consistent governance and meet your compliance requirements, while enabling users to quickly deploy only the approved IT services they need.

However, as the number of Service Catalog portfolios and products increases across an organization, centralized management and scaling can become a challenge. In this blog post, I walk you through a solution that simplifies management of AWS Service Catalog portfolios and related products. This solution also enables portfolio sharing with other accounts, portfolio tagging, and granting access to users. Finally, the solution delivers updates to the products using a continuous delivery in AWS CodePipeline. This enables you to maintain them in version control, thereby adopting “Infrastructure as Code” practices.

Solution overview

1. Authors (developers, operations, architects, etc.) create the AWS CloudFormation templates based on the needs of their organizations. These templates are the reusable artifacts. They can be shared among various teams within the organizations. You can name these templates product-A.yaml or product-B.yaml. For example, if the template creates an Amazon VPC that is based on organization needs, as described in the Amazon VPC Architecture Quick Start, you can save it as product-vpc.yaml.

The authors also define a mapping.yaml file, which includes the list of products that you want to include in the portfolio and related metadata. The mapping.yaml file is the core configuration component of this solution. This file defines your portfolio and its associated permissions and products. This configuration file determines how your portfolio will look in AWS Service Catalog, after the solution deploys it. A sample mapping.yaml is described here. Configuration properties of this mapping.yaml are explained here.

2. Product template files and the mappings are committed to version control. In this example, we use AWS CodeCommit. The folder structure on the file system looks like the following:
   • portfolio-infrastructure (folder name)
     – product-a.yaml
     – product-b.yaml
     – product-c.yaml
     – mapping.yaml
   • portfolio-example (folder name)
     – product-c.yaml
     – product-d.yaml
     – mapping.yaml

   The name of the folder must start with portfolio- because the AWS Lambda function iterates through all folders whose names start with portfolio-, and syncs them with AWS Service Catalog.

   Checking in any code in the repository triggers an AWS CodePipeline orchestration and invokes the Lambda function.

3. The Lambda function downloads the code from version control and iterates through all folders with names that start with portfolio-. The function gets a list of all existing portfolios in AWS Service Catalog. Then it checks whether the display name of the portfolio matches the “name” property in the mapping.yaml under each folder. If the name doesn’t match, a new portfolio is created. If the name matches, the description and owner fields are updated and synced with what is in the file. There must be only one mapping.yaml file in each folder with a name starting with portfolio-.
4. and 5. The Lambda function iterates through the list of products in the mapping.yaml file. If the name of product matches any of the products already associated with the portfolio, a new version of the product is created and is associated with the portfolio. If the name of the product doesn’t match, a new product is created. The CloudFormation template file (as specified in the template property for that product in the mapping file) is uploaded to Amazon S3 with a unique ID. A new version of the product is created and is pointed to the unique S3 path.

Try it out!

Get started using this solution, which is available in this AWSLabs GitHub repository.

1. Clone the repository. It contains the AWS CloudFormation templates that we use in this walkthrough.

git clone https://github.com/awslabs/aws-pipeline-to-service-catalog.git
cd aws-pipeline-to-service-catalog

2. Examine mapping.yaml under the portfolio-infrastructure folder. Replace the account number with the account number with which to share the portfolio. To share the portfolio with multiple other accounts, you can append more account numbers to the list. These account numbers must be valid AWS accounts, and must not include the account number in which this solution is being created. Optionally, edit this file and provide the values you want for the name, description, and owner properties. You can also choose to leave these values as they are, which creates a portfolio with the name, description, and owners described in the file.
3. Optional – If you don’t have the AWS Command Line Interface (AWS CLI) installed, install it as described here. To prepare your access keys or assumed role to make calls to AWS, configure the AWS CLI as described here.
4. Create a pipeline. This orchestrates continuous integration with the AWS CodeCommit repository created in step 2, and continuously syncs AWS Service Catalog with the code.

aws cloudformation deploy --template-file pipeline-to-service-catalog.yaml \
--stack-name service-catalog-sync-pipeline --capabilities CAPABILITY_NAMED_IAM \
--parameter-overrides RepositoryName=blogs-pipeline-to-service-catalog

This creates the following resources.

1. An AWS CodeCommit repository to push the code to. You can get the repository URL to push the code from the outputs of the stack that we just created. Connect, commit, and push code to this repository as described here.

2. 2. An S3 bucket, which holds the built artifacts (CloudFormation templates) and the Lambda function code.
   3. The AWS IAM roles and policies, with least privileges for this solution to work.
   4. An AWS CodeBuild project, which builds the Lambda function. This Python-based Lambda function has the logic, as explained earlier.
   5. A pipeline with the following four stages:
      • Stage-1: Checks out source from the repository created in step 2
      • Stage-2: Builds the Lambda function using AWS CodeBuild, which has the logic to sync the AWS Service Catalog products and portfolios with code.
      • Stage-3: Deploys the Lambda function using CloudFormation.
      • Stage-4: Invokes the Lambda function. Once this stage completes successfully, you see an AWS Service Catalog portfolio and two products created, as shown below.

Optional next steps!

You can deploy the Lambda function as we explained in this post to sync AWS Service Catalog products, portfolios, and permissions across multiple accounts that you own with version control. You can create a secure cross-account continuous delivery pipeline, as explained here. To do this:

1. Delete all the resources created earlier.

aws cloudformation delete-stack -- stack-name service-catalog-sync-pipeline

2. Follow the steps in this blog post. The sample Lambda function, described here, is the same as what I explained in this post.

Conclusion

You can use AWS Lambda to make API calls to AWS Service Catalog to keep portfolios and products in sync with a mapping file. The code includes the CloudFormation templates and the mapping file and folder structure, which resembles the portfolios in AWS Service Catalog. When checked in to an AWS CodeCommit repository, it invokes the Lambda function, orchestrated by AWS CodePipeline.

Just as a herd can move only as fast as its slowest member, companies must increase the speed of all parts of their release process, especially the database change process, which is often manual. One bad database change can bring down an app or compromise data security.

We need to make database code deployment as fast and easy as application release automation, while eliminating risks that cause application downtime and data security vulnerabilities. Let’s take a page from the application development playbook and bring a continuous deployment approach to the database.

By creating a continuous deployment database, you can:

• Discover mistakes more quickly.
• Deliver updates faster and frequently.
• Help developers write better code.
• Automate the database release management process.

The database deployment package can be promoted automatically with application code changes. With database continuous deployment, application development teams can deliver smaller, less risky deployments, making it possible to respond more quickly to business or customer needs.

In our previous post, Building End-to-End Continuous Delivery and Deployment Pipelines in AWS, we walked through steps for implementing a continuous deployment and automated delivery pipeline for your application.

In this post, we walk through steps for building a continuous deployment workflow for databases using AWS CodePipeline (a fully managed continuous delivery service) and Datical DB (a database release automation application). We use AWS CodeCommit for source code control and Amazon RDS for database hosting to demonstrate end-to-end database change management — from check-in to final deployment.

As part of this example, we will show how a database change that does not meet standards is rejected automatically and actionable feedback is provided to the developer. Just like a code unit test, Datical DB evaluates changes and enforces your organization’s standards. In the sample use case, database table indexes of more than three columns are disallowed. In some cases, this type of index can slow performance.

Prerequisites

You’ll need an AWS account, an Amazon EC2 key pair, and administrator-level permissions for AWS Identity and Access Management (IAM), AWS CodePipeline, AWS CodeCommit, Amazon RDS, Amazon EC2, and Amazon S3.

From Datical DB, you’ll need access to software.datical.com portal, your license key, a database, and JDBC drivers. You can request a free trial of Datical here.

Overview

Here are the steps:

1. Install and configure Datical DB.
2. Create an RDS database instance running the Oracle database engine.
3. Configure Datical DB to manage database changes across your software development life cycle (SDLC).
4. Set up database version control using AWS CodeCommit.
5. Set up a continuous integration server to stage database changes.
6. Integrate the continuous integration server with Datical DB.
7. Set up automated release management for your database through AWS CodePipeline.
8. Enforce security governance and standards with the Datical DB Rules Engine.

1. Install and configure Datical DB

Navigate to https://software.datical.com and sign in with your credentials. From the left navigation menu, expand the Common folder, and then open the Datical_DB_Folder. Choose the latest version of the application by reviewing the date suffix in the name of the folder. Download the installer for your platform — Windows (32-bit or 64-bit) or Linux (32-bit or 64-bit).

Verify the JDK Version

In a terminal window, run the following command to ensure you’re running JDK version 1.7.x or later.

# java –version
java version "1.7.0_75"
Java(TM) SE Runtime Environment (build 1.7.0_75-b13)
Java HotSpot(TM) Client VM (build 24.75-b04, mixed mode, sharing)

The Datical DB installer contains a graphical (GUI) and command line (CLI) interface that can be installed on Windows and Linux operating systems.

Install Datical DB (GUI)

1. Double-click on the installer
2. Follow the prompts to install the application.
3. When prompted, type the path to a valid license.

Install JDBC drivers

1. Open the Datical DB application.
2. From the menu, choose Help, and then choose Install New Software.
3. From the Work with drop-down list, choose Database Drivers – http://update.datical.com/drivers/updates.
4. Follow the prompts to install the drivers.

Install Datical DB (CLI)

Datical DB (CLI only) can be installed on a headless Linux system. Select the correct 32-bit or 64-bit Linux installer for your system.

1. Run the installer as root and install it to /usr/local/DaticalDB.

   sudo java -jar ../installers/<Datical Installer>.jar -console

2. Follow the prompts to install the application.
3. When prompted, type the path to a valid license.

Install JDBC drivers

1. Copy JDBC drivers to /usr/local/DaticalDB/jdbc_drivers.

   sudo mkdir /usr/local/DaticalDB/jdbc_drivers
   copy JDBC Drivers from software.datical.com to /usr/local/DaticalDB/jdbc_drivers

2. Copy the license file to /usr/local/DaticalDB/repl.

   sudo cp <license_filename> /usr/local/DaticalDB/repl
   sudo chmod 777 /usr/local/DaticalDB/repl/<license_filename>

2. Create an RDS instance running the Oracle database engine

Datical DB supports database engines like Oracle, MySQL, Microsoft SQL Server, PostgreSQL, and IBM DB2. The example in this post uses a DB instance running Oracle. To create a DB instance running Oracle, follow these steps.
Make sure that you can access the Oracle port (1521) from the location where you will be using Datical DB. Just like SQLPlus or other database management tools, Datical DB must be able to connect to the Oracle port. When you configure the security group for your RDS instance, make sure you can access port 1521 from your location.

3. Manage database changes across the SDLC

This one-time process is required to ensure databases are in sync so that you can manage database changes across the SDLC:

1. Create a Datical DB deployment plan with connections to the databases to be managed.
2. Baseline the first database (DEV/CI). This must be the original or best configured database – your reference database.
3. For each additional database (TEST and PROD):
   a. Compare databases to ensure the application schema are in sync.
   b. Resolve any differences.
   c. Perform a change log sync to get each setup for Datical DB management.

Datical DB creates an initial model change log from one of the databases. It also creates in each database a metadata table named DATABASECHANGELOG that will be used to track the state. Now the databases look like this:

Note: In the preceding figure, the Datical DB model and metadata table are a representation of the actual model.

Create a deployment plan

1. 1. In Datical DB, right-click Deployment Plans, and choose New.
   2. On the New Deployment Plan page, type a name for your project (for example, AWS-Sample-Project), and then choose Next.
   3. Select Oracle 11g Instant Client, type a name for the database (for example, DevDatabase), and then choose Next.
   4. On the following page, provide the database connection information.
      1. For Hostname, enter the RDS endpoint..
      2. Select SID, and then type ORCL.
      3. Type the user name and password used to connect to the RDS instance running Oracle.
      4. Before you choose Finish, choose the Test Connection button.

When Datical DB creates the project, it also creates a baseline snapshot that captures the current state of the database schema. Datical DB stores the snapshot in Datical change sets for future forecasting and modification.

Create a database change set

A change set describes the change/refactoring to apply to the database.
From the AWS-Sample-Project project in the left pane, right-click Change Log, select New, and then select Change Set. Choose the type of change to make, and then choose Next. In this example, we’re creating a table. For Table Name, type a name. Choose Add Column, and then provide information to add one or more columns to the new table. Follow the prompts, and then choose Finish.

The new change set will be added at the end of your current change log. You can tag change sets with a sprint label. Depending on the environment, changes can be deployed based on individual labels or by the higher-level grouping construct.
Datical DB also provides an option to load SQL scripts into a database, where the change sets are labeled and captured as objects. This makes them ready for deployment in other environments.

Best practices for continuous delivery

Change sets are stored in an XML file inside the Datical DB project. The file, changelog.xml, is stored inside the Changelog folder. (In the Datical DB UI, it is called Change Log.)

Just like any other files stored in your source code repository, the Datical DB change log can be branched and merged to support agile software development, where individual work spaces are isolated until changes are merged into the parent branch.

To implement this best practice, your Datical DB project should be checked into the same location as your application source code. That way, branches and merges will be applied to your Datical DB project automatically. Use unique change set IDs to avoid collisions with other scrum teams.

4. Set up database version control using AWS CodeCommit

To create a new CodeCommit repository, follow these steps.

Note: On some versions of Windows and Linux, you might see a pop-up dialog box asking for your user name and password. This is the built-in credential management system, but it is not compatible with the credential helper for AWS CodeCommit. Choose Cancel.

Commit the contents located in the Datical working directory (for example, ~/datical/AWS-Sample-Project) to the AWS CodeCommit repository.

5. Set up a continuous integration server to stage database changes

In this example, Jenkins is the continuous integration server. To create a Jenkins server, follow these steps. Be sure your instance security group allows port 8080 access.

sudo wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo

For more information about installing Jenkins, see the Jenkins wiki.

After setup, connect to your Jenkins server, and create a job.

1. Install the following Jenkins plugins:
   For this project, you will need to install the following Jenkins plugins:
   1. AWS CodeCommit plugin
   2. DaticalDB4Jenkins plugin
   3. Hudson Post Build Task plugin
   4. HTML Publisher plugin
2. To configure Jenkins for AWS CodeCommit, follow these steps.
3. To configure Jenkins with Datical DB, navigate to Jenkins, choose Manage Jenkins, and then choose Configure System. In the Datical DB section, provide the requested directory information.

For example:

Add a build step:

Go to your newly created Jenkins project and choose Configure. On the Build tab, under Build, choose Add build step, and choose Datical DB.

In Project Dir, enter the Datical DB project directory (in this example, /var/lib/jenkins/workspace/demo/MyProject). You can use Jenkins environment variables like $WORKSPACE. The first build action is Check Drivers. This allow you to verify that Datical DB and Jenkins are configured correctly.

Choose Save. Choose Build Now to test the configuration.

After you’ve verified the drivers are installed, add forecast and deploy steps.

Add forecast and deploy steps:

Choose Save. Then choose Build Now to test the configuration.

6. Configure the continuous integration server to publish Datical DB reports

In this step, we will configure Jenkins to publish Datical DB forecast and HTML reports. In your Jenkins project, select Delete workspace before build starts.

Add post-build steps

1. Archive the Datical DB reports, logs, and snapshots

To expose Datical DB reports in Jenkins, you must create a post-build task step to copy the forecast and deployment HTML reports to a location easily published, and then publish the HTML reports.

2. Copy the forecast and deploy HTML reports

mkdir /var/lib/jenkins/workspace/Demo/MyProject/report
cp -rv /var/lib/jenkins/workspace/Demo/MyProject/Reports/*/*/*/forecast*/* /var/lib/jenkins/workspace/Demo/MyProject/report 2>NUL
cp -rv /var/lib/jenkins/workspace/Demo/MyProject/Reports/*/*/*/deploy*/deployReport.html /var/lib/jenkins/workspace/Demo/MyProject/report 2>NUL

3. Publish HTML reports

Use the information in the following screen shot. Depending on the location where you configured Jenkins to build, your details might be different.

Note: Datical DB HTML reports use CSS, so update the JENKINS_JAVA_OPTIONS in your config file as follows:

Edit /etc/sysconfig/jenkins and set JENKINS_JAVA_OPTIONS to:

JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true -Dhudson.model.DirectoryBrowserSupport.CSP= "

7. Enable automated release management for your database through AWS CodePipeline

To create an automated release process for your databases using AWS CodePipeline, follow these instructions.

1. Sign in to the AWS Management Console and open the AWS CodePipeline console at http://console.aws.amazon.com/codepipeline.
2. On the introductory page, choose Get started. If you see the All pipelines page, choose Create pipeline.
3. In Step 1: Name, in Pipeline name, type DatabasePipeline, and then choose Next step.
4. In Step 2: Source, in Source provider, choose AWS CodeCommit. In Repository name, choose the name of the AWS CodeCommit repository you created earlier. In Branch name, choose the name of the branch that contains your latest code update. Choose Next step.
   
   
5. In Step 3: Build, chose Jenkins.
   
   

To complete the deployment workflow, follow steps 6 through 9 in the Create a Simple Pipeline Tutorial.

8. Enforce database standards and compliance with the Datical DB Rules Engine

The Datical DB Dynamic Rules Engine automatically inspects the Virtual Database Model to make sure that proposed database code changes are safe and in compliance with your organization’s database standards. The Rules Engine also makes it easy to identify complex changes that warrant further review and empowers DBAs to efficiently focus only on the changes that require their attention. It also provides application developers with a self-service validation capability that uses the same automated build process established for the application. The consistent evaluation provided by the Dynamic Rules Engine removes uncertainty about what is acceptable and empowers application developers to write safe, compliant changes every time.

Earlier, you created a Datical DB project with no changes. To demonstrate rules, you will now create changes that violate a rule.

First, create a table with four columns. Then try to create an index on the table that comprises all four columns. For some databases, having more than three columns in an index can cause performance issues. For this reason, create a rule that will prevent the creation of an index on more than three columns, long before the change is proposed for production. Like a unit test that will fail the build, the Datical DB Rules Engine fails the build at the forecast step and provides feedback to the development team about the rule and the change to fix.

Create a Datical DB rule

To create a Datical DB rule, open the Datical DB UI and navigate to your project. Expand the Rules folder. In this example, you will create a rule in the Forecast folder.

Right-click the Forecast folder, and then select Create Rules File. In the dialog box, type a unique file name for your rule. Use a .drl extension.

.

In the editor window that opens, type the following:

package com.datical.hammer.core.forecast
import java.util.Collection;
import java.util.List;
import java.util.Arrays;
import java.util.ArrayList;
import org.apache.commons.lang.StringUtils;
import org.apache.commons.collections.ListUtils;
import com.datical.db.project.Project;
import com.datical.hammer.core.rules.Response;
import com.datical.hammer.core.rules.Response.ResponseType;

// ************************************* Models *************************************

// Database Models

import com.datical.dbsim.model.DbModel;
import com.datical.dbsim.model.Schema;
import com.datical.dbsim.model.Table;
import com.datical.dbsim.model.Index;
import com.datical.dbsim.model.Column;
import org.liquibase.xml.ns.dbchangelog.CreateIndexType;
import org.liquibase.xml.ns.dbchangelog.ColumnType;


/* @return false if validation fails; true otherwise */

function boolean validate(List columns)
{

// FAIL If more than 3 columns are included in new index
if (columns.size() > 3)
return false;
else
return true;

}

rule "Index Too Many Columns Error"
salience 1
when
$createIndex : CreateIndexType($indexName: indexName, $columns: columns, $tableName: tableName, $schemaName: schemaName)
eval(!validate($columns))
then
String errorMessage = "The new index [" + $indexName + "] contains more than 3 columns.";
insert(new Response(ResponseType.FAIL, errorMessage, drools.getRule().getName()));
end

Save the new rule file, and then right-click the Forecast folder, and select Check Rules. You should see “Rule Validation returned no errors.”

Now check your rule into source code control and request a new build. The build will fail, which is expected. Go back to Datical DB, and change the index to comprise only three columns. After your check-in, you will see a successful deployment to your RDS instance.

The following forecast report shows the Datical DB rule violation:

To implement database continuous delivery into your existing continuous delivery process, consider creating a separate project for your database changes that use the Datical DB forecast functionality at the same time unit tests are run on your code. This will catch database changes that violate standards before deployment.

Summary:

In this post, you learned how to build a modern database continuous integration and automated release management workflow on AWS. You also saw how Datical DB can be seamlessly integrated with AWS services to enable database release automation, while eliminating risks that cause application downtime and data security vulnerabilities. This fully automated delivery mechanism for databases can accelerate every organization’s ability to deploy software rapidly and reliably while improving productivity, performance, compliance, and auditability, and increasing data security. These methodologies simplify process-related overhead and make it possible for organizations to serve their customers efficiently and compete more effectively in the market.

I hope you enjoyed this post. If you have any feedback, please leave a comment below.

About the Authors

Balaji Iyer is an Enterprise Consultant for the Professional Services Team at Amazon Web Services. In this role, he has helped several Fortune 500 customers successfully navigate their journey to AWS. His specialties include architecting and implementing highly scalable distributed systems, serverless architectures, large scale migrations, operational security, and leading strategic AWS initiatives. Before he joined Amazon, Balaji spent more than a decade building operating systems, big data analytics solutions, mobile services, and web applications. In his spare time, he enjoys experiencing the great outdoors and spending time with his family.

Robert Reeves is a Co-Founder & Chief Technology Officer at Datical. In this role, he advocates for Datical’s customers and provides technical architecture leadership. Prior to cofounding Datical, Robert was a Director at the Austin Technology Incubator. At ATI, he provided real-world entrepreneurial expertise to ATI member companies to aid in market validation, product development, and fundraising efforts. Robert cofounded Phurnace Software in 2005. He invented and created the flagship product, Phurnace Deliver, which provides middleware infrastructure management to multiple Fortune 500 companies. As Chief Technology Officer for Phurnace, he led technical evangelism efforts, product vision, and large account technical sales efforts. After BMC Software acquired Phurnace in 2009, Robert served as Chief Architect and lead worldwide technology evangelism.

Written by AWS Solutions Architects Jason Barto and Heitor Lessa

It’s an operational and security best practice to create and maintain custom Amazon Machine Images. Because it’s also a best practice to maintain infrastructure as code, it makes sense to use automated tooling to script the creation and configuration of AMIs that are used to quickly launch Amazon EC2 instances.

In this first of two posts, we’ll use AWS CodeBuild to programmatically create AMIs for use in our environment. As a part of the AMI generation, we will apply OS patches, configure a banner statement, and install some common software, forming a solid base for future Amazon EC2-based deployments.
(more…)

Most organizations create multiple AWS accounts because they provide the highest level of resource and security isolation. In this blog post, I will discuss how to use cross account AWS Identity and Access Management (IAM) access to orchestrate continuous integration and continuous deployment.

Do I need multiple accounts?

If you answer “yes” to any of the following questions you should consider creating more AWS accounts:

• Does your business require administrative isolation between workloads? Administrative isolation by account is the most straightforward way to grant independent administrative groups different levels of administrative control over AWS resources based on workload, development lifecycle, business unit (BU), or data sensitivity.
• Does your business require limited visibility and discoverability of workloads? Accounts provide a natural boundary for visibility and discoverability. Workloads cannot be accessed or viewed unless an administrator of the account enables access to users managed in another account.
• Does your business require isolation to minimize blast radius? Separate accounts help define boundaries and provide natural blast-radius isolation to limit the impact of a critical event such as a security breach, an unavailable AWS Region or Availability Zone, account suspensions, and so on.
• Does your business require a particular workload to operate within AWS service limits without impacting the limits of another workload? You can use AWS account service limits to impose restrictions on a business unit, development team, or project. For example, if you create an AWS account for a project group, you can limit the number of Amazon Elastic Compute Cloud (Amazon EC2) or high performance computing (HPC) instances that can be launched by the account.
• Does your business require strong isolation of recovery or auditing data? If regulatory requirements require you to control access and visibility to auditing data, you can isolate the data in an account separate from the one where you run your workloads (for example, by writing AWS CloudTrail logs to a different account).
• Do your workloads depend on specific instance reservations to support high availability (HA) or disaster recovery (DR) capacity requirements? Reserved Instances (RIs) ensure reserved capacity for services such as Amazon EC2 and Amazon Relational Database Service (Amazon RDS) at the individual account level.

Use case

The identities in this use case are set up as follows:

• DevAccount

Developers check the code into an AWS CodeCommit repository. It stores all the repositories as a single source of truth for application code. Developers have full control over this account. This account is usually used as a sandbox for developers.

• ToolsAccount

A central location for all the tools related to the organization, including continuous delivery/deployment services such as AWS CodePipeline and AWS CodeBuild. Developers have limited/read-only access in this account. The Operations team has more control.

• TestAccount

Applications using the CI/CD orchestration for test purposes are deployed from this account. Developers and the Operations team have limited/read-only access in this account.

• ProdAccount

Applications using the CI/CD orchestration tested in the ToolsAccount are deployed to production from this account. Developers and the Operations team have limited/read-only access in this account.

Solution

In this solution, we will check in sample code for an AWS Lambda function in the Dev account. This will trigger the pipeline (created in AWS CodePipeline) and run the build using AWS CodeBuild in the Tools account. The pipeline will then deploy the Lambda function to the Test and Prod accounts.

Setup

1. Clone this repository. It contains the AWS CloudFormation templates that we will use in this walkthrough.

git clone https://github.com/awslabs/aws-refarch-cross-account-pipeline.git

2. Follow the instructions in the repository README to push the sample AWS Lambda application to an AWS CodeCommit repository in the Dev account.
3. Install the AWS Command Line Interface as described here. To prepare your access keys or assume-role to make calls to AWS, configure the AWS CLI as described here.

Walkthrough

Note: Follow the steps in the order they’re written. Otherwise, the resources might not be created correctly.

1. In the Tools account, deploy this CloudFormation template. It will create the customer master keys (CMK) in AWS Key Management Service (AWS KMS), grant access to Dev, Test, and Prod accounts to use these keys, and create an Amazon S3 bucket to hold artifacts from AWS CodePipeline.

aws cloudformation deploy --stack-name pre-reqs \
--template-file ToolsAcct/pre-reqs.yaml --parameter-overrides \
DevAccount=ENTER_DEV_ACCT TestAccount=ENTER_TEST_ACCT \
ProductionAccount=ENTER_PROD_ACCT

In the output section of the CloudFormation console, make a note of the Amazon Resource Number (ARN) of the CMK and the S3 bucket name. You will need them in the next step.

2. In the Dev account, which hosts the AWS CodeCommit repository, deploy this CloudFormation template. This template will create the IAM roles, which will later be assumed by the pipeline running in the Tools account. Enter the AWS account number for the Tools account and the CMK ARN.

aws cloudformation deploy --stack-name toolsacct-codepipeline-role \
--template-file DevAccount/toolsacct-codepipeline-codecommit.yaml \
--capabilities CAPABILITY_NAMED_IAM \
--parameter-overrides ToolsAccount=ENTER_TOOLS_ACCT CMKARN=FROM_1st_Step

3. In the Test and Prod accounts where you will deploy the Lambda code, execute this CloudFormation template. This template creates IAM roles, which will later be assumed by the pipeline to create, deploy, and update the sample AWS Lambda function through CloudFormation.

aws cloudformation deploy --stack-name toolsacct-codepipeline-cloudformation-role \
--template-file TestAccount/toolsacct-codepipeline-cloudformation-deployer.yaml \
--capabilities CAPABILITY_NAMED_IAM \
--parameter-overrides ToolsAccount=ENTER_TOOLS_ACCT CMKARN=FROM_1st_STEP  \
S3Bucket=FROM_1st_STEP

4. In the Tools account, which hosts AWS CodePipeline, execute this CloudFormation template. This creates a pipeline, but does not add permissions for the cross accounts (Dev, Test, and Prod).

aws cloudformation deploy --stack-name sample-lambda-pipeline \
--template-file ToolsAcct/code-pipeline.yaml \
--parameter-overrides DevAccount=ENTER_DEV_ACCT TestAccount=ENTER_TEST_ACCT \
ProductionAccount=ENTER_PROD_ACCT CMKARN=FROM_1st_STEP \
S3Bucket=FROM_1st_STEP--capabilities CAPABILITY_NAMED_IAM

5. In the Tools account, execute this CloudFormation template, which give access to the role created in step 4. This role will be assumed by AWS CodeBuild to decrypt artifacts in the S3 bucket. This is the same template that was used in step 1, but with different parameters.

aws cloudformation deploy --stack-name pre-reqs \
--template-file ToolsAcct/pre-reqs.yaml \
--parameter-overrides CodeBuildCondition=true

6. In the Tools account, execute this CloudFormation template, which will do the following:
   1. Add the IAM role created in step 2. This role is used by AWS CodePipeline in the Tools account for checking out code from the AWS CodeCommit repository in the Dev account.
   2. Add the IAM role created in step 3. This role is used by AWS CodePipeline in the Tools account for deploying the code package to the Test and Prod accounts.

aws cloudformation deploy --stack-name sample-lambda-pipeline \
--template-file ToolsAcct/code-pipeline.yaml \
--parameter-overrides CrossAccountCondition=true \
--capabilities CAPABILITY_NAMED_IAM

What did we just do?

1. The pipeline created in step 4 and updated in step 6 checks out code from the AWS CodeCommit repository. It uses the IAM role created in step 2. The IAM role created in step 4 has permissions to assume the role created in step 2. This role will be assumed by AWS CodeBuild to decrypt artifacts in the S3 bucket, as described in step 5.
2. The IAM role created in step 2 has permission to check out code. See here.
3. The IAM role created in step 2 also has permission to upload the checked-out code to the S3 bucket created in step 1. It uses the KMS keys created in step 1 for server-side encryption.
4. Upon successfully checking out the code, AWS CodePipeline triggers AWS CodeBuild. The AWS CodeBuild project created in step 4 is configured to use the CMK created in step 1 for cryptography operations. See here. The AWS CodeBuild role is created later in step 4. In step 5, access is granted to the AWS CodeBuild role to allow the use of the CMK for cryptography.
5. AWS CodeBuild uses pip to install any libraries for the sample Lambda function. It also executes the aws cloudformation package command to create a Lambda function deployment package, uploads the package to the specified S3 bucket, and adds a reference to the uploaded package to the CloudFormation template. See here.
6. Using the role created in step 3, AWS CodePipeline executes the transformed CloudFormation template (received as an output from AWS CodeBuild) in the Test account. The AWS CodePipeline role created in step 4 has permissions to assume the IAM role created in step 3, as described in step 5.
7. The IAM role assumed by AWS CodePipeline passes the role to an IAM role that can be assumed by CloudFormation. AWS CloudFormation creates and updates the Lambda function using the code that was built and uploaded by AWS CodeBuild.

This is what the pipeline looks like using the sample code:

Conclusion

Creating multiple AWS accounts provides the highest degree of isolation and is appropriate for a number of use cases. However, keeping a centralized account to orchestrate continuous delivery and deployment using AWS CodePipeline and AWS CodeBuild eliminates the need to duplicate the delivery pipeline. You can secure the pipeline through the use of cross account IAM roles and the encryption of artifacts using AWS KMS. For more information, see Providing Access to an IAM User in Another AWS Account That You Own in the IAM User Guide.

References

• AWS Multiple Account Security Strategy
• AWS Multiple Account Billing Strategy