{"id":238081,"date":"2021-10-01T07:33:00","date_gmt":"2021-10-01T11:33:00","guid":{"rendered":"https:\/\/wordpress-756359-3782526.cloudwaysapps.com\/?p=238081"},"modified":"2024-09-21T04:53:26","modified_gmt":"2024-09-21T04:53:26","slug":"2021-10-01-terraform","status":"publish","type":"post","link":"https:\/\/www.travis-ci.com\/blog\/2021-10-01-terraform\/","title":{"rendered":"Terraform, S3, Lambda, Travis and API Gateway"},"content":{"rendered":"\n

Terraform is an open-source infrastructure as code software tool that enables you to safely and predictably create, change, and improve infrastructure. This is a little longer blog post, but Terraform is becoming more essential in developers tech stack, so let\u2019s start out with some basics and slowly olive-branch to more complex use cases of Terraform and Travis CI. Let\u2019s do this.<\/p>\n\n\n\n

Getting started<\/h2>\n\n\n\n

Let\u2019s take a look at my terraform_config<\/code> file:<\/p>\n\n\n\n

# Montana's Terraform config\n\nprovider \"aws\" {\n  region = \"eu-west-1\"\n  #access_key = \"PUT-YOUR-ACCESS-KEY-HERE\"\n  #secret_key = \"PUT-YOUR-SECRET-KEY-HERE\"\n  version = \"~> 2.55.0\"\n}\n\nterraform {\n  required_providers {\n    aws = {\n      source = \"hashicorp\/aws\"\n    }\n  }\n  required_version = \">= 0.13\"\n}<\/code><\/pre>\n\n\n\n
Couple of things you\u2019ll notice in my config file I selected AWS<\/code> as my provider, there\u2019s ENVIRONMENT_VARIABLES<\/code>, what version of Terraform, and what version I want to enforce.<\/p>\n\n\n\n
We are going to assume this project is in a shared setting, so local state<\/code> will not be good enough, we\u2019re going to have to use remote state<\/code>. In short with remote state in Terraform, Terraform writes the state data to a remote data store, so in this case probably cloud storage, which can then be shared between all members of a team. Think of it as a Google Doc, and you can pick and choose who has access to it.<\/p>\n\n\n\n
Before we move to the architecture section, remember this Terraform list of configuration files:<\/p>\n\n\n\n
credentials\ncredentials_helper\ndiable_checkpoint\ndisable_checkpoint_signature\nplugin_cache_dir\nprovider_installation<\/code><\/pre>\n\n\n\n
The Arch<\/h2>\n\n\n\n
So you as a developer decide to use Travis CI, the best tool for building on the internet, so first off, good choice! Here are the things you\u2019re going to need to get this off the ground:<\/p>\n\n\n\n
    \n
  • Travis CI hooked up to your GitHub account and select your copy of the repository in question<\/li>\n\n\n\n
  • An AWS IAM user created, that\u2019s been granted programmatic access<\/em><\/li>\n<\/ul>\n\n\n\n
    We know what Travis does, but incase you don\u2019t in this use case, Travis is going build the website artifacts, deploy the infrastructure, and push the artifacts to production instead of a staging environment.<\/p>\n\n\n\n
    This example I\u2019m going to be using an Amazon S3<\/code> backend with DynamoDB<\/code> for Terraform, I\u2019ve found DynamoDB is the most user friendly with Teraform. Terraform will store the state (now remember not local, but remote) within S3 and use DynamoDB to acquire a lock while performing changes. The lock is important to avoid that two Terraform binaries are modifying the same state concurrently, if two Terraform instances were doing this – you could see the trouble and headache this could cause.<\/p>\n\n\n\n
    Terraform configuration files<\/h2>\n\n\n\n
    Let\u2019s start with the S3 backend config file:<\/p>\n\n\n\n
    terraform {\n  required_version = \">= 0.12\"\n}\n\nprovider \"aws\" {}\n\ndata \"aws_caller_identity\" \"current\" {}\n\nlocals {\n  account_id = data.aws_caller_identity.current.account_id\n}\n\nresource \"aws_s3_bucket\" \"terraform_state\" {\n  bucket = \"${local.account_id}-terraform-states\"\n\n  versioning {\n    enabled = true\n  }\n\n  # I strongly encourage enabling server side encryption by default - Montana\n  \n  server_side_encryption_configuration {\n    rule {\n      apply_server_side_encryption_by_default {\n        sse_algorithm = \"AES256\"\n      }\n    }\n  }\n}<\/code><\/pre>\n\n\n\n
    Now let\u2019s create the DynamoDB<\/code> Terraform config file:<\/p>\n\n\n\n
    resource \"aws_dynamodb_table\" \"terraform_lock\" {\n  name         = \"terraform-lock\"\n  billing_mode = \"PAY_PER_REQUEST\"\n  hash_key     = \"LockID\"\n\n  attribute {\n    name = \"LockID\"\n    type = \"S\"\n  }\n}<\/code><\/pre>\n\n\n\n
    You see the \u2018lock\u2019 method in place to enforce only one instance of Terraform to spawn. Next you\u2019re going to want verboseness in this process, so let\u2019s setup some Terraform outputs:<\/p>\n\n\n\n
    output \"s3_bucket_name\" {\n  value       = aws_s3_bucket.terraform_state.id\n  description = \"Montana's Terraform S3 Bucket\"\n}\n\noutput \"s3_bucket_arn\" {\n  value       = aws_s3_bucket.terraform_state.arn\n  description = \"Montana's ARN Bucket\"\n}\n\noutput \"s3_bucket_region\" {\n  value       = aws_s3_bucket.terraform_state.region\n  description = \"Montana's S3 Region of the Bucket\"\n}\n\noutput \"dynamodb_table_name\" {\n  value       = aws_dynamodb_table.terraform_lock.name\n  description = \"Montana's ARN of the DynamoDB table\"\n}\n\noutput \"dynamodb_table_arn\" {\n  value       = aws_dynamodb_table.terraform_lock.arn\n  description = \"The ARN of the DynamoDB table\"\n}<\/code><\/pre>\n\n\n\n
    Back and Forth<\/h2>\n\n\n\n
    You know maybe asking yourself, how can we use Terraform to setup the S3 bucket and DynamoDB table we want to use for the remote state backend? There\u2019s a few ways we can do this:<\/p>\n\n\n\n
      \n
    • If you have a shared local state. you then need to commit the local state to your VCS and share it in a remote repository.<\/li>\n\n\n\n
    • You do a migrated state. You first migrate the local state to a remote state backend.<\/li>\n<\/ul>\n\n\n\n
      Both of the at solutions involve creating the remote state<\/code> resources using local state<\/code>. Remember that Terraform state might contain secrets. In the case of only the Amazon S3<\/code> bucket and DynamoDB<\/code> table there is only one variable which might be problematic: The AWS access key<\/em>.<\/p>\n\n\n\n
      Secrets<\/h2>\n\n\n\n
      In Terraform if you are working with a private repository, this might not be a huge issue. When working on open source code it might be useful to encrypt the state file, as I described earlier, before committing it to GitHub to the repository or whatever VCS you use. You can do this with OpenSSL<\/code> or even more specailized tools.<\/p>\n\n\n\n
      The Stack<\/h2>\n\n\n\n
      We are currently using:<\/p>\n\n\n\n
        \n
      • Terraform<\/li>\n\n\n\n
      • Jekyll<\/li>\n\n\n\n
      • Git<\/li>\n<\/ul>\n\n\n\n
        Let\u2019s create two workspaces in this scenario state<\/code> and prod<\/code>. The state workspace will manage the remote state resources, so, the S3 bucket and the DynamoDB table. The prod<\/code> workspace will manage the production environment of our website. You can add more workspaces for staging or testing later but this is beyond the scope of this.<\/p>\n\n\n\n
        Bootstrapping<\/h2>\n\n\n\n
        Let\u2019s create three folders containing Terraform files,<\/p>\n\n\n\n
          \n
        • Bootstrap<\/li>\n\n\n\n
        • Backend<\/li>\n\n\n\n
        • Website<\/li>\n<\/ul>\n\n\n\n
          This is now the outline of your project in question:<\/p>\n\n\n\n
          .\n\u251c\u2500\u2500 locals.tf\n\u251c\u2500\u2500 providers.tf\n\u251c\u2500\u2500 backend\n\u2502   \u251c\u2500\u2500 backend.tf\n\u2502   \u251c\u2500\u2500 backend.tf.tmpl\n\u2502   \u251c\u2500\u2500 locals.tf -> ..\/locals.tf\n\u2502   \u251c\u2500\u2500 providers.tf -> ..\/providers.tf\n\u2502   \u2514\u2500\u2500 state.tf -> ..\/bootstrap\/state.tf\n\u251c\u2500\u2500 bootstrap\n\u2502   \u251c\u2500\u2500 locals.tf -> ..\/locals.tf\n\u2502   \u251c\u2500\u2500 providers.tf -> ..\/providers.tf\n\u2502   \u2514\u2500\u2500 state.tf\n\u2514\u2500\u2500 website\n    \u251c\u2500\u2500 backend.tf -> ..\/backend\/backend.tf\n    \u251c\u2500\u2500 locals.tf -> ..\/locals.tf\n    \u251c\u2500\u2500 providers.tf -> ..\/providers.tf\n    \u2514\u2500\u2500 website.tf<\/code><\/pre>\n\n\n\n
          Let\u2019s take a look at the contents of bootstrap\/state.tf<\/code>:<\/p>\n\n\n\n
          # Montana's config files \/bootstrap\/state.tf\n\nlocals {\n  state_bucket_name = \"${local.project_name}-${data.aws_caller_identity.current.account_id}-${data.aws_region.current.name}\"\n  state_table_name = \"${local.state_bucket_name}\"\n}\n\nresource \"aws_dynamodb_table\" \"locking\" {\n  name           = \"${local.state_table_name}\"\n  read_capacity  = \"20\"\n  write_capacity = \"20\"\n  hash_key       = \"LockID\"\n\n  attribute {\n    name = \"LockID\"\n    type = \"S\"\n  }\n}\n\nresource \"aws_s3_bucket\" \"state\" {\n  bucket = \"${local.state_bucket_name}\"\n  region = \"${data.aws_region.current.name}\"\n\n  versioning {\n    enabled = true\n  }\n\n  server_side_encryption_configuration {\n    \"rule\" {\n      \"apply_server_side_encryption_by_default\" {\n        sse_algorithm = \"AES256\"\n      }\n    }\n  }\n\n  tags {\n    Name = \"terraform-state-bucket\"\n    Environment = \"global\"\n    project = \"${local.project_name}\"\n  }\n}\n\noutput \"BACKEND_BUCKET_NAME\" {\n  value = \"${aws_s3_bucket.state.bucket}\"\n}\n\noutput \"BACKEND_TABLE_NAME\" {\n  value = \"${aws_dynamodb_table.locking.name}\"\n}<\/code><\/pre>\n\n\n\n
          Above I\u2019ve defined the S3 buckets, enabled encryption as well as versioning. Encryption is important because Terraform more of often than not might contain secrets<\/code>.<\/p>\n\n\n\n
          You\u2019ll notice in Terraform I called an attribute called LockID<\/code> so we have to create it and make it the primary key, again so there\u2019s not two instances of Terraform running at once.<\/p>\n\n\n\n
          Let\u2019s cerate the state<\/code> workspace by running these commands:<\/p>\n\n\n\n
          terraform workspace new state\nterraform init bootstrap\nterraform apply bootstrap<\/code><\/pre>\n\n\n\n
          After this, the S3 bucket and DynamoDB table are created and we will migrate the local state. Let\u2019s look at the backend\/backend.tf.tmpl<\/code> file, this is the Terraform it will follow, you can generate an environment variable, or in my case I set the environment variables from key value pairs. You can use this crafty bash script I\u2019ve provided if you don\u2019t want to go down the traditional route:<\/p>\n\n\n\n
          #!\/bin\/bash\ndotenv () {\n  set -a\n  [ -f .env ] && . .env\n  set +a\n}\n\ndotenv\n\ncd () {\n  builtin cd $@\n  dotenv\n}<\/code><\/pre>\n\n\n\n
          So now the config file should look like this:<\/p>\n\n\n\n
          terraform {\n  backend \"s3\" {\n    bucket         = \"${BACKEND_BUCKET_NAME}\"\n    key            = \"terraform.tfstate\"\n    region         = \"eu-central-1\"\n    dynamodb_table = \"${BACKEND_TABLE_NAME}\"\n  }\n}<\/code><\/pre>\n\n\n\n
          Let\u2019s initialize the backend, run:<\/p>\n\n\n\n
          terraform init backend<\/code><\/pre>\n\n\n\n
          So you should have some core assets, now we have to specify the HTML and CSS files. This is a bit cumbersome as we cannot tell Terraform to upload a whole folder much like you can with wget<\/code> or curl<\/code>, but this is Terraform\u2019s structure, let\u2019s look at website.tf<\/code>:<\/p>\n\n\n\n
          locals {\n  site_root = \"website\/static\/_site\"\n  index_html = \"${local.site_root}\/index.html\"\n  about_html = \"${local.site_root}\/about\/index.html\"\n  post_html = \"${local.site_root}\/jekyll\/update\/2018\/06\/30\/welcome-to-jekyll.html\"\n  error_html = \"${local.site_root}\/404.html\"\n  main_css = \"${local.site_root}\/assets\/main.css\"\n}\n\nresource \"aws_s3_bucket_object\" \"index\" {\n  bucket = \"${aws_s3_bucket.website.id}\"\n  key    = \"index.html\"\n  source = \"${local.index_html}\"\n  etag   = \"${md5(file(local.index_html))}\"\n  content_type = \"text\/html\"\n}\n\nresource \"aws_s3_bucket_object\" \"post\" {\n  bucket = \"${aws_s3_bucket.website.id}\"\n  key    = \"jekyll\/update\/2018\/06\/30\/welcome-to-jekyll.html\"\n  source = \"${local.post_html}\"\n  etag   = \"${md5(file(local.post_html))}\"\n  content_type = \"text\/html\"\n}\n\nresource \"aws_s3_bucket_object\" \"about\" {\n  bucket = \"${aws_s3_bucket.website.id}\"\n  key    = \"about\/index.html\"\n  source = \"${local.about_html}\"\n  etag   = \"${md5(file(local.about_html))}\"\n  content_type = \"text\/html\"\n}\n\nresource \"aws_s3_bucket_object\" \"error\" {\n  bucket = \"${aws_s3_bucket.website.id}\"\n  key    = \"error.html\"\n  source = \"${local.error_html}\"\n  etag   = \"${md5(file(local.error_html))}\"\n  content_type = \"text\/html\"\n}\n\nresource \"aws_s3_bucket_object\" \"css\" {\n  bucket = \"${aws_s3_bucket.website.id}\"\n  key    = \"assets\/main.css\"\n  source = \"${local.main_css}\"\n  etag   = \"${md5(file(local.main_css))}\"\n  content_type = \"text\/css\"\n}\n\noutput \"url\" {\n  value = \"http:\/\/${local.website_bucket_name}.s3-website.${aws_s3_bucket.website.region}.amazonaws.com\"\n}<\/code><\/pre>\n\n\n\n
          Now let\u2019s run some of these final commands:<\/p>\n\n\n\n
          terraform workspace new prod\nterraform init website\ncd website\/static && jekyll build && cd -\nterraform apply website<\/code><\/pre>\n\n\n\n
          We\u2019ve now created a static website using Terraform and DynamoDB. Let\u2019s now implement Travis!<\/p>\n\n\n\n
          Travis<\/h2>\n\n\n\n
          Let\u2019s get to implementing Travis to catch any bugs! As you know we need to make out .travis.yml.<\/code>, this simply put, tells the build server which commands to execute. Let\u2019s take a look at the .travis.yml<\/code> file:<\/p>\n\n\n\n
          ---\nlanguage: generic # this can also be left out\n\ninstall:\n  - gem install bundler jekyll\n\nscript:\n  - .\/build.sh<\/code><\/pre>\n\n\n\n
          We need that build.sh<\/code> file, but remember to give it proper permissions, here\u2019s the contents of that:<\/p>\n\n\n\n
          cd website\/static\nbundle install\nbundle exec jekyll build\ncd -\n\n.\/terraform-linux init\n.\/terraform-linux validate website\n\nif [[ $TRAVIS_BRANCH == 'master' ]]\nthen\n    .\/terraform-linux workspace select prod\n    .\/terraform-linux apply -auto-approve website\nfi<\/code><\/pre>\n\n\n\n
          Conclusions<\/h2>\n\n\n\n
          If possible I would probably go for local state only and store it directly within the repository, this would make a lot of the steps easier, luckily though I\u2019ve made a bash script specifically for Terraform and Travis to make life a bit easier, this is meant if you\u2019re in a testing environment – so make sure before you use this you\u2019re running this in staging, or some sort of testing env:<\/p>\n\n\n\n
          #!\/bin\/bash\nset=+x\n\n\/ # set -v && echo $HOME\n\/root\n\/ # set +v && echo $HOME\nset +v && echo $HOME\n\/root\n\n\/ # set -x && echo $HOME\n+ echo \/root\n\/root\n\/ # set +x && echo $HOME\n+ set +x\n\/root\n\n# Check the creds made sure by Montana Mendy. \n\ncred=\"-var \"do_token=${DO_PAT}\"\"\ntf=$(which terraform)\n\ninit() {\n    echo \"\"\n    echo \"Init the provider\"\n    echo \"\"\n    $tf init\n\n    echo \"Formatting files\"\n    echo \"\"\n    $tf fmt\n    echo \"\"\n    echo \"Validating files\"\n    echo \"\"\n    $tf validate\n}\n\nclone() {\n    if [ -d ansible ]; then\n        rm -rf ansible\n        git clone --depth=1 $ansible_repo\n        echo \"\"\n    else\n        git clone --depth=1 $ansible_repo\n        echo \"\"\n    fi\n\n    if [ -d shell_scripts ]; then\n        rm -rf shell_scripts\n        git clone --depth=1 $shell_script_repo\n        echo \"\"\n    else\n        git clone --depth=1 $shell_script_repo\n        echo \"\"\n    fi\n\n    if [ -d cloud_init ]; then\n        rm -rf cloud_init\n        git clone --depth=1 $cloud_init_repo\n        echo \"\"\n    else\n        git clone --depth=1 $cloud_init_repo\n        echo \"\"\n    fi\n}\n\n# Clean up shell scripts.\n\nclean() {\n    if [ -d shell_scripts ]; then\n        rm -rf shell_scripts\n    fi\n    if [ -d ansible ]; then\n        rm -rf ansible\n        if [ -f inventory ]; then\n            rm inventory\n        fi\n    fi\n    if [ -d cloud_init ]; then\n        rm -rf cloud_init\n    fi\n    if [ -f terraform.tfstate ]; then\n        rm -rf terraform*\n        rm -rf .terraform\n    fi\n}\n\n# Run Terraform commands.\n\nplan() {\n    clone\n    init\n    $tf plan $cred\n}\n\napply() {\n    clone\n    init\n    $tf apply -auto-approve $cred\n}\n\ncase $1 in\n-i | init)\n    $tf init\n    ;;\nclone)\n    clone\n    ;;\nplan)\n    plan\n    ;;\napply)\n    apply\n    ;;\nshow)\n    $tf show\n    ;;\nlist)\n    $tf state list\n    ;;\n    \n# Run 'terraform destroy'. \n    \ndestroy)\n    $tf destroy $cred\n    clean\n    ;;\nclean)\n    clean\n    ;;\n-f | fmt)\n    $tf fmt\n    ;;\n-v | validate)\n    $tf validate\n    ;;\n*)\n    echo \"$0 [options]\"\n    echo \"\"\n    echo \"Options are: clone | plan | apply| show| list | destroy | fmt | validata\"\n    echo \"\"\n    ;;\nesac<\/code><\/pre>\n\n\n\n
          Well there you have it, we used Terraform, Travis CI, Amazon, API Gateways, to create and deploy a Jekyll website with some of my tips and tricks attached that I think you may find useful when using it provisionally.<\/p>\n\n\n\n
          If you have any questions at all, please email me at montana@travis-ci.org<\/a>.<\/p>\n\n\n\n
          As always, happy building!<\/p>\n","protected":false},"excerpt":{"rendered":"
          Terraform is an open-source infrastructure as code software tool that enables you to safely and predictably create, change, and improve infrastructure. This is a little longer blog post, but Terraform is becoming more essential in developers tech stack, so let\u2019s start out with some basics and slowly olive-branch to more complex use cases of Terraform […]<\/p>\n","protected":false},"author":2,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_breakdance_hide_in_design_set":false,"_breakdance_tags":"","footnotes":""},"categories":[16],"tags":[7,19,20,5],"class_list":["post-238081","post","type-post","status-publish","format-standard","hentry","category-news","tag-community","tag-feature","tag-infrastructure","tag-news"],"_links":{"self":[{"href":"https:\/\/www.travis-ci.com\/wp-json\/wp\/v2\/posts\/238081","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.travis-ci.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.travis-ci.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.travis-ci.com\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.travis-ci.com\/wp-json\/wp\/v2\/comments?post=238081"}],"version-history":[{"count":1,"href":"https:\/\/www.travis-ci.com\/wp-json\/wp\/v2\/posts\/238081\/revisions"}],"predecessor-version":[{"id":242656,"href":"https:\/\/www.travis-ci.com\/wp-json\/wp\/v2\/posts\/238081\/revisions\/242656"}],"wp:attachment":[{"href":"https:\/\/www.travis-ci.com\/wp-json\/wp\/v2\/media?parent=238081"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.travis-ci.com\/wp-json\/wp\/v2\/categories?post=238081"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.travis-ci.com\/wp-json\/wp\/v2\/tags?post=238081"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}