GCP Journey

In this post, I'll share my journey from manual uploads of new blog posts to a CI/CD automated approach. Buckle up, because we're harnessing the power of GCP Cloud Build to say goodbye to manual deployments and hello to effortless updates! Let's explore how I built a system that automatically builds and deploys my latest blog content directly from my GitLab repository to my dedicated Cloud Storage bucket, ensuring my website is always serving the freshest content after a code push.

In a previous blog post I documented how I quickly got this blog up and running as a static hosted site on GCP. I made the concious decision to get it to "production", aka publicly available on the web, as my goal and accepted that it would be a laborious and manual task to re-upload my site to the Cloud Storage bucket everytime I created a new post. My new goal is to utilise GCP Cloud Build as my CI/CD (Continuous Integration/Continuous Delivery) pipleline to automate this process.

Current process

On my local machine, I create a new blog post in 11ty as a markdown file. Once I have finished writing the post I perform a build using npm run build which generates my site and places it in the _site directory. I then login into my Google Cloud Console, navigate to my Cloud Storage bucket and manually upload all of the files and directories from my newly generated _site directory into the bucket, overriding the previous file if it exists already.

This is tedious, error prone and not what anyone wants to spend their time doing.

Desired outcome

The workflow I want to create starts with on my local machine finishing writing a new blog post. Once I do so, I perform a git push to upload my new content to my remote git repository in GitLab. This push to my main/master branch should then trigger Cloud Build to fetch the latest code that I just pushed, perform the npm run build command and move all of the contents of the _site directory to my Cloud Storage Bucket.

Alternative approaches

This CI/CD could be achieved via GitHub Actions, or the equivalent GitLab CI/CD. The difference being is I would need to use a Google specific GitHub Action to access my Cloud Storage Bucket or leverage the gcloud command-line in GitLab Runner.

Given the context of this blog, I am making the active decision to use GCP Cloud Build as a learning experience. (Plus I have found GitHub Actions to not be the most reliable or performant service so interested in comparing with Cloud Build).

Implementation

Thankfully there is once again a well documented guide provided by GCP for how to build respositories from GitHub which was almost identical to setting up in GitLab.

The process I took involved the following:

Create cloudbuild.yaml file in root of my 11ty blog project. This is what I will point Cloud Build at once it triggers when a push occurs to the main branch of my GitLab repo.

steps:
  - name: 'gcr.io/cloud-builders/npm' # Use the official npm builder image
    args: ['ci']      
  - name: 'gcr.io/cloud-builders/npm'
    args: ['run', 'build']

  - name: 'gcr.io/cloud-builders/gsutil'  # Use the official gsutil builder image
    args: ['-m', 'rsync', '-r', '-c', '-d', '-u', '_site', 'gs://gcp-journey-blog/']  # Uploads content to Cloud Storage

logsBucket: 'gs://gcp-journey-logs'

serviceAccount: 'cloud-build-gcp-journey-sa@gcp-journey-blog.iam.gserviceaccount.com'

explanation of my `cloudbuild.yaml` file:

The intitial steps makes use of official Google Cloud Builders for Cloud Build. These Cloud Builders are a trusted source and are designed and optimised for usage within Cloud Build. I personally like the consistency they offer rather than relying on standard public Docker images.

They checkout install dependencies from the repo and run the build command that generates the static site, placing it in a _site directory.

The gsutil builder then uploads the contents of this directory to my Cloud Storage bucket.

The args in the gsutil line of my cloudbuild.yaml file are command-line arguments that are passed to the gsutil command. Here's what each argument does:

-m: This option tells gsutil to perform operations concurrently. This can significantly speed up the operation by taking advantage of multiple connections, especially when dealing with many files or large files.

rsync: This is the command that tells gsutil to synchronize the contents of the source and destination. It makes the destination identical to the source by adding, deleting, and updating files.

-r: This option tells gsutil to perform a recursive synchronization, meaning it will synchronize all files and subdirectories within the specified directory.

-c: This option tells gsutil to compute a checksum for each file while synchronizing. This ensures that the file contents are identical, not just the sizes and modification times.

-d: This option tells gsutil to delete files in the destination that are not present in the source.

_site: This is the source directory that gsutil will synchronize. It's the directory where your built site files are located.

gs://gcp-journey-blog/: This is the destination where gsutil will synchronize the files to. It's a Google Cloud Storage bucket.

So, in summary, this line is telling gsutil to perform a parallel, recursive synchronization of the _site directory to the gs://gcp-journey-blog/ bucket, computing checksums for each file and deleting any files in the destination that are not present in the source.

I also created a new Cloud Storage bucket via the Google Cloud Console for storing my logs related to Cloud Builds which I reference in my cloudbuild.yaml file.

Finally I specificy which Service Account to use for these actions (more and which roles I gave to this Service Account below).

Enable Cloud Build API in Google Cloud Console under my blog project.
Enable Secret Manager API in Google Cloud Console under my blog project.
Granted my GCP user account the Cloud Build Connection Admin (roles/cloudbuild.connectionAdmin) role.
Created x2 Personal Access token in GitLab for Cloud Build to use, one with api scope and one with read_api scope.
In Cloud Build console added a host connection using the GitLab tokens created in the previous step.
Linked my gcp-blog GitLab repository.
In IAM, created a new service account with the roles Cloud Build Editor: This role grants permissions to trigger builds, manage build triggers, and view build logs. Storage Admin: This role provides full access to manage Cloud Storage buckets, including creating, deleting, updating, and modifying permissions.
Create a trigger in Cloud Build that points to Gitlab repo master branch and utilises the service account I created in step 8. This trigger is configured to fire on a push to the main branch and get it's configuration of what to do from my cloudbuild.yaml file in my GitLab repo that I created in step 1.
I repeatedly tested firing my trigger via the Google Cloud Console without the need to keep pushing changes to GitLab while I worked through some syntax issues in my cloudbuild.yaml file.

One gotcha that I learned, was the fact I received and error message stating there were restrictions on the region I was using to perform my Cloud Build. I had chosen the europe-west2 region as this is where my Cloud Storage bucket resides but found via this document: https://cloud.google.com/build/docs/locations#restricted_regions_for_some_projects that it wasn't possible. I therefore had to delete my host connection, linked repository and trigger and re-create them in the europe-west1 region which resolved the problem. Would have been nice to know this up front though.

Conclusion

Once again the documentation provided by GCP was great and I can now successfully push a newly written blog post to the main branch of my GitLab blog repo and Cloud Build will take care of loading the resultant static site and new content to my Cloud Storage bucket. My Load Balancer takes care of incoming traffic requests and serves this to the web.

I am very aware that I have used 'click-ops' for this whole process. So implementing all this infrastructure including my Cloud Storage bucket via IaC (Infrastructure as Code) with Terraform is on my to do list for the future.