Last week, our systems engineer Chris Lucas-McMillan shared how he has streamlined a variety of general site infrastructure tasks as part of his static sites project. This week, Chris outlines some enhancements he has made since the first successful implementation of the static sites project, including improvements to how static site staging builds are handled, site analytics, and more.
HyperWire was the first site created using my static sites infrastructure, and working with the site after launch was a great way to work out priorities for future enhancements. Here’s what I’ve rolled out since the first version of the static sites project.
Ensuring that users viewing staging builds stay within their own site’s repository
Initially, clicking on a build process in GitLab took users away from HyperWire’s repository to the static site repository, because that’s where the new environment was being created. This could be a bit disorienting. Luckily for me, this was actually an issue that GitLab had just fixed in a new version! So after a quick side quest to get GitLab updated, I could create an environment in GitLab with the pipeline that built the site, instead of the one that deployed it.
Converting the CI trigger job to a template
Getting the pipelines and environments working properly added a lot of configuration to the CI. And given one of the aims of this project was to simplify the infrastructure CI needed in the website repositories, I didn’t want to leave things as they were. To solve this, I created a templates repository, and moved all of the extra configuration there. Now, all we need to do is import that template, and specify the site-specific variables, for example:
variables:
PROD_DOMAIN: 'blog.softiron.com'
CUSTOM_ERROR_PAGES: '{ "404": "/404.html" }'
LETSENCRYPT_NOTIFICATION_EMAIL:challenge-ownership-address@softiron.com
include:
- project: 'pe/gitlab-ci-templates'
file: 'static-sites-infra-downstream.yml'
Triggering an upstream build from downstream
The configuration we had was working well, but it presented a small problem; if the production web infrastructure failed and needed to be rebuilt from scratch, the websites it should be hosting will only be deployed when a change is made to each specific site. So, I needed to add a trigger to the infrastructure pipeline which would start a build pipeline on each site, which in turn triggers the infrastructure pipeline again to deploy the built site - in this instance, the HyperWire site:
build_web_blog:
stage: trigger_upstream_builds
trigger:
project: web/blog
rules:
- !reference [.infra_prod_rules, rules]
Every time a new site is created using the static sites infrastructure, it will similarly need to be added to my upstream trigger yaml file to trigger the push.
Of course, this means we will need to have this trigger stage included in gitlab-ci.yml, in this case:
- trigger_upstream_builds
Solving this problem gave me an easy fix for something else I’d need to handle - rebuilding sites on a schedule! In the case of HyperWire, any post which is future-dated will not be included in the build process until that date has been reached. However, that means we need to trigger a regular build process to ensure that those posts are included in a new build as soon as possible once that date has been reached. So, to make sure those posts go out on time, I set up a scheduled pipeline on the static site infrastructure repository to trigger a build periodically.
Enabling future-dated content to be viewed in staging builds
I’ve talked about the fact that builds don’t include by default any content that is future-dated, in HyperWire’s case, that’s blog posts. However, when it comes to staging builds, it makes our editors’ lives much easier being able to preview all posts, regardless of when they are set to publish.
To do this, I made a few changes in gitlab-ci.yml .
- if [ "$CI_COMMIT_BRANCH" == "$CI_DEFAULT_BRANCH" ]; then hugo ; fi
- if [ "$CI_COMMIT_BRANCH" != "$CI_DEFAULT_BRANCH" ]; then hugo -DF ; fi
So essentially, if the branch being built is not main, don’t run the Hugo publishing rules. This neatly steps around the publishing build rules without unnecessary complication, allowing our contributors to the blog to easily preview how their posts will look without needing to mess with their publishing dates.
Setting up site analytics with Matomo and a privacy opt-in
At the footer of this website you’ll find a privacy link; this leads to a page that lets users opt in or out of our web analytics collection (whether you default to being opted in or out initially will depend on your browser’s privacy settings).
To implement this capability, the static sites project includes a Matomo install and configuration managed by Ansible, which includes setting up a URL for authenticated users at SoftIron to browse the Matomo analytics site.
While most of this can be handled within the static sites project, we will still need to add a tracking code individually for every SoftIron site that we want to collect analytics for. Fortunately, this isn’t a lot of extra work, it just requires adding a small tag, with an unique ID to identify the site to Matomo, into the layout template of each site.
Custom error pages
The static sites repository also includes support for each site to have its own custom error pages. We can specify a list of status codes and the path for their respective pages as key value pairs:
CUSTOM_ERROR_PAGES: '{ "403": "/403.html", "404": "/404.html" }'
This is then used by Ansible to configure the error pages in nginx:
- name: Set fact for custom_error_pages
set_fact:
custom_error_pages_fact: "{{ custom_error_pages }}"
when: custom_error_pages | length > 0
- name: Custom Error Pages
lineinfile:
path: /etc/nginx/sites-enabled/{{ domain }}.conf
line: " error_page {{ item.key }} {{ item.value }};"
insertafter: " # Custom Error Pages"
state: present
when: custom_error_pages_fact is defined
loop: "{{ custom_error_pages_fact | dict2items }}"
What I’m working on next
Restricted sites and authentication
This is what I’ve been working on most recently, and it’s almost ready!
Some of our static sites are for specific eyes only, which means they require authentication to view. Usually this is an SSO (Sign Sign On) login, though sites can also be password locked with basic HTTP authentication (.htpasswd files). We can specify which authentication methods should be used by a site by setting AUTH_REQUIRED as needed in the site’s gitlab-ci.yml.
I’ve got all of the authentication working, I’m currently just getting each site’s custom login page to work correctly and then it will be good to go!
More CI templates
Once I’ve finished getting the custom login pages working, I’ll be moving on to making CI templates for the various other static site frameworks we use (such as Eleventy) and moving those sites over to the new static sites infrastructure, teasing out bugs and refining the configuration as I go.
So, does this mean we can now ‘push button, get site’?
Not 100% - much like in my examples with the blog, there will still need to be updates to the static sites repo to cater to specific new site requirements (such as the CI templates) and individual sites will still need some minor adjustments, such as adding the specific Matomo tracking code for that specific site to its template.
However, by standardizing the rest of the CI, we’ll now have more time to focus on elements like these. That means more time to build unique features and functionality for our users, and less time on all the repeatable grind-y stuff. And a shortened time to launch for any new static sites we want to spin up in future!
