In the last post, we were left with a kubernetes cluster, and a test deployment that would break once every 24 hours, because of the preemptible instances we are using. So the highest priority right now is to fix that.

The next step is going to require you to own a domain. I recommend namecheap. You can get a .com for $8.88/yr for the first year, then $10.98/yr after that.

Table of Contents

• Table of Contents
• Create a DNS Zone
• Deploy External DNS Controller
• Edit the hello app to include the domain, and updated ttl
• Conclusion

To make your life easier, I’ve variablized the commands in this post, so that you can simply set the variables, then copy/paste the commands without much trouble.

export PROJECT_NAME="kubernetes-on-the-cheap"
export CLUSTER_NAME="hobby-1"
export DOMAIN="foo.com"

Create a DNS Zone

gcloud beta dns \
  managed-zones create "${DOMAIN}" \
  --description="${DOMAIN}" \
  --dns-name="${DOMAIN}."

Once the DNS zone is created, you need to update your domain registrar with the google name servers.

When viewing the zone, you should see an NS record with values like ns-cloud-<??>.googledomains.com.

If you chose to use namecheap, here’s how to update DNS on namecheap.com.

Deploy External DNS Controller

Some of these steps came from knative

1. Create a new service account for Cloud DNS admin role.

   export CLOUD_DNS_SA=cloud-dns-admin
   
   gcloud --project $PROJECT_NAME iam service-accounts \
       create $CLOUD_DNS_SA \
       --display-name "Service Account to support ACME DNS-01 challenge."

2. Bind the role dns.admin to the newly created service account.

   # Fully-qualified service account name also has project-id information.
   export CLOUD_DNS_SA_FQ=$CLOUD_DNS_SA@$PROJECT_NAME.iam.gserviceaccount.com
   
   gcloud projects add-iam-policy-binding $PROJECT_NAME \
       --member serviceAccount:$CLOUD_DNS_SA_FQ \
       --role roles/dns.admin

3. Download the secret key file for your service account.

   gcloud iam service-accounts keys create ~/credentials.json \
     --iam-account=$CLOUD_DNS_SA_FQ

4. Upload the service account credential to your cluster. This command uses the secret name cloud-dns-key, but you can choose a different name.

   export CLOUD_DNS_SECRET_NAME="cloud-dns-key"
   
   kubectl create secret generic "${CLOUD_DNS_SECRET_NAME}" \
     --from-file=credentials.json=$HOME/credentials.json

5. Deploy external dns.

   curl https://ghostsquad.me/files/kubernetes-on-the-cheap-part-2/external-dns.yaml -o external-dns.yaml
   sed -i "s/__PROJECT_NAME__/${PROJECT_NAME}/g" external-dns.yaml
   sed -i "s/__CLUSTER_NAME__/${CLUSTER_NAME}/g" external-dns.yaml
   sed -i "s/__CLOUD_DNS_SECRET_NAME__/${CLOUD_DNS_SECRET_NAME}/g" external-dns.yaml
   kubectl apply -f external-dns.yaml

Edit the hello app to include the domain, and updated ttl

curl https://ghostsquad.me/files/kubernetes-on-the-cheap-part-2/hello.yaml -o hello-part-2.yaml
sed -i "s/__DOMAIN__/${DOMAIN}/g" hello-part-2.yaml
kubectl apply -f hello-part-2.yaml

Conclusion

We deployed external DNS so that it can keep the cluster updated with an IP address for your domain name. I even watched it go to work by scaling up to 2 nodes, then scaling back down.

It doesn’t appear though that externalDNS is adding multiple A records (for each IP) when you have 2 nodes. This deserves some more research.

• Snag a domain that points to the cluster, so that we don’t have to update the ingress
• Deploy external-dns to automatically update our domain with the list of IPs of hosts (when the get preempted)
• Deploy cert-manager to get HTTPS!
• Setup node-termination-handler to further improve shutdowns
• Setup SpotInst in order to handle rolling instances on a regular basis in a controlled way, and scaling the cluster temporarily while doing so.

Stay tuned for Part 3!

I wanted to play around with Kubernetes for personal learning, and even some personal projects. I wanted to deploy a website with a live backend, but I didn’t really want to be bound to cPanel, or terrible performance issues on shared hosting providers. This was also an opportunity for me to learn more about deploying an application end2end and managing it myself. This post details the exact steps to deploy a Kubernetes cluster to Google Cloud Platform, and run a single, low-utilization application for about $5/mo. It includes all the pitfalls, gotchas, shortcuts, workarounds, optimizations, etc in order to make it work, step by step.

Table of Contents

• Table of Contents
• Create a GCP Account & Project
• Create a GKE Cluster
• Resource Utilization Overview
• Deploy Nginx Ingress
• Create a firewall rule to allow traffic to the nodes
• Deploy a simple app
• Conclusion

Create a GCP Account & Project

1. You can create or select a GCP Project from the Project Selector Page.
2. Make sure that billing is enabled for your Google Cloud Platform project. Learn how to confirm billing is enabled for your project.

For this article, I’ve created a new project called kubernetes-on-the-cheap. For the remainder of the article, I’ll be showing you gcloud or kubectl commands to run instead of walking through how to do this from the UI, though it’s also pretty simple from the UI, it just requires a few more steps and screenshots.

Unless otherwise noted, all commands listed will be run from the GCP Cloud Shell.

To reduce repetition, I’ve aliased kubectl to k

alias k=kubectl

or

echo "alias k=kubectl" >> $HOME/.bashrc

To make your life easier, I’ve variablized the commands in this post, so that you can simply set the variables, then copy/paste the commands without much trouble.

Create a GKE Cluster

GKE (Google Kubernetes Engine) is actually FREE on GCP… well the master nodes are. You can create a cluster without any worker nodes at no cost. That’s the next step. First, navigate to https://console.cloud.google.com/kubernetes/list?project=kubernetes-on-the-cheap but replacing the project with your own project name. Google has to enable the Kubernetes Engine API, and it takes a few minutes.

Here’s the command to run, but you’ll have to fill in a few things yourself. The important part to note here is that this creates a single-zone GKE cluster (for the master nodes). If you don’t do this, when you create node groups, you’ll be forced to have a minimum of 3 nodes (1 per zone). I couldn’t find a way around there. You can read more about this in the docs. The main downside is that when the cluster upgrades, you won’t be able to access the control plane. The data plane (where your apps run) is unaffected.

Most of the next commands can be run from the google cloud console via their website.

Run this locally, and copy the output (or visit the website below to get your home IP address)

MY_HOME_IP="$(curl https://ifconfig.co/ip)"
echo "MY_HOME_IP=${MY_HOME_IP}"

Run this from the Cloud Shell console

export PROJECT_NAME="kubernetes-on-the-cheap"
export CLUSTER_NAME="hobby-1"
export REGION="us-west1"
export ZONE_ID="a"
export ZONE="${REGION}-${ZONE_ID}"

gcloud beta container \
  --project "${PROJECT_NAME}" \
  clusters create "${CLUSTER_NAME}" \
  --zone "${ZONE}" \
  --no-enable-basic-auth \
  --release-channel "regular" \
  --machine-type "g1-small" \
  --image-type "COS" \
  --disk-type "pd-standard" \
  --disk-size "30" \
  --metadata disable-legacy-endpoints=true \
  --scopes "https://www.googleapis.com/auth/devstorage.read_only","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/monitoring","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append" \
  --preemptible \
  --num-nodes "1" \
  --enable-stackdriver-kubernetes \
  --enable-ip-alias \
  --network "projects/${PROJECT_NAME}/global/networks/default" \
  --subnetwork "projects/${PROJECT_NAME}/regions/${REGION}/subnetworks/default" \
  --default-max-pods-per-node "110" \
  --enable-master-authorized-networks \
  --master-authorized-networks "${MY_HOME_IP}/32" \
  --addons HorizontalPodAutoscaling \
  --enable-autoupgrade \
  --enable-autorepair \
  --maintenance-window-start "2019-11-15T10:00:00Z" \
  --maintenance-window-end "2019-11-15T14:00:00Z" \
  --maintenance-window-recurrence "FREQ=WEEKLY;BYDAY=MO,TU,WE,TH"

Resource Utilization Overview

Much of this is coming from StackDriver addon pods. Maybe in another post, I’ll look into an alternative solution, but for now, the remainder is more than enough for a small backend webserver.

Deploy Nginx Ingress

Loadbalancers in GCP have a minimum of $0.60/day or $18/mo charge, so to avoid creating one, we need to setup nginx ingress in a special way.

If nginx ingress controller is set to use the host network, it can bind on port 80 and 443. This means that if we can create a route to any given node on the cluster, 80 and 443 will route to nginx.

The following manifests are derived from the manifests online, but differ in the following ways:

• No service resource is necessary (or desired)
• Remove the reference in the controller args to the service resource (that we aren’t deploying)
• Set CPU/Mem resource requests
• Changing from deployment to daemonset
• Set hostNetwork: true and dnsPolicy: ClusterFirstWithHostNet

Run this to deploy the nginx ingress controller:

kubectl apply -f https://ghostsquad.me/files/kubernetes-on-the-cheap-part-1/nginx-ingress-controller.yaml

Create a firewall rule to allow traffic to the nodes

export TARGET_TAG=$(gcloud compute instances list --format=json | jq -r '.[0].tags.items[0]')

gcloud compute \
  --project=kubernetes-on-the-cheap \
  firewall-rules create http-ingress \
  --direction=INGRESS \
  --priority=1000 \
  --network=default \
  --action=ALLOW \
  --rules=tcp:80,tcp:443 \
  --source-ranges=0.0.0.0/0 \
  --target-tags=$TARGET_TAG

Deploy a simple app

export EXT_IP=$(gcloud compute instances list --format=json | jq -r '.[0].networkInterfaces[0].accessConfigs[0].natIP')
echo $EXT_IP

kubectl apply -f https://ghostsquad.me/files/kubernetes-on-the-cheap-part-1/hello.yaml

You should now be able to visit the IP address in $EXT_IP and see something like this:

Hello, world!
Version: 1.0.0
Hostname: hello-567b7dcdc9-vgb8v

This works, but we have one major issue, and that’s the node IP address will change every 24 hours because the instance is pre-emptible. (and also this means the host in the ingress will need to be updated every 24 hours)

We’ll address this in the next part of this series!

Conclusion

We’ve deployed a cluster to GKE running a single node. We deployed nginx ingress to run on the host network to avoid creating a load balancer, and we deployed a simple test application.

The problems which we need to address in the next part are:

• Snag a domain that points to the cluster, so that we don’t have to update the ingress
• Deploy external-dns to automatically update our domain with the list of IPs of hosts (when the get preempted)
• Deploy cert-manager to get HTTPS!
• Setup node-termination-handler to further improve shutdowns
• Setup SpotInst in order to handle rolling instances on a regular basis in a controlled way, and scaling the cluster temporarily while doing so.

Head over to part 2

Git Commits

Following conventional commits standards can allow you to do things like:

1. Automatically generate CHANGELOGs
2. Automatically determine semantic version bump
3. Communicate nature of changes to teammates, public, stakeholders
4. Used to change build pipeline behavior at runtime to publish code automatically
5. Enhance readability of git commit history

I’m in the SRE/DevOps/PaaS space, and I often find myself wishing I had more time to write weekly newsletters or similar communications to internal teams to increase the visibility of the new features, bugfixes, and other changes that are being delivered. This includes beta releases, and getting volunteers to provide feedback on tools/services before distributing them to the wider engineering organization. With this said, I cannot overstate the value of #1 and #3 in the above list. I don’t mean a hard-to-read, never-ending list either. Imagine being able to generate a releases page like Gitlab’s 12.2 release. This is theoretically possible through a combination of well-written commits (that are machine parseable) and well-written epics, that can be queried.

Combine this with Chris Beams’: How to Write a Git Commit Message and it makes it trivial to go back in time an figure out why something was done.

Project Structure & Naming

Conventions over Configuration. This applies to a project directory structure & naming too. Here’s some benefits of coming up with and sticking to a polyglot convention, similar to Go project structure recommendations.

1. CI/CD templated/generated pipelines
   • If your project name is valid DNS (and all lowercase), it makes it easy to use (without modification) as your kubernetes namespace (or namespace prefix).
   • Dockerfile in the root of your repo? We have a pre-baked step to build/publish without any additional configuration.
   • Using make? A standard pipeline can make some assumptions for you about commands to run without requiring explicit configuration.
2. Enhance readability & maintenance by teammates and external teams. Sticking to a convention means that someone doesn’t need to learn “how your repo does it”.
3. New projects can be templatized/generated. If you have the pleasure of being able to spinup & deploy greenfield applications on a regular basis, this becomes an automatable step.

Tags, Labels, Annotations

Again, this is all about automation. It makes sense to have a standard set of keys and a convention for custom keys that are forward compatible with changes to the standard. Use this same standard for your cloud provider, like AWS, as well as Kubernetes resources.

company.io/environment: prd
company.io/team: sig-api
company.io/contact: sig-api@company.io
company.io/managed-by: terraform
company.io/fingerprint: abc123
company.io/fingerprint-type: sha1
company.io/component: database
company.io/part-of: wordpress
custom.company.io/something-not-standard: this-is-project-or-team-specific

Some of these are duplicates of recommended annotations/labels for kubernetes. Don’t worry about duplication. Use them both, because you’ll apply the ones above to more things than just K8s resources. Tools that may not be written by your team could potential benefit from the k8s standard.

This will certainly be a multi-part series. Stay tuned for more!