Kubernetes on the Cheap - Part 1

How to run Kubernetes for less than $6 per month?

I wanted to play around with Kubernetes for personal learning, and even some personal projects. I wanted to deploy a website with a live backend, but I didn’t really want to be bound to cPanel, or terrible performance issues on shared hosting providers. This was also an opportunity for me to learn more about deploying an application end2end and managing it myself. This post details the exact steps to deploy a Kubernetes cluster to Google Cloud Platform, and run a single, low-utilization application for about $5/mo. It includes all the pitfalls, gotchas, shortcuts, workarounds, optimizations, etc in order to make it work, step by step.

Table of Contents

• Table of Contents
• Create a GCP Account & Project
• Create a GKE Cluster
• Resource Utilization Overview
• Deploy Nginx Ingress
• Create a firewall rule to allow traffic to the nodes
• Deploy a simple app
• Conclusion

Create a GCP Account & Project

1. You can create or select a GCP Project from the Project Selector Page.
2. Make sure that billing is enabled for your Google Cloud Platform project. Learn how to confirm billing is enabled for your project.

For this article, I’ve created a new project called kubernetes-on-the-cheap. For the remainder of the article, I’ll be showing you gcloud or kubectl commands to run instead of walking through how to do this from the UI, though it’s also pretty simple from the UI, it just requires a few more steps and screenshots.

Unless otherwise noted, all commands listed will be run from the GCP Cloud Shell.

To reduce repetition, I’ve aliased kubectl to k

alias k=kubectl

or

echo "alias k=kubectl" >> $HOME/.bashrc

To make your life easier, I’ve variablized the commands in this post, so that you can simply set the variables, then copy/paste the commands without much trouble.

Create a GKE Cluster

GKE (Google Kubernetes Engine) is actually FREE on GCP… well the master nodes are. You can create a cluster without any worker nodes at no cost. That’s the next step. First, navigate to https://console.cloud.google.com/kubernetes/list?project=kubernetes-on-the-cheap but replacing the project with your own project name. Google has to enable the Kubernetes Engine API, and it takes a few minutes.

Here’s the command to run, but you’ll have to fill in a few things yourself. The important part to note here is that this creates a single-zone GKE cluster (for the master nodes). If you don’t do this, when you create node groups, you’ll be forced to have a minimum of 3 nodes (1 per zone). I couldn’t find a way around there. You can read more about this in the docs. The main downside is that when the cluster upgrades, you won’t be able to access the control plane. The data plane (where your apps run) is unaffected.

Most of the next commands can be run from the google cloud console via their website.

Run this locally, and copy the output (or visit the website below to get your home IP address)

MY_HOME_IP="$(curl https://ifconfig.co/ip)"
echo "MY_HOME_IP=${MY_HOME_IP}"

Run this from the Cloud Shell console

export PROJECT_NAME="kubernetes-on-the-cheap"
export CLUSTER_NAME="hobby-1"
export REGION="us-west1"
export ZONE_ID="a"
export ZONE="${REGION}-${ZONE_ID}"

gcloud beta container \
  --project "${PROJECT_NAME}" \
  clusters create "${CLUSTER_NAME}" \
  --zone "${ZONE}" \
  --no-enable-basic-auth \
  --release-channel "regular" \
  --machine-type "g1-small" \
  --image-type "COS" \
  --disk-type "pd-standard" \
  --disk-size "30" \
  --metadata disable-legacy-endpoints=true \
  --scopes "https://www.googleapis.com/auth/devstorage.read_only","https://www.googleapis.com/auth/logging.write","https://www.googleapis.com/auth/monitoring","https://www.googleapis.com/auth/servicecontrol","https://www.googleapis.com/auth/service.management.readonly","https://www.googleapis.com/auth/trace.append" \
  --preemptible \
  --num-nodes "1" \
  --enable-stackdriver-kubernetes \
  --enable-ip-alias \
  --network "projects/${PROJECT_NAME}/global/networks/default" \
  --subnetwork "projects/${PROJECT_NAME}/regions/${REGION}/subnetworks/default" \
  --default-max-pods-per-node "110" \
  --enable-master-authorized-networks \
  --master-authorized-networks "${MY_HOME_IP}/32" \
  --addons HorizontalPodAutoscaling \
  --enable-autoupgrade \
  --enable-autorepair \
  --maintenance-window-start "2019-11-15T10:00:00Z" \
  --maintenance-window-end "2019-11-15T14:00:00Z" \
  --maintenance-window-recurrence "FREQ=WEEKLY;BYDAY=MO,TU,WE,TH"

Resource Utilization Overview

Much of this is coming from StackDriver addon pods. Maybe in another post, I’ll look into an alternative solution, but for now, the remainder is more than enough for a small backend webserver.

Deploy Nginx Ingress

Loadbalancers in GCP have a minimum of $0.60/day or $18/mo charge, so to avoid creating one, we need to setup nginx ingress in a special way.

If nginx ingress controller is set to use the host network, it can bind on port 80 and 443. This means that if we can create a route to any given node on the cluster, 80 and 443 will route to nginx.

The following manifests are derived from the manifests online, but differ in the following ways:

• No service resource is necessary (or desired)
• Remove the reference in the controller args to the service resource (that we aren’t deploying)
• Set CPU/Mem resource requests
• Changing from deployment to daemonset
• Set hostNetwork: true and dnsPolicy: ClusterFirstWithHostNet

Run this to deploy the nginx ingress controller:

kubectl apply -f https://ghostsquad.me/files/kubernetes-on-the-cheap-part-1/nginx-ingress-controller.yaml

Create a firewall rule to allow traffic to the nodes

export TARGET_TAG=$(gcloud compute instances list --format=json | jq -r '.[0].tags.items[0]')

gcloud compute \
  --project=kubernetes-on-the-cheap \
  firewall-rules create http-ingress \
  --direction=INGRESS \
  --priority=1000 \
  --network=default \
  --action=ALLOW \
  --rules=tcp:80,tcp:443 \
  --source-ranges=0.0.0.0/0 \
  --target-tags=$TARGET_TAG

Deploy a simple app

export EXT_IP=$(gcloud compute instances list --format=json | jq -r '.[0].networkInterfaces[0].accessConfigs[0].natIP')
echo $EXT_IP

kubectl apply -f https://ghostsquad.me/files/kubernetes-on-the-cheap-part-1/hello.yaml

You should now be able to visit the IP address in $EXT_IP and see something like this:

Hello, world!
Version: 1.0.0
Hostname: hello-567b7dcdc9-vgb8v

This works, but we have one major issue, and that’s the node IP address will change every 24 hours because the instance is pre-emptible. (and also this means the host in the ingress will need to be updated every 24 hours)

We’ll address this in the next part of this series!

Conclusion

We’ve deployed a cluster to GKE running a single node. We deployed nginx ingress to run on the host network to avoid creating a load balancer, and we deployed a simple test application.

The problems which we need to address in the next part are:

• Snag a domain that points to the cluster, so that we don’t have to update the ingress
• Deploy external-dns to automatically update our domain with the list of IPs of hosts (when the get preempted)
• Deploy cert-manager to get HTTPS!
• Setup node-termination-handler to further improve shutdowns
• Setup SpotInst in order to handle rolling instances on a regular basis in a controlled way, and scaling the cluster temporarily while doing so.

Head over to part 2