Sign in Subscribe
By Samir F in ghost —

Running Ghost in Kubernetes

Running Ghost in Kubernetes

Reference Table

Reference Table

ACME: Automated Certificate Management Environment, a way to get a TLS certificate commonly used by LetsEncrypt.
CRDs: CustomResourceDefinition in K8s. A way to define a custom data type that will be used K8s entities.
Horizontal scaling: Scaling by adding more nodes that allow for work to be done.
Gateway: The K8s API that replaces Ingress
Ingress: An K8s API that defines a load balancer now replaced with its successor.
K8s : Kubernetes
K3s: A lightweight Kubernetes Distribution
Vertical scaling: upgrading a single or set of cluster to high end compute nodes able to handle the new load.

My home lab is mostly running off a K3s install these days, so anything I run tends to be in kubernetes. Not everything needs to be in k8s and ghost is definitely not optimized for horizontal scaling. Still, it is nice to create a full install using yaml files.

The main limitation with ghost is that you can't really scale the replica count. You can only reliably run a single instance.

Limitations:

  • Scheduler conflicts: Ghost's built-in job scheduler (for email newsletters, post scheduling) runs in every instance. Without coordination, multiple pods will fire the same jobs simultaneously, causing duplicate emails sent to subscribers, duplicate post publishes, and race conditions on time-sensitive tasks. This could be solved with redis or by using postgres that has a pub/sub pattern but with the way things are this would be a blocker.
  • File-based storage: The default local file storage for images/media lives on disk. In a multi-pod setup, each pod has its own filesystem — uploads to Pod A won't be visible to Pod B. You must replace local storage with an object store (S3, GCS, Azure Blob) via a Ghost Storage Adapter. Or you would need to install a plugin for ghost to allow that. Both of the ones I'm aware of have not had a lot of love.
  • In-memory caching: Ghost caches settings, routes, and some content in process memory. Pods don't share this cache, so cache invalidation across instances is not automatic.

So basically, keep your replica count at 1 to avoid upsetting your subscribers or creating a website with multiple 404s that are non deterministic.

For

Requirements:

There are two high level areas of components. One category is general infrastructure, and the other category is the application and supporting elements.

Infrastructure:

  • DHCP server equivalent IPPool ( in K8s)
  • Load Balancer of some kind (Gateway Envoy in my case)
  • Valid TLS

Application:

  • Database (MariaDB/MySQL)
  • Ghost

Infrastructure


A lot of this is not really inherent to running ghost but it should be touched on. I'm going to do a pretty high level introduction but I'll reference all the components that are needed.

MetalLB


metallb is the easiest way I've found to get an IPPool. Essentially no matter if you run Ingress or Gateway you will need something that manages IP allocation. If you use a cloud provider that's handled by your cloud provider. If you run things locally, metallb does that for you.

You'll need to install metallb first. You can follow the docs but helm chart is the pattern I used:

helm repo add metallb https://metallb.github.io/metallb
helm install metallb metallb/metallb
helm install metallb metallb/metallb -f values.yaml

I didn't really add anything to the values.yaml but you can add your customizations.

apiVersion: metallb.io/v1beta1
kind: IPAddressPool
metadata:
  name: external-pool
  namespace: metallb-system
spec:
  addresses:
    - 1.2.3.4/32
    - fd37:beef:cafe::1/64
  autoAssign: true
---
apiVersion: metallb.io/v1beta1
kind: L2Advertisement
metadata:
  name: external
  namespace: metallb-system
spec:
  ipAddressPools:
    - external-pool

Envoy Gateway

If you are using ingress skip over this and to your own thing.

You need to install the gateway API CRDs. 

helm template eg oci://docker.io/envoyproxy/gateway-crds-helm \
  --version v1.8.0 \
  --set crds.gatewayAPI.enabled=true \
  --set crds.gatewayAPI.channel=standard \
  --set crds.envoyGateway.enabled=true \
  | kubectl apply --server-side -f -

Then install the actual helm chart that installs the controller:

helm install eg oci://docker.io/envoyproxy/gateway-helm \
  --version v1.8.0 \
  -n envoy-gateway-system \
  --create-namespace \
  --skip-crds

This sets up envoy but once again you have no config, this just installs the watcher.

You may need to add the gateway class as well:

apiVersion: gateway.networking.k8s.io/v1
kind: GatewayClass
metadata:
  name: eg
spec:
  controllerName: gateway.envoyproxy.io/gatewayclass-controller

Cert-Manager


Cert-Manager is how you do ACME/Lets Encrypt in a K8s cluster. Its the equivalent of installing certbot. It won't do anything without you asking it to do so, but it installs the tool. Same pattern as before.

helm install \
  cert-manager oci://quay.io/jetstack/charts/cert-manager \
  --version v1.20.2 \
  --create-namespace \
  -f values.yml

My values.yml file is below, the main thing that's import since I'm using gateway is to ensure the gateway API is enabled:

crds:
  enabled: true
  keep: true

config:
  enableGatewayAPI: true

namespace: cert-manager

Here, we'll create two issuers that you can use:

"Production", gets a real cert:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: somev@alidemail.com
    privateKeySecretRef:
      name: letsencrypt-prod-account-key
    solvers:
      - http01:
          gatewayHTTPRoute:
            parentRefs:
              - name: http-gateway
                namespace: default
                kind: Gateway

And one called "Staging" for testing:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: somev@alidemail.com
    privateKeySecretRef:
      name: letsencrypt-staging-account-key
    solvers:
      - http01:
          gatewayHTTPRoute:
            parentRefs:
              - name: http-gateway
                namespace: default
                kind: Gateway

Envoy Continued:

Now, we're ready to configure the Gateway and Certs

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: gateway-tls-cert
  namespace: default
spec:
  secretName: gateway-tls-cert
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
    - auth.domain.com  ## will be used in part 2.
    - blog.domain.com  ## your blog site

I'm using the default NS for the cert and gateway but feel free to put in anywhere you like but the gateway will need to be able to access the cert by either being in the same namespace or adding a ReferenceGrant that will grant it access.

Standard gateway for HTTP/HTTPS:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: http-gateway
  namespace: default
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt-prod ##Point this to staging for test
spec:
  gatewayClassName: eg
  listeners:
    - name: http
      protocol: HTTP
      port: 80
      allowedRoutes:
        namespaces:
          from: All
    - name: https
      protocol: HTTPS
      port: 443
      allowedRoutes:
        namespaces:
          from: All
      tls:
        mode: Terminate
        certificateRefs:
          - name: gateway-tls-cert
            kind: Secret

That's it for Infrastructure. Any other app that you add will usually just get a new DNS name added and we'll attach a route to the gateway.

Backups

I'll add this as a footnote. I use velero which backs up all my volumes to S3/GCP. So my database, volume, DB are all available for restore. A DB backup is recommended but this gives me a baseline. I'll let you explore velero or longhorn on your own.

Application

Okay, so now we get to the app itself. We need a database to run with this. You COULD just run a mariadb but since I have a cluster might as well do it right.

MariaDB

You need to install the operator and the CRDs:

helm repo add mariadb-operator https://helm.mariadb.com/mariadb-operator
helm repo update mariadb-operator
## CRDs
helm install mariadb-operator-crds mariadb-operator/mariadb-operator-crds
## Operator
helm install mariadb-operator mariadb-operator/mariadb-operator

Setting up the database

Feel free to tweak these settings but this does expect a secret to exist named ghost-db with the credentials. I use External Secret Operator and left my pattern there in the manifest as an example:

## Define your secrets
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: ghost-db
  namespace: mariadb
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: onepassword-cluster-store
    kind: ClusterSecretStore
  target:
    name: ghost-db
    creationPolicy: Owner
  data:
    - secretKey: password
      remoteRef:
        key: seafile-db
        property: ghost_user_password
---
## Define Database
apiVersion: k8s.mariadb.com/v1alpha1
kind: Database
metadata:
  name: ghost
  namespace: mariadb
spec:
  mariaDbRef:
    name: mariadb
  characterSet: utf8mb4
  collate: utf8mb4_general_ci
  cleanupPolicy: Skip
  requeueInterval: 10h
  retryInterval: 30s
---
## Create the DB user
apiVersion: k8s.mariadb.com/v1alpha1
kind: User
metadata:
  name: ghost
  namespace: mariadb
spec:
  mariaDbRef:
    name: mariadb
  passwordSecretKeyRef:
    name: ghost-db
    key: password
  host: "%"
  cleanupPolicy: Skip
  requeueInterval: 10h
  retryInterval: 30s
---
## Define your Grant giving the user access to the DB
apiVersion: k8s.mariadb.com/v1alpha1
kind: Grant
metadata:
  name: ghost
  namespace: mariadb
spec:
  mariaDbRef:
    name: mariadb
  privileges:
    - "ALL PRIVILEGES"
  database: "ghost"
  table: "*"
  username: ghost
  host: "%"
  cleanupPolicy: Delete
  requeueInterval: 10h
  retryInterval: 30s

That's it. Now I should be able to connect to my database via: mariadb.mariadb.svc.cluster.local in my case the mariadb operator and app are in the mariadb ns with a service named mariadb as well.

Now, this is still technically a single pod, no replica. If you really wanted to you could scale mariadb but since we're only running one ghost instance it seems silly.

For science though setting up a HA cluster docs can be found here.

Ghost

Now that we have our database up and running we really just need to bring the pods up and add a route.

First PVC, where the data is stored

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: ghost-content
  namespace: ghost
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi ## make this bigger

I'm VERY conservative with my allocation, I'm running far too many thing on a tiny VPS, but adjust it to your resources.

Deployment, this assumes we have a ghost-db secret created. If you mostly followed what I did all of this should be copy/paste with the exception of the domain name.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ghost
  namespace: ghost
spec:
  replicas: 1
  revisionHistoryLimit: 2
  selector:
    matchLabels:
      app: ghost
  template:
    metadata:
      labels:
        app: ghost
    spec:
      containers:
        - name: ghost
          image: ghost:6.39.0
          ports:
            - containerPort: 2368
          env:
            - name: NODE_ENV
              value: production
            - name: url
              value: https://blog.domain.com
            - name: database__client
              value: mysql
            - name: database__connection__host
              value: mariadb.mariadb.svc.cluster.local
            - name: database__connection__port
              value: "3306"
            - name: database__connection__database
              value: ghost
            - name: database__connection__user
              value: ghost
            - name: database__connection__password
              valueFrom:
                secretKeyRef:
                  name: ghost-db
                  key: password
            - name: mail__transport
              value: SMTP
            - name: mail__options__host
              value: smtp.mailgun.org
            - name: mail__options__port
              value: "587"
            - name: mail__options__secure
              value: "false"
            - name: mail__from
              value: donotreply@mailings.somedomain.com
            - name: mail__options__auth__user
              valueFrom:
                secretKeyRef:
                  name: ghost-db
                  key: smtp-username
            - name: mail__options__auth__pass
              valueFrom:
                secretKeyRef:
                  name: ghost-db
                  key: smtp-password
          volumeMounts:
            - name: content
              mountPath: /var/lib/ghost/content
      volumes:
        - name: content
          persistentVolumeClaim:
            claimName: ghost-content

Service:

apiVersion: v1
kind: Service
metadata:
  name: ghost
  namespace: ghost
spec:
  selector:
    app: ghost
  ports:
    - port: 80
      targetPort: 2368

Route, how we connect to our app:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: ghost-https
  namespace: ghost
spec:
  parentRefs:
    - name: http-gateway
      namespace: default
      sectionName: https
  hostnames:
    - blog.domain.com
  rules:
    - backendRefs:
        - name: ghost
          port: 80

OR if you'd rather serve traffic off a URI path:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: ghost-https
  namespace: ghost
spec:
  parentRefs:
    - name: http-gateway
      namespace: default
      sectionName: https
  hostnames:
    - domain.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /blog
      backendRefs:
        - name: ghost
          port: 80

And update your deployment accordingly:

        env:
            - name: url
              value: https://blog.domain.com/blog

That's it. log in to your blog: https://blog.domain.com/ghost/ or https://blog.domain.com/blog/ghost/.

Finally, if you want people who for some reason are not using https by default to find you, may want to add a redirect. Adjust this accordingly based on the setup you have. 

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: ghost-http-redirect
  namespace: ghost
spec:
  parentRefs:
    - name: http-gateway
      namespace: default
      sectionName: http
  hostnames:
    - blog.domain.com
  rules:
    - filters:
        - type: RequestRedirect
          requestRedirect:
            scheme: https
            statusCode: 301

References:

Subscribe to csgeek blog

Don’t miss out on the latest issues. Sign up now to get access to the library of members-only issues.
jamie@example.com
Subscribe