Blogging in the Wesland

Flow Overview

#

You arrived at this site via:

sequenceDiagram
  actor User
  participant DNS
  participant Edge
  participant RadosGateway
  participant S3

  User->>Edge: Connect to https://tb.luxolus.com
  Edge->>RadosGateway: Route to internal Ceph RGW replica-set
  RadosGateway-->>DNS: Lookup tb.luxolus.com CNAME for bucket routing
  RadosGateway->>S3: Route to (librados) S3 website bucket content
  S3->>User: Navigate to index.html

This blog post was published via the following:

❯ just hugo deploy
[blog@f230607|~]: Requesting IAM credentials from https://secret.i.st8l.com (you may be required to authenticate)
Password (will be hidden):

[blog@f230607|~]: Running `hugo deploy -e production --target production`
Deploying to target "production" (s3://w3-luxolus-blog?endpoint=https://storage.st8l.com)
Identified 40 file(s) to upload, totaling 942 kB, and 0 file(s) to delete.
Success!

And in just a few seconds, my new post was up and available.

1. Zero cloud technologies involved.
2. Everything along this stack is opensource.
3. You can do this too.

Selfhosting Weslandia

#

Weslandia is a children’s book by Paul Fleischman, and was one of my favorite books when I was growing up. The brief, relevant synopsis:

When Wesley, a somewhat eccentric boy with no friends, discovers a mysterious plant magically growing in his parents’ backyard, he cultivates the plant over his summer vacation. The plant, which he names “swist”, provides him with a food source, and allows him to build shelter, tools, and even create his own entertainment and inspires Wesley to create his own writing system. Wesley’s resourcefulness and meticulous research eventually allow to him the basis of his own civilization which he names “Weslandia”, an eponymous micro-nation in his parents’ backyard. His efforts are successful, and instead of being a social outcast, he gains a group of followers made up of his former grade-school tormentors.

I have built my own digital Wesland. It has all of the managed services that are relevant:

• Managed block, filesystem & blob storage
• Managed VMs
• Managed workload scheduling & placement
• Managed databases
• Managed email MTAs
• Managed load balancers
• Managed queues
• Managed networks / DNS zones
• Managed logging + metrics
• Managed secrets engine

Okay I lie, almost all: I don’t like Lambda/Function runtimes so I don’t use them.

All of the above is highly available excluding the MTA but that’s an intentional choice on my part to save money on public virtual IPs.

I had better uptime in 2024 than every major cloud provider, and unless something horrible happens in 2025, I’m going to beat them all again, this year.

Oh yes, and the price: I run a ~500G mem, ~64 core, 150T disk cluster for slightly over $350/mo. The “equivalent” cost of just the cpu/mem in EC2s excluding storage, because AWS doesn’t really offer an equivalent of what I have is north of 5 grand a month. Add in my 26 databases and many terabytes of storage and we’re pretty easily over 20 grand a month.¹

Instead of $350. For worse uptime. For stagnant service offerings.

Unrelated note but did you hear that AWS’ gross margin as of the last earnings report was just short of 50%? I’m sure that has nothing to do with the prices above.

Oh yeah, last thing: I maintain this in my free time. After a 40-60 hour work week. This is not taking up multiple IT teams worth of budget to manage.

The point of my rant is this:

<endRant />

Right what were we talking about? A blog?!

#

I invite you to join me in Weslandia. Over the next many posts I’m going to go into detail about parts of my setup, so you can build your own.

Going over the command line from above:

1. just is a task runner

I use it for approximately 99% of the usage Make gets outside of c/c++ build systems. If you want a build system use Make (actually, use Bazel but that’s whole ’nother series of posts!), if you want a task runner use just.

You want a task runner by the way.

2. We then reach out to secrets.i.st8l.com

My internal “secrets as a service”… service. It’s just a highly available OpenBao cluster that has a plugin for managing Ceph Rados Gateway credentials.

I login via OIDC, provided by a OpenBao<->LDAP sync bridge plugin.

3. We then invoke hugo deploy with the following config

targets:
  - name: production
    url: s3://w3-luxolus-blog?endpoint=https://storage.st8l.com

matchers:
- cacheControl: max-age=31536000, no-transform, public
  gzip: true
  pattern: ^.+\.(js|css|svg|ttf)$
- cacheControl: max-age=31536000, no-transform, public
  gzip: false
  pattern: ^.+\.(png|jpg)$
- contentType: application/xml
  gzip: true
  pattern: ^sitemap\.xml$

4. Deploying my site to w3-luxolus-blog.storage.st8l.com

Which is an S3 website bucket. With a twist.

If you dig this site’s DNS record you’ll see something impossible interesting:

❯ drill tb.luxolus.com
;; ANSWER SECTION:
tb.luxolus.com. 241     IN      CNAME   w3-luxolus-blog.w3storage.st8l.com.
w3-luxolus-blog.w3storage.st8l.com.     241     IN      CNAME   w3storage.st8l.com.
w3storage.st8l.com.     241     IN      CNAME   st8l.com.
st8l.com.       241     IN      A       159.69.232.211

Which, if you have worked with S3 website buckets before should be very surprising to you. This… shouldn’t work. AWS’ own documentation² says this setup isn’t supported:

Amazon S3 website endpoints do not support HTTPS or access points. If you want to use HTTPS, you can use Amazon CloudFront to serve a static website hosted on Amazon S3.

So rather than using 1 cheap service to accomplish what free, opensource software can do, you can instead use 3 services, one of them being expensive: S3, R53, and CloudFront.

Finitione

#

I really like my setup. It’s easier to deal with than substack, and I have half a mind to add a git post-receive hook on master pushes that automatically deploys to prod when I push, so I don’t even need to remember to hugo deploy myself.

It reminds me of the wonder I felt when I first used Heroku, pushed to GitHub the first time and had a running site.

And it’s all mine…

but you could have it too