Abhay Rana aka Nemo

Running terraform and docker on my home server

The last time I’d posted about my Home Server build in September, I’d just gotten it working. Since then, I’ve made a lot of progress. It is now running almost 10 services, up from just Kodi back then. Now it has a working copy of:

Kodi
I was running kodi-standalone-service, set to run on boot, as per the ArchLinux Wiki, but switched in favor of openbox to a simple autorun.
Steam
The current setup uses Steam as the application launcher. This lets me ensure that the Steam Controller works across all applications.
Openbox
Instead of running Kodi on xinit, I’m now running openbox with autologin against a non-privileged user.
PulseAudio
I tried fighting it, but it was slightly easier to configure compared to dmix. Might move to dmix if I get time.
btrfs
I now have the following disks:
  1. 128GB root volume. (Samsung EVO-850)
  2. 1TB volume for data backups
  3. 3TB RAID0 configuration across 2 disks. There are some btrfs subvolumes in the 3TB raid setup, including one specifically for docker volumes. The docker guide recommends running btrfs subvolumes on the block device, which I didn’t like, so I’m running docker volumes in normal mode on a btrfs disk. I don’t have enough writes to care much yet, but might explore this further.
Docker
This has been an interesting experiment. Kodi is still installed natively, but I’ve been trying to run almost everything else as a docker container. I’ve managed to do the configuration entirely via terraform, which has been a great learning experience. I’ve found terraform much more saner as a configuration system compared to something like ansible, which gets quite crazy. (We have a much more crazy terraform config at work, though).
Terraform
I have a private repository on GitLab called nebula which I use as the source of truth for the configuration. It doesn’t hold everything yet, just the following:
  1. Docker Configuration (not the docker service, just the container/volumes)
  2. CloudFlare - I’m using bb8.fun as the root domain, which is entirely managed using the CloudFlare terraform provider.
  3. MySQL - Running a MariaDB container, which has been configured by-hand till this PR gets merged.
Gitea
Running as a docker container, provisioned using terraform. Plan to proxy this using git.captnemo.in.
Emby
Docker Container. Nothing special. Plan to set this up as the Kodi backend.
Couchpotato
Experimental setup for now. Inside a docker container.
Flexget
I wish I knew how to configure this. Also inside docker.
traefik
Running as a simple reverse proxy for most of the above services
elibsrv
A simple OPDS server, which I use against my Kindle. If you don’t know what OPDS is, you should [check this out][]. Running on a simple apache setup on the archlinux box for now. WIP for dockerization.
ubooquity
Simple ebook server. Proxied over the internet. Has a online ebook reader, which is pretty cool.
MariaDB
I set this up planning to shift Kodi’s data to this, but now that I have emby setup - I’m not so sure. Still, keeping this running for now.
Transmission
Hooked up to couchpotato,flexget, and sickrage so it can do things.
Sickrage
Liking this more than flexget so far, much more easier to configure and use.
AirSonic
This is the latest fork of libresonic, which was itself forked off subsonic. My attempt at getting off Google Play Music.

Learnings

I forgot to do this on the last blog post, so here is the list:

  1. archlinux has official packages for intel-microcode-updates.
  2. wireguard is almost there. I’m running openvpn for now, waiting for the stable release.
  3. While traefik is great, I’m concerned about the security model it has for connecting to Docker (uses the docker unix socket over a docker mounted volume, which gives it root access on the host). Scary stuff.
  4. Docker Labels are a great signalling mechanism.
  5. Terraform still needs a lot of work on their docker provider. A lot of updates destroy containers, which should be applied without needing a destroy.
  6. I can’t proxy gitea’s SSH authentication easily, since traefik doesn’t support TCP proxying yet.
  7. The docker_volume resource in terraform is useless, since it doesn’t give you any control over the volume location on the host.
  8. The upload block inside a docker_container resource is a great idea. Lets you push configuration straight inside a container. This is how I push configuration straight inside the traefik container for eg:
    upload {
      content = "${file("${path.module}/conf/traefik.toml")}"
      file    = "/etc/traefik/traefik.toml"
    }
    

Advice

This section is if you’re venturing into a docker-heavy terraform setup:

  1. Use traefik. Will save you a lot of pain with proxying requests.
  2. Repeat the ports section for every IP you want to listen on. CIDRs don’t work.
  3. If you want to run the container on boot, you want the following:
     restart = "unless-stopped"
     destroy_grace_seconds = 10
     must_run = true
    
  4. If you have a single docker_registry_image resource in your state, you can’t run terraform without internet access.
  5. Breaking your docker module into images.tf, volumes.tf, and data.tf (for registry_images) works quite well.
  6. Memory limits on docker containers can be too contrained. Keep an eye on logs to see if anything is getting killed.
  7. Setup Docker TLS auth first. I tried proxying Docker over apache with basic auth, but it didn’t work out well.

When I’d started the project, I’d looked at GUI based docker management solutions (Portainer, Shipyard, Rancher), but I’ve found that the tough parts for me (container linking, ports, environments, volumes) were all handled much more cleanly by just terraform.

TODO

A few things off my TODO list:

  1. Create a Docker image for elibsrv that comes with both ebook-convert and kindlegen pre-installed
  2. Do the same for ubooquity as well

Home Server Build

I’d been planning to run my own home server for a while, and this culminated in a mini-ITX build recently. The exact part list is up at https://in.pcpartpicker.com/list/krc8Gf.

In no particular order, here were the constraints:

  • The case should be small (I preferred the Elite 110, but it was unavailable everywhere).
  • Dual LAN, if possible (decided against it at the end). The plan was to run the entire home network from this directly by plugging in the ISP into the server.
  • Recent i3/i5 for amd64 builds.
  • Enough SATA bays in the cabinet for storage

The plans for the server:

  1. Scheduled backups from other sources (Android/Laptop)
  2. Run Kodi (or perhaps switch to Emby)
  3. Run torrents. Transmission-daemon works. Preferably something pluggable and that works with RSS
  4. Do amd64 builds. See https://github.com/captn3m0/ideas#arch-linux-package-build-system
  5. Host a webserver. This is primarily for serving resources off the internet
    • Host some other minor web-services
    • A simple wiki
    • Caldav server
    • Other personal projects
  6. Sync Server setup. Mainly for the Kindle and the phone.
  7. Calibre-server, koreader sync server for the Kindle
    • Now looking at libreread as well
  8. Tiny k8s cluster for running other webapps
  9. Run a graylog server for sending other system log data (using papertrail now, has a 200MB limit)

No plans to move mail hosting. That will stay at migadu.com for now.

I had a lot of spare HDDs that I was going to re-use for this build:

  1. WD MyBook 3TB (external, shelled).
  2. Seagate Expansion: 1TB
  3. Seagate Expansion 3TB (external, shelled)
  4. Samsung EVO 128GB SSD

The 2x3TB disks are setup with RAID1 over btrsfs. Important data is snapshotted to the other 1TB disk using btrfs snapshots and subvolumes. In total giving me ~4TB of storage

Software

Currently running kodi-standalone-service on boot. Have to decide on a easy-to-use container orchestration platform. Choices as of now are:

  1. Rancher
  2. Docker Swarm
  3. Shipyard
  4. Terraform
  5. Portainer

Most of these are tuned for multi-host setups, and bring in a lot of complexity as a result. Looking at Portainer, which seems well suited to a single-host setup.

Other services I’m currently running:

  1. elibsrv. Running a patched build with support for ebook-convert
  2. ubooquity for online reading of comics

Project Updates

Over the last couple of years, I’ve been involved with lots of side projects, both online and offline. Some of them, I’ve written about on the blog, like my music visualizer project. A few of them, got their own project page, like the website for my niece (but no blog post) while some didn’t even get a mention. I thought I’d write about the many-many side projects I’ve started (and abandoned). You might also wanna visit the /projects page for the larger projects.

Home Server Build
Sep 2017 Built a home server, mostly as a HTPC but also as a learning exercise for managing services over Docker.
Sushi Go
Summer 2017 This is a work-in-progress conversion of Sushi Go (original), the popular card game by Gamewright into Ruby.
youtube-ripper
June 2017 Downloads music-compilations from YouTube and rips them into multiple tagged MP3 files.
cosmere-books
September 2017 Wrote a EPUB generator for multiple books in the Cosmere. Currently covers 4 different serializations at Tor.com. Also created a project page on all of my ebooks projects at /ebooks/
ideas
Ongoing I maintain a CC0 licensed list of personal ideas. Feel free to use.
spectrumyzer
May 2017 Created an animated wallpaper using spectrumyzer. Wrote a blog post about it.
google-sre
Feb/Sep 2017 EPUB generator for the Google SRE ebook. Started in February in Python. Gave up and redid it properly in September.
CodeChef Offline
March 2012 I attempted to make a offline repository for CodeChef problems. I spent some time in May 2017 upgrading the project with a cleaner scraper and a Jekyll base.
Hoshruba
June 2015 I wrote a script that scraped Tor’s serialized publication of the first book in Hoshruba series to generate EPUB and MOBI files. I would recommend the book if you are interested in reading what many would term the “original fantasy book”
HackerTray
December 2013 - I wrote a Linux PyGTK application that sits in your taskbar using Indicator Applet to show you the latest stories from Hacker News. Looking for a maintainer.
MagicMuggle
May 2017 I wrote a script to convert Magic Muggle (A Harry Potter fanfic about a muggle who accidentally gets into Hogwarts) books from their original reddit posts to EPUB and MOBI files.
Kerala IT Policy
March 2017 Attempted to transcribe the draft IT policies put up by the Government of Kerala. Lots of OCR followed by manual fixes. I stopped working on this when I realized that the government had actually put up a really nice website for this (with clear plaintext, not the bad PDF I was using as the source).
lightsaber
August 2015 I created a DNS based HTTP-3xx redirect service. Useful if you own a domain and you want it to be redirected, but don’t have a webserver with you. Made as part of the Django Hackathon organized by HackerEarth in Ruby.
HackerCouch
November 2015 My hack during hackbeach 2015. Created something best described as “couchsurfing for hackers”. Simple Jekyll/Ruby website hosted on GitHub Pages.