Running terraform and docker on my home server

09 Nov 2017

The last time I’d posted about my Home Server build in September, I’d just gotten it working. Since then, I’ve made a lot of progress. It is now running almost 10 services, up from just Kodi back then. Now it has a working copy of:

Kodi

I was running kodi-standalone-service, set to run on boot, as per the ArchLinux Wiki, but switched in favor of openbox to a simple autorun.

Steam

The current setup uses Steam as the application launcher. This lets me ensure that the Steam Controller works across all applications.

Openbox

Instead of running Kodi on xinit, I’m now running openbox with autologin against a non-privileged user.

PulseAudio

I tried fighting it, but it was slightly easier to configure compared to dmix. Might move to dmix if I get time.

btrfs

I now have the following disks:

128GB root volume. (Samsung EVO-850)
1TB volume for data backups
3TB RAID0 configuration across 2 disks. There are some btrfs subvolumes in the 3TB raid setup, including one specifically for docker volumes. The docker guide recommends running btrfs subvolumes on the block device, which I didn’t like, so I’m running docker volumes in normal mode on a btrfs disk. I don’t have enough writes to care much yet, but might explore this further.

Docker

This has been an interesting experiment. Kodi is still installed natively, but I’ve been trying to run almost everything else as a docker container. I’ve managed to do the configuration entirely via terraform, which has been a great learning experience. I’ve found terraform much more saner as a configuration system compared to something like ansible, which gets quite crazy. (We have a much more crazy terraform config at work, though).

Terraform

I have a private repository on GitLab called nebula which I use as the source of truth for the configuration. It doesn’t hold everything yet, just the following:

Docker Configuration (not the docker service, just the container/volumes)
CloudFlare - I’m using bb8.fun as the root domain, which is entirely managed using the CloudFlare terraform provider.
MySQL - Running a MariaDB container, which has been configured by-hand till this PR gets merged.

Gitea

Running as a docker container, provisioned using terraform. Plan to proxy this using git.captnemo.in.

Emby

Docker Container. Nothing special. Plan to set this up as the Kodi backend.

Couchpotato

Experimental setup for now. Inside a docker container.

Flexget

I wish I knew how to configure this. Also inside docker.

traefik

Running as a simple reverse proxy for most of the above services

elibsrv

A simple OPDS server, which I use against my Kindle. If you don’t know what OPDS is, you should [check this out][]. Running on a simple apache setup on the archlinux box for now. WIP for dockerization.

ubooquity

Simple ebook server. Proxied over the internet. Has a online ebook reader, which is pretty cool.

MariaDB

I set this up planning to shift Kodi’s data to this, but now that I have emby setup - I’m not so sure. Still, keeping this running for now.

Transmission

Hooked up to couchpotato,flexget, and sickrage so it can do things.

Sickrage

Liking this more than flexget so far, much more easier to configure and use.

AirSonic

This is the latest fork of libresonic, which was itself forked off subsonic. My attempt at getting off Google Play Music.

Learnings

Moved these to a separate blog post

TODO

A few things off my TODO list:

Create a Docker image for elibsrv that comes with both ebook-convert and kindlegen pre-installed
~~Do the same for ubooquity as well~~ (Using the linuxserver/ubooquity docker image)

If you’re interested in my self-hosting setup, I’m using Terraform + Docker, the code is hosted on the same server, and I’ve been writing about my experience and learnings:

If you have any comments, reach out to me

Home Server Build

17 Sep 2017

I’d been planning to run my own home server for a while, and this culminated in a mini-ITX build recently. The current build configuration is available at /setup/homeserver/.

In no particular order, here were the constraints:

The case should be small (I preferred the Elite 110, but it was unavailable in India).
Dual LAN, if possible (decided against it at the end). The plan was to run the entire home network from this directly by plugging in the ISP into the server.
Recent i3/i5 for amd64 builds.
Enough SATA bays in the cabinet for storage

The plans for the server:

Scheduled backups from other sources (Android/Laptop)
Run Kodi (or perhaps switch to Emby)
Run torrents. Transmission-daemon works. Preferably something pluggable and that works with RSS
Do amd64 builds. See https://github.com/captn3m0/ideas#arch-linux-package-build-system
Host a webserver. This is primarily for serving resources off the internet
- Host some other minor web-services
- A simple wiki
- Caldav server
- Other personal projects
Sync Server setup. Mainly for the Kindle and the phone.
Calibre-server, koreader sync server for the Kindle
- Now looking at libreread as well
Tiny k8s cluster for running other webapps
Run a graylog server for sending other system log data (using papertrail now, has a 200MB limit)

No plans to move mail hosting. That will stay at migadu.com for now.

I had a lot of spare HDDs that I was going to re-use for this build:

WD MyBook 3TB (external, shelled).
Seagate Expansion: 1TB
Seagate Expansion 3TB (external, shelled)
Samsung EVO 128GB SSD

The 2x3TB disks are setup with RAID1 over btrsfs. Important data is snapshotted to the other 1TB disk using btrfs snapshots and subvolumes. In total giving me ~4TB of storage.

Software

Currently running kodi-standalone-service on boot. Have to decide on a easy-to-use container orchestration platform. Choices as of now are:

Rancher
Docker Swarm
Shipyard
Terraform
Portainer

Most of these are tuned for multi-host setups, and bring in a lot of complexity as a result. Looking at Portainer, which seems well suited to a single-host setup.

Other services I’m currently running:

elibsrv. Running a patched build with support for ebook-convert
ubooquity for online reading of comics

If you’re interested in my self-hosting setup, I’m using Terraform + Docker, the code is hosted on the same server, and I’ve been writing about my experience and learnings:

If you have any comments, reach out to me

Project Updates

16 Sep 2017

Over the last couple of years, I’ve been involved with lots of side projects, both online and offline. Some of them, I’ve written about on the blog, like my music visualizer project. A few of them, got their own project page, like the website for my niece (but no blog post) while some didn’t even get a mention. I thought I’d write about the many-many side projects I’ve started (and abandoned). You might also wanna visit the /projects page for the larger projects.

Home Server Build: Sep 2017 Built a home server, mostly as a HTPC but also as a learning exercise for managing services over Docker.
Sushi Go: Summer 2017 This is a work-in-progress conversion of Sushi Go (original), the popular card game by Gamewright into Ruby.
youtube-ripper: June 2017 Downloads music-compilations from YouTube and rips them into multiple tagged MP3 files.
cosmere-books: September 2017 Wrote a EPUB generator for multiple books in the Cosmere. Currently covers 4 different serializations at Tor.com. Also created a project page on all of my ebooks projects at /ebooks/
ideas: Ongoing I maintain a CC0 licensed list of personal ideas. Feel free to use.
spectrumyzer: May 2017 Created an animated wallpaper using spectrumyzer. Wrote a blog post about it.
google-sre: Feb/Sep 2017 EPUB generator for the Google SRE ebook. Started in February in Python. Gave up and redid it properly in September.
CodeChef Offline: March 2012 I attempted to make a offline repository for CodeChef problems. I spent some time in May 2017 upgrading the project with a cleaner scraper and a Jekyll base.
Hoshruba: June 2015 I wrote a script that scraped Tor’s serialized publication of the first book in Hoshruba series to generate EPUB and MOBI files. I would recommend the book if you are interested in reading what many would term the “original fantasy book”
HackerTray: December 2013 - I wrote a Linux PyGTK application that sits in your taskbar using Indicator Applet to show you the latest stories from Hacker News. Looking for a maintainer.
MagicMuggle: May 2017 I wrote a script to convert Magic Muggle (A Harry Potter fanfic about a muggle who accidentally gets into Hogwarts) books from their original reddit posts to EPUB and MOBI files.
Kerala IT Policy: March 2017 Attempted to transcribe the draft IT policies put up by the Government of Kerala. Lots of OCR followed by manual fixes. I stopped working on this when I realized that the government had actually put up a really nice website for this (with clear plaintext, not the bad PDF I was using as the source).
lightsaber: August 2015 I created a DNS based HTTP-3xx redirect service. Useful if you own a domain and you want it to be redirected, but don’t have a webserver with you. Made as part of the Django Hackathon organized by HackerEarth in Ruby.
HackerCouch: November 2015 My hack during hackbeach 2015. Created something best described as “couchsurfing for hackers”. Simple Jekyll/Ruby website hosted on GitHub Pages.

Older Newer