Home Server May 2017
Tuesday, May 02, 2017
It has been no secret that I've been working on a few personal projects of my own. My friends know how
Since I've been repeating myself a lot lately and it's quite hard to explain quickly without the aid of diagrams, I created some diagrams.
Firstly, I'm using an open source solution called Docker for hosting my services and it's amazing and I urge you devs out there to check it out. In simple terms, it's a way of virtualizing services rather than the OS itself. The benefits are that there is less overhead, starting and stopping services are just a few simple commands away and you can duplicate as many services as you want across a cluster(swarm).
Right now I have a few services that are always turned on.
- Automated torrent downloading
- Google Drive-like service for files
- Internal DNS service
- Service manager for docker containers
- VPN service
I also have some development sites that runs on and off based on changing needs (eg. web, database service).
Secondly, I've been reading up quite a lot about reverse proxy and I've been wanting to set them up for months/years(?!) but just haven't had the time to get to them. In simple terms, a reverse proxy is kind of like a seamless way of redirecting users to the correct server based on the URL. It could be used for load balancing, or in my case, used as a transparent way for end users to access the services*.
* | why not DNS? Because DNS only resolves IP but cannot resolve port numbers. My services are only exposed by port and are (mostly) on a single machine so I had to do it this way.
However, I still have a DNS service running because I need to be able to resolve URLs to a physical machine. (no messing with host files anymore)
An overview of how it works is shown in the diagram below.
--- update ---
Made a mistake in the diagram, when accessing data.service it goes directly to the DNS service then to the NAS. Does not require nginx to resolve at all.
Made a mistake in the diagram, when accessing data.service it goes directly to the DNS service then to the NAS. Does not require nginx to resolve at all.
When the reverse proxy server (aka. nginx) is hit with a URL query, eg. (http://torrent.service), it looks in the configuration where to redirect the user to seamlessly in the background. The seamlessness/transparency is a very powerful tool. I will explain it in more detail after illustrating how the general flow of logic of this setup.
* | client machine must be configured to use my DNS server (forced over VPN)
When user visits any site ending with .service or .dev, the DNS service will return the address of the services. Because most of the services are hosted on one physical machine, the nginx (reverse proxy) service resolves the incoming request to the corresponding service/dev.
* | if user visits another site eg. google.com, it will be resolved normally to the internet (albeit with a very slight latency overhead)
Going back to why nginx (reverse proxy) is a powerful tool, it's really the transparency that makes it awesome. When the user visits http://torrent.service, the DNS resolves it to the machine (eg. 172.16.1.66:80*). The nginx service listening on port 80 takes that query and forward it to the service (eg. 172.16.1.50:12345). However, to the end user, the URL never changes, the visible IP never changes, it all works seamlessly in the background.
* | IP and port number separated by semi-colon
eg. You navigate to the "downloads" page. The visible URL changes to http://torrent.service/downloads, it looks like it's 172.16.1.66:80/downloads to the system, but it's actually 172.16.1.50:12345/downloads. It all takes place automagically.
It's great when everything comes together but I definitely had my fair share of problems when setting it up and have to combine multiple sources of answers online before I could get my services running the way that I want them to. There is still 2 amazing projects that I'm looking to tackle before school starts proper, my fingers are crossed.
I'll like to clarify that when I say service, it may be made up of one or more services. For example http://torrent.service is made up of Transmission (the downloader service), and Flexget (the background service that searches for latest episode).
To end it off, everything is hosted either on my secondhand Thinkpad X200 (released in 2008), or on a Raspberry Pi 3. So if there's anyone who's interested in sponsoring me for anything (I can even make a video about it), contact me.
Labels: featured, geek, lordofgeeks, server
