it – GypsyNet-W

The Home Lab

Learning professional skills at home with unprofessional gains

After years of toying with the ideas of how I want to do homelab things at home, I finally had a scope of what I wanted to do and how I wanted to implement it. Professionally I am 90% a Microsoft / Windows guy. I dabble in .NET and administrate a mostly Windows environment. I speak all the terms pretty well but there’s this other 5% that I know dangerously little about, and I feel like it’s something I need to stop making fun of and just get to it.

The Scope

I’ve maintained a media collection for quite some time, cause it’s legal to own a digital copy of physical media (That’s my story and I’m sticking to it.) and being able to stream stuff locally eliminates issues with connection speed. I want to make the platform around that media work better and be smarter. Ideally the frontend service will need to be universal, it must work everywhere, on the TV in the living room, on mobile devices, inside a web-browser, etc. It also must be quick. If it’s a hassle to get to the media, I’ll be less likely to use it. If I’m less likely to use it, my home users will be even less likely to want to put time into it. Lastly the interface points need to be easy, I’m here for the complicated setup so that further interfacing with it is easy.

That seems simple enough I have a couple options. I’ve looked in to Plex, and well it just put a sour taste in my mouth from the start. Then I saw some things around Emby but hadn’t really given it a fair shake, so last on the list was Jellyfin. The interface is pretty quick, and it incorporates quite a bit of metadata that doesn’t really take much effort to get in to place, so the whole platform has a polish about it that makes it really attractive.

Now, I’m not going to settle for just serving the media. I, of course, need to incorporate a better way to bringing media into the library (let’s face it, I’m going to want more media…). So onward to find ways to automate fetching media. These are all well known tools and since I don’t really have to interface with them it’ll be simple enough to set them and forget them. On the opposite end of the stack is JellySeerr, a search engine of sorts to hand terms to the well-known tools to make the magic happen and to integrate with Jellyfin.

Okay application stack out of the way, how do I serve it? Jumping on the whale late here, I’ll use Docker, of course. Since my professional life has me constantly looking at processes and identifying the failure points, I saw an immediate need for more than one host, cause the best part about redundancy is redundancy.

Seems like I have a rough shopping list, time to get into it.

The Hardware

Now, I’ve operated some form of storage on my home network for quite some time, I’ve got redundancy built in to the arrays so disk failure isn’t going to stop me. But I only have one NAS… this may change in the future but for now it’s just going to be a weak link in the chain. Media takes up a lot of storage and replicating that across multiple NASs would be prohibitively expensive at this juncture.

The docker hosts; I have some criteria, since these will live in the rack behind my computer chair, they had to be quiet, and preferably small. I was holding out to capitalize on surplus hardware when my office did a hardware refresh but that didn’t happen. I had seen something around using Micro Form Factor desktop PCs to perform these tasks, and I knew this was going to be my solution. Off to Ebay to find a few identical machines for my “server”. I found a really good deal on some Dell Optiplex 3040 MFFs, they had a decent but not fantastic CPU and half the RAM that the board could handle and as expected no Hard Drive. I got 4. The RAM and SSDs came from amazon, 4x1TB SSDs cause they’re dumb cheap this time of year, and enough RAM to max out all the boxes. The irritation came when I had to order the external power supplies, not something I was used to in the desktop space, but it makes sense when you realize that MFF PCs are really just laptops with better cooling and a bit more room for I/O. The boxes came with WiFi capability which got my gears turning but ultimately, I opted to just remove it. SSDs and max out the RAM and I was ready for OS.

The Software Part 1

Something I hadn’t really considered up until this point was what OS did I want for the Docker hosts. I had previously used Debian, and I had some recommendations for RHEL, but I opted to go with Ubuntu Server for the most universal Linux experience, I may catch some flak for it, but at the end of the day I have a working product. Got the OS installed on all the machines and made sure that I did everything identically on each one.** Once each machine was ready for production I was off to take on my next consideration; which flavor of containerization/orchestration do I want to use? The two products I considered were Kubernetes and Docker Swarm. After not much research I figured that Kubernetes was going to be too much for what I wanted to do so Docker Swarm it is! Setup was dead simple I had the cluster up and running in no time. Adding in Portainer for management and I had a pretty solid foundation to work with.

Without getting fully into the gritty details I have the following containers to stand up; Jellyfin, JellySeerr, Radarr, Sonarr, Transmission, Jackett (prowlarr is probably going to replace this). I want them to be able to run on any host at any time, so I had to come up with a way to make the storage the same on each host. After learning how mounts work in Linux, I opened up an NFS share on my Synology and mounted that on each host. Now regardless of where the container is running it can see it’s configuration data and do it’s job.

Some quick configuration to get all the services talking to each other and it was time to test. I picked a movie I hadn’t seen and requested it through JellySeerr, which in turn handed the search over to Radarr, which grabbed some information from jackett and sent the request to transmission, great right? Once completed Radarr moved the file into my media library and we were ready to rock.

Trouble in paradise

This story isn’t without a little peril though. I found that Radarr and Sonarr weren’t fully doing what they were supposed to and after a couple of restarts I found that they were really struggling to work with their internal database (which really really really dislikes being shared over NFS) so it was time to figure out how to have persistent storage available across the cluster. First suggestion was to use Ceph, but that really requires having an available disk to dedicate to the application and these MFF PCs really don’t have that. Next was to figure out GlusterFS which looks like it can slot in on top of an existing file system, you know like that 1TB SSD I have in each of the hosts? Another really simple setup and solution that only required a little bit of work to get functional. Though adding in some options to wait for Gluster to start up before mounting the storage didn’t really occur to me, I added it in so should give me persistent replicated storage available to all 4 hosts.

The Software Part 2: The Search for Curly’s Certificate

Now that I have all the services working, it was time to condense them in to a stack, for easier deployment and replication should my entire cluster fail. Exposing Jellyfin to the outside world I realized a couple of issues. First; Port mapping from my router to the cluster doesn’t really allow me to do proper load balancing or failover to point at different hosts. Second; I should figure out a reverse proxy to hide all the services behind, so everything is secured encrypted and happy.

Not knowing how to approach the first one I started to try to fix the second. I had trouble getting Traefik configured right to properly handle inbound requests and forward them to the appropriate service, so I opted to use the gui for nginx built in to the Synology for the time being.

After doing some more research on the subject I found that there is a linux application KeepAliveD that will enable all the nodes in the cluster to use the same virtual IP and through VRRP determine which host answers on the address, and allowing any node in the cluster to fail without taking all the services with it.

The End of Our Story

So, now I have this semi-complex setup to acquire new media and to consume said media in a friendly way, that flexes some principles that I use in my day job. In the near future, I want to utilize some internal process monitoring and external monitoring so I know if the services go down. I’d also like to take a better look at Traefik to move the reverse proxy into the docker environment. Not too shabby for someone who doesn’t know Linux.

** If only there was a tool for this, that could make sure that my environment stays homogenous and all commands are executed the same way every time on multiple machines…

OMGWTFBOOOM

Now, I know what you’re thinking, ‘What the hell kind of title is that?’. Well, I have been busy lately with kids and work and my usual environment for writing has changed for the better but is no longer as conducive to me getting any writing done. So, now I am picking a different time-slot to put together something for reference when I come back to change things.

To properly set the stage I need to give some background. I am part of a local group of IT folk who tend to do things together, on our Slack channel someone posted that they have a set of old home theater speakers free for whomever wants them. I had designs to build good sound for my living room television, along with a more secure cabinet and whatnot. I got the speakers, they were some high-end Polk Audio with solid enclosures, after removing the protective screens all the drivers appeared intact. I was excited. Then I started shopping for feature rich receivers and the price made me shy away, never mind the amount of work it would take to build a cabinet to fit in the space and I would have no way to protect the drivers from the errant Hot Wheels car thrown by a child not getting their way, so the speakers sat in my office taking up space.

Fast forward to a few weeks ago I was going down the rabbit hole on YouTube (Google’s term) and I re-discovered people making their own Bluetooth powered speakers. While watching one of the videos I happened to look over at the 6 speaker cabinets that I had just taking up space in the corner of my office. Longer story short I had done some cleaning, my workbench was available, and the tinkering could begin.

First some research on what I had and what it was capable of; 1 CS1000p Powered Center Channel, 2 FX500I Surround Speakers, 2 RT800I Floorstanding Speakers, 1 PSW450 Powered Subwoofer. Right off the bat I had keyed in on some terms that piqued my interest, specifically ‘powered center channel’. I hefted this beast on to the work bench and now it was time for discovery. On the back of the CS1000p, there were 2 screw down inputs one for mid/high frequency and one for low, an RCA jack for subwoofer, a switch for high level input on and off, and a traditional power input. On the front I could see a total of 5 drivers, what appeared to be 2 woofers, 2 drivers for mids and a single tweeter in the center, there’s also a volume knob and a green LED. I did not see a power switch of any kind, so I assumed that this was an always on sort of thing.

Next step was to test functionality, I broke apart a headphone cable to get leads that I could connect to the inputs on the back, I powered the speaker up, connected it to my tablet, and nothing. The speaker powered (evident by the green light on the front) but nothing was coming through the drivers. Well, this may end up being a little more difficult. Time to break out the trusty screwdriver and let’s see what’s inside.

The first thing I noticed was that there were a bunch of screws on the metal back plate. I pulled what appeared to be the screws holding the plate in place and was able to remove it from the enclosure pulling it away and disconnecting the cables attaching the guts to the drivers. I was able to mostly identify what circuitry was at work here. I had 2 channels but unsure which speakers they were attached to and some thinner gauge wires I assumed for the volume knob and the LED. Still attached to the plate were a power conditioner, what appeared to be an amplification board, and a power converter from AC to DC power. Upon closer inspection the plastic housing around the Subwoofer input and the Switch for High Level input looked like they either had some weird epoxy around them, or they had melted. I had a hope for epoxy but turns out they were melted.

Now is the point where I must interject that I do not have very much electrical knowledge, but I usually have an approximate knowledge of what I am looking at. I was able to identify the amplification chips and put together a rough idea on the path of audio signal. I wanted to try one more time to power the components and push audio through, but the results were the same as before. The parts on the back plate were going to have to be replaced. Okay, let’s look at the drivers. A couple hundred screws later I was able to get part numbers on the individual drivers and get what their power rating was. I also found that there was a crossover board behind the center 3 drivers, which handled throwing the highs to the tweeter. Some shopping and YouTube lead me to a replacement amplifier and Bluetooth module. The video made everything look simple.

Being one to like having multiple opinions, and not wanting to throw a bunch of cash at a learning project I looked around to see what others were doing. I then stumbled on a video where someone had found an amplifier board with Bluetooth built in (Hey, less wires!) and it was cheaper than the separate boards, I also found that it had a bunch of extra headers for expansion. Almost like this board was built for this specific purpose.

The board in line for purchase is manufactured by Dayton Audio, it is a KAB-250v3 it delivers 50 watts to left and right channels (remember I have 2 channels inside the speaker) it’s a Class D amplifier board, with Bluetooth 4.0. In the box it has the board and wiring harnesses for power, speakers, a Bluetooth pairing control button and an external Bluetooth antenna. Also, in the box was a quick-start guide, which was just detailing out which headers are for what feature.

Right about here is where the anxiety set in. In the past I have caused some irreparable damage to 12v electronics by being careless, so I wanted to be extra cautious, I had monetary investment into this project at this point. Before moving forward, I wanted to make sure that the speakers and my amplifier would be compatible. While on an errand with my wife we were close to a music store that sold pro audio equipment, I gave it a shot, talked with their pro-audio guy. He gave a bit of a shrug and said, “I think you’re just going to have to be brave.” He did clue me in to make sure that the impedance of the drivers matched up and to start low before going bigger.

Okay, moment of truth. Time to test the Amplifier board with a single driver. I pulled the driver out of the enclosure and put together a bunch of temporary connections, time to power everything up. The outlets on my workbench have individual switches to turn them on and off makes it nice for instances like this where I can flip a switch faster than I can frantically unplug something. I checked all of my connections to make sure I had continuity in all the places I should and then flipped the switch. I don’t know what I expected, maybe some sort of startup noise, a flourish of non-existent LEDs, something. All I got was a single green LED on the amplifier board and a little EM feedback or coil whine (I assume) on the driver. I pulled out my phone and set off to discover Bluetooth devices and there it was “DAKAB”. Initial pairing was a little shaky, but I really think I just didn’t have a handle on what the controller button actually did. I got paired and time to fire up Google Music and hit play on a playlist and I immediately find that my Bluetooth volume was set to max. But it worked!!!

I did a little happy dance, my wife laughed with/at me, and now it was time to test the rest of the enclosure. I broke my temporary connections, reinstalled the driver, connected to the channel and re-ran the test, another success! One more test, this time powering the whole enclosure and one more success! Now it was time to plan out how I wanted to tuck this all back in to the enclosure to make this fully function.

I opted to continue to use the AC to DC 12V 1.5A converter that I had on hand, just connecting it to the AC conditioner that was already present. I used Butt connecters for all the connections cause a more permanent solution would be further down the evolutionary path of this project. About an hour of testing connections and wire management I was able to fully re-assemble the enclosure and a final test to mark the completion of version 1.0 of what I have titled OMGWTFBOOOM.

What I have right now is a surprisingly capable powered Bluetooth speaker. My plans for the future of the project is to re-surface the outside of the enclosure to introduce a little bit of weather resistance, replace the soft cover on the front with a metal grate of some form, and add wheels cause this thing is understandably a bit on the heavy side. Additionally, I would like to put together a better method for delivering power and including a battery for some true portability. Some more upgrades down the road would be an auxiliary input and some USB power for extra functionality. Last, I would like to add a paint-job to complete the cosmetics. But as with all projects, this will be a full evolutionary process.

Public Image

I get asked on the regular why I’m not on Facebook, Twitter, Instagram, or any other social media platform. I usually just respond with some short response that terminates that rabbit hole of an explanation right then and there. What am I saving these people that ask from? This, this shit right here;

Reason number 1 why I don’t do social media; Public image. Yeah, it sounds silly, but I tend to do dumb things. I really don’t need these things showing up at inopportune times. I especially don’t want to have to explain to my son why there is a traffic cone on my head in that picture, and that’s on the tame side of things. I am 2 very different people on the internet, there is my troll gamer self that will set his Steam display name to something like Rectal Bleeding for the lulz, and there is my professional self, answering questions on IT forums and posting reviews etc. It is incredibly rare that these two operate under the same name, or in the same place. Here is an exception, since I will probably never reveal my name here.

Reason number 2 why I don’t do social media; I want more fulfilling relationships with people. I don’t want to know about the amazing meal friend 1 had last week. When I see them next I want them to tell me about it. I want to be able to go to a social gathering and I hear from them what they have been up to since I saw them last. People tend to interject at this point that they keep in touch with family that aren’t local to them through the platform, the counter-point is to ask if their phone can still make phone calls.

Reason number 3 why I don’t do social media; No more stress. It’s downright silly how much effort people put in to keeping up with everything that is going on in the world. I don’t think I put enough emphasis on this; It’s downright silly how much effort people put in to keeping up with everything going on in the world. That’ll have to do. I’ve had to point out to people that the internet will still be there tomorrow, or in a couple hours. You don’t get anything other than a hollow accomplishment for being up-to-date on your feed. In my opinion it’s better to concentrate your attention on things that will be more rewarding in the long run.

Reason number 4 why I don’t do social media; Tin foil hats! Yeah, I’m one of those people that believes that most social media platforms have roots in surveillance. Could be government, could be corporations, could be that weird guy down the street that doesn’t go out during the day. Either way, I don’t want to broadcast my life to them in an easy to consume manner.

Number 5; If I am the product, I am going to chose which store I am in. “If a service is free, you are the product” Over the majority of the past decade profitability through ad-revenue has gone from supplemental income to a legitimate business model. As a firm believer in voting with my wallet or controlling where money goes, I tend to not support companies that I just plain don’t like. This is why I tend to not go to WalMart. Most social media platforms are free to the user because they get their revenue from advertisements, or someone that wants the data (see reason #4). So, since I am going to be the product, I am going to chose where I am featured.

Personally I think that social media is responsible for a substantial amount of mental health issues in the world. The culture is that people must read all their notifications to make sure they are up to date with all their friends and make sure they know everything that’s going on with their Aunt Tina in Kentucky even though all she posts is pictures of her McDonald’s Lunch daily.

Of course, I should address the elephant in the room here; I’m posting on a blog, which is in essence a form of social media. Yep. There’s a counter-point to everything I listed above. Pretty much it all boils down to; While it may look like there is a target to my writing, I am doing this more for myself than anything else.

Insights on Security

As anyone in the IT world can tell you, security is by far the biggest concern in any organization. With the prevalence of Crypto-Locker RansomWare variants, on top of the already innumerable amount of malicious software in the wild, it’s enough to keep anyone up at night. In less than a decade IT had to shift its mindset from the traditional “I’ve got anti-virus, and it updates daily” to a more aggressive stance, parsing logs and network activity, and behavioral tracking. The problem has become that traditional Anti-Virus isn’t enough anymore.

I remember fondly being the go-to for what Anti-Virus / Anti-Malware / General Internet Condom to use that is the “best” where I would take in to account performance weight against effectiveness, and choosing the software package that hits the best ratio of performance to functionality to Price.

Now, since the internet is so ubiquitous and there are developers everywhere that view it as their sole purpose to exploit vulnerabilities in computer systems. Regardless if their intention is to promote security or something more nefarious, it is nearly impossible to stay ahead of the game, even in my small environment.

Anatomy of an attack

To better understand how to best secure your environment you first need to understand how an attack happens. Knowing the fundamentals of an attack lay the groundwork that will put you in the best position possible.

Reconnaissance
- Just as we’re doing here, an attacker is going to gather all the possible information about their target that they can.
- A determined attacker will utilize any means necessary to gather information on the target, including going through waste.
- Other methods include network scanning, determining open ports and services on your network and what the responses are on those ports
- An attacker may also do some online research about the target, looking for job postings, or even mirror the target’s web-site to get information about the structure of the network.
Assessment
- With reconnaissance done, it’s time to assess the information.
- Comparing responses from the network scan against published vulnerabilities and determining what the most effective mode of entry is going to be.
- Determine the direction of the attack, go to the target, or coerce the target to come to the attacker.
Exploit
- The attacker has gathered what the attack surface is, determined the vulnerability, and how it is going to be used.
- During the exploit different payloads may yield different results, and the prime directive is to gain “persistence” or permanent access. It is possible that to avoid detection after access is gained an attack may lay dormant for some time.
- Once access is gained an attack will typically “pivot” to some other local resource that is more valuable (think Database, File Server, User Data)

Mitigation

Prevention is an idea of the past, so get that out of your mind. Mitigation is attainable. Knowing how an attack plays out makes it easier to take the steps necessary to secure your environment. The saving grace of most attacks is that they are done with Free or Open Source tools, meaning you can perform all the steps of an attack yourself. Particularly important are the Reconnaissance and Assessment steps (you can also exploit as well and then re-perform Reconnaissance and Assessment from within the network if you’ve gained access.)

Once you’ve gathered appropriate data, you now have something to work with. This will give you a view on where your vulnerabilities are and how to mitigate them, be it through software patching or updating firewall settings etc.

So, you’ve secured your network, you’re good now right? Nope. All securing your network does is mitigate attacks originating outside the network and attempting to gain access. There’s still attacks that get the target to come to the attacker. These attacks are the hard one to protect against because they can come from almost anywhere, and all they usually require is user interaction.

Some methods to mitigate reverse attacks.

Instate application white-listing
- Rather than pick what applications to block pick which applications are able to run
block executable content wherever possible (temp directories, user profiles, etc.)
Train the human element to be mindful
- Far and away this is the toughest hurdle. Users will always be the weakest link in IT Security.
- Training should include:
  - Email best practices
  - Removable media (USB keys, CDs, DVDs etc..) should not be trusted
  - Authorized vs Unauthorized personnel
  - What information is pertinent to the conversation and what isn’t

While security is an ever evolving topology an IT professional should be pragmatic and vigilant in efforts to not become the victim.