Category: technology

The Home Lab

Learning professional skills at home with unprofessional gains

After years of toying with the ideas of how I want to do homelab things at home, I finally had a scope of what I wanted to do and how I wanted to implement it. Professionally I am 90% a Microsoft / Windows guy. I dabble in .NET and administrate a mostly Windows environment. I speak all the terms pretty well but there’s this other 5% that I know dangerously little about, and I feel like it’s something I need to stop making fun of and just get to it.

The Scope

I’ve maintained a media collection for quite some time, cause it’s legal to own a digital copy of physical media (That’s my story and I’m sticking to it.) and being able to stream stuff locally eliminates issues with connection speed. I want to make the platform around that media work better and be smarter. Ideally the frontend service will need to be universal, it must work everywhere, on the TV in the living room, on mobile devices, inside a web-browser, etc. It also must be quick. If it’s a hassle to get to the media, I’ll be less likely to use it. If I’m less likely to use it, my home users will be even less likely to want to put time into it. Lastly the interface points need to be easy, I’m here for the complicated setup so that further interfacing with it is easy.


That seems simple enough I have a couple options. I’ve looked in to Plex, and well it just put a sour taste in my mouth from the start. Then I saw some things around Emby but hadn’t really given it a fair shake, so last on the list was Jellyfin. The interface is pretty quick, and it incorporates quite a bit of metadata that doesn’t really take much effort to get in to place, so the whole platform has a polish about it that makes it really attractive.


Now, I’m not going to settle for just serving the media. I, of course, need to incorporate a better way to bringing media into the library (let’s face it, I’m going to want more media…). So onward to find ways to automate fetching media. These are all well known tools and since I don’t really have to interface with them it’ll be simple enough to set them and forget them. On the opposite end of the stack is JellySeerr, a search engine of sorts to hand terms to the well-known tools to make the magic happen and to integrate with Jellyfin.

Okay application stack out of the way, how do I serve it? Jumping on the whale late here, I’ll use Docker, of course. Since my professional life has me constantly looking at processes and identifying the failure points, I saw an immediate need for more than one host, cause the best part about redundancy is redundancy.

Seems like I have a rough shopping list, time to get into it.

The Hardware

Now, I’ve operated some form of storage on my home network for quite some time, I’ve got redundancy built in to the arrays so disk failure isn’t going to stop me. But I only have one NAS… this may change in the future but for now it’s just going to be a weak link in the chain. Media takes up a lot of storage and replicating that across multiple NASs would be prohibitively expensive at this juncture.

The docker hosts; I have some criteria, since these will live in the rack behind my computer chair, they had to be quiet, and preferably small. I was holding out to capitalize on surplus hardware when my office did a hardware refresh but that didn’t happen. I had seen something around using Micro Form Factor desktop PCs to perform these tasks, and I knew this was going to be my solution. Off to Ebay to find a few identical machines for my “server”. I found a really good deal on some Dell Optiplex 3040 MFFs, they had a decent but not fantastic CPU and half the RAM that the board could handle and as expected no Hard Drive. I got 4. The RAM and SSDs came from amazon, 4x1TB SSDs cause they’re dumb cheap this time of year, and enough RAM to max out all the boxes. The irritation came when I had to order the external power supplies, not something I was used to in the desktop space, but it makes sense when you realize that MFF PCs are really just laptops with better cooling and a bit more room for I/O. The boxes came with WiFi capability which got my gears turning but ultimately, I opted to just remove it. SSDs and max out the RAM and I was ready for OS.

The Software Part 1

Something I hadn’t really considered up until this point was what OS did I want for the Docker hosts. I had previously used Debian, and I had some recommendations for RHEL, but I opted to go with Ubuntu Server for the most universal Linux experience, I may catch some flak for it, but at the end of the day I have a working product. Got the OS installed on all the machines and made sure that I did everything identically on each one.** Once each machine was ready for production I was off to take on my next consideration; which flavor of containerization/orchestration do I want to use? The two products I considered were Kubernetes and Docker Swarm. After not much research I figured that Kubernetes was going to be too much for what I wanted to do so Docker Swarm it is! Setup was dead simple I had the cluster up and running in no time. Adding in Portainer for management and I had a pretty solid foundation to work with.

Without getting fully into the gritty details I have the following containers to stand up; Jellyfin, JellySeerr, Radarr, Sonarr, Transmission, Jackett (prowlarr is probably going to replace this). I want them to be able to run on any host at any time, so I had to come up with a way to make the storage the same on each host. After learning how mounts work in Linux, I opened up an NFS share on my Synology and mounted that on each host. Now regardless of where the container is running it can see it’s configuration data and do it’s job.

Some quick configuration to get all the services talking to each other and it was time to test. I picked a movie I hadn’t seen and requested it through JellySeerr, which in turn handed the search over to Radarr, which grabbed some information from jackett and sent the request to transmission, great right? Once completed Radarr moved the file into my media library and we were ready to rock.

Trouble in paradise

This story isn’t without a little peril though.  I found that Radarr and Sonarr weren’t fully doing what they were supposed to and after a couple of restarts I found that they were really struggling to work with their internal database (which really really really dislikes being shared over NFS) so it was time to figure out how to have persistent storage available across the cluster.  First suggestion was to use Ceph, but that really requires having an available disk to dedicate to the application and these MFF PCs really don’t have that.  Next was to figure out GlusterFS which looks like it can slot in on top of an existing file system, you know like that 1TB SSD I have in each of the hosts?  Another really simple setup and solution that only required a little bit of work to get functional.  Though adding in some options to wait for Gluster to start up before mounting the storage didn’t really occur to me, I added it in so should give me persistent replicated storage available to all 4 hosts.

The Software Part 2: The Search for Curly’s Certificate

Now that I have all the services working, it was time to condense them in to a stack, for easier deployment and replication should my entire cluster fail. Exposing Jellyfin to the outside world I realized a couple of issues. First; Port mapping from my router to the cluster doesn’t really allow me to do proper load balancing or failover to point at different hosts. Second; I should figure out a reverse proxy to hide all the services behind, so everything is secured encrypted and happy.

Not knowing how to approach the first one I started to try to fix the second. I had trouble getting Traefik configured right to properly handle inbound requests and forward them to the appropriate service, so I opted to use the gui for nginx built in to the Synology for the time being.

After doing some more research on the subject I found that there is a linux application KeepAliveD that will enable all the nodes in the cluster to use the same virtual IP and through VRRP determine which host answers on the address, and allowing any node in the cluster to fail without taking all the services with it.

The End of Our Story

So, now I have this semi-complex setup to acquire new media and to consume said media in a friendly way, that flexes some principles that I use in my day job. In the near future, I want to utilize some internal process monitoring and external monitoring so I know if the services go down. I’d also like to take a better look at Traefik to move the reverse proxy into the docker environment. Not too shabby for someone who doesn’t know Linux.

** If only there was a tool for this, that could make sure that my environment stays homogenous and all commands are executed the same way every time on multiple machines…

OMGWTFBOOOM

Now, I know what you’re thinking, ‘What the hell kind of title is that?’.   Well, I have been busy lately with kids and work and my usual environment for writing has changed for the better but is no longer as conducive to me getting any writing done.  So, now I am picking a different time-slot to put together something for reference when I come back to change things.

To properly set the stage I need to give some background.  I am part of a local group of IT folk who tend to do things together, on our Slack channel someone posted that they have a set of old home theater speakers free for whomever wants them.  I had designs to build good sound for my living room television, along with a more secure cabinet and whatnot.  I got the speakers, they were some high-end Polk Audio with solid enclosures, after removing the protective screens all the drivers appeared intact. I was excited.  Then I started shopping for feature rich receivers and the price made me shy away, never mind the amount of work it would take to build a cabinet to fit in the space and I would have no way to protect the drivers from the errant Hot Wheels car thrown by a child not getting their way, so the speakers sat in my office taking up space.

Fast forward to a few weeks ago I was going down the rabbit hole on YouTube (Google’s term) and I re-discovered people making their own Bluetooth powered speakers.  While watching one of the videos I happened to look over at the 6 speaker cabinets that I had just taking up space in the corner of my office.  Longer story short I had done some cleaning, my workbench was available, and the tinkering could begin.

First some research on what I had and what it was capable of; 1 CS1000p Powered Center Channel, 2 FX500I Surround Speakers, 2 RT800I Floorstanding Speakers, 1 PSW450 Powered Subwoofer.  Right off the bat I had keyed in on some terms that piqued my interest, specifically ‘powered center channel’.  I hefted this beast on to the work bench and now it was time for discovery.  On the back of the CS1000p, there were 2 screw down inputs one for mid/high frequency and one for low, an RCA jack for subwoofer, a switch for high level input on and off, and a traditional power input.  On the front I could see a total of 5 drivers, what appeared to be 2 woofers, 2 drivers for mids and a single tweeter in the center, there’s also a volume knob and a green LED.  I did not see a power switch of any kind, so I assumed that this was an always on sort of thing.

Next step was to test functionality, I broke apart a headphone cable to get leads that I could connect to the inputs on the back, I powered the speaker up, connected it to my tablet, and nothing.  The speaker powered (evident by the green light on the front) but nothing was coming through the drivers.  Well, this may end up being a little more difficult.  Time to break out the trusty screwdriver and let’s see what’s inside.

The first thing I noticed was that there were a bunch of screws on the metal back plate.  I pulled what appeared to be the screws holding the plate in place and was able to remove it from the enclosure pulling it away and disconnecting the cables attaching the guts to the drivers.  I was able to mostly identify what circuitry was at work here.  I had 2 channels but unsure which speakers they were attached to and some thinner gauge wires I assumed for the volume knob and the LED.  Still attached to the plate were a power conditioner, what appeared to be an amplification board, and a power converter from AC to DC power.  Upon closer inspection the plastic housing around the Subwoofer input and the Switch for High Level input looked like they either had some weird epoxy around them, or they had melted.  I had a hope for epoxy but turns out they were melted.

Now is the point where I must interject that I do not have very much electrical knowledge, but I usually have an approximate knowledge of what I am looking at.  I was able to identify the amplification chips and put together a rough idea on the path of audio signal.  I wanted to try one more time to power the components and push audio through, but the results were the same as before.  The parts on the back plate were going to have to be replaced.  Okay, let’s look at the drivers.  A couple hundred screws later I was able to get part numbers on the individual drivers and get what their power rating was.  I also found that there was a crossover board behind the center 3 drivers, which handled throwing the highs to the tweeter.  Some shopping and YouTube lead me to a replacement amplifier and Bluetooth module.  The video made everything look simple.

Being one to like having multiple opinions, and not wanting to throw a bunch of cash at a learning project I looked around to see what others were doing.  I then stumbled on a video where someone had found an amplifier board with Bluetooth built in (Hey, less wires!) and it was cheaper than the separate boards, I also found that it had a bunch of extra headers for expansion.  Almost like this board was built for this specific purpose.

The board in line for purchase is manufactured by Dayton Audio, it is a KAB-250v3 it delivers 50 watts to left and right channels (remember I have 2 channels inside the speaker) it’s a Class D amplifier board, with Bluetooth 4.0.  In the box it has the board and wiring harnesses for power, speakers, a Bluetooth pairing control button and an external Bluetooth antenna.  Also, in the box was a quick-start guide, which was just detailing out which headers are for what feature.

Right about here is where the anxiety set in.  In the past I have caused some irreparable damage to 12v electronics by being careless, so I wanted to be extra cautious, I had monetary investment into this project at this point.  Before moving forward, I wanted to make sure that the speakers and my amplifier would be compatible.  While on an errand with my wife we were close to a music store that sold pro audio equipment, I gave it a shot, talked with their pro-audio guy.  He gave a bit of a shrug and said, “I think you’re just going to have to be brave.”  He did clue me in to make sure that the impedance of the drivers matched up and to start low before going bigger.

Okay, moment of truth.  Time to test the Amplifier board with a single driver.  I pulled the driver out of the enclosure and put together a bunch of temporary connections, time to power everything up.  The outlets on my workbench have individual switches to turn them on and off makes it nice for instances like this where I can flip a switch faster than I can frantically unplug something.  I checked all of my connections to make sure I had continuity in all the places I should and then flipped the switch.  I don’t know what I expected, maybe some sort of startup noise, a flourish of non-existent LEDs, something.  All I got was a single green LED on the amplifier board and a little EM feedback or coil whine (I assume) on the driver.  I pulled out my phone and set off to discover Bluetooth devices and there it was “DAKAB”.  Initial pairing was a little shaky, but I really think I just didn’t have a handle on what the controller button actually did.  I got paired and time to fire up Google Music and hit play on a playlist and I immediately find that my Bluetooth volume was set to max.  But it worked!!!

I did a little happy dance, my wife laughed with/at me, and now it was time to test the rest of the enclosure.  I broke my temporary connections, reinstalled the driver, connected to the channel and re-ran the test, another success!  One more test, this time powering the whole enclosure and one more success! Now it was time to plan out how I wanted to tuck this all back in to the enclosure to make this fully function.

I opted to continue to use the AC to DC 12V 1.5A converter that I had on hand, just connecting it to the AC conditioner that was already present.  I used Butt connecters for all the connections cause a more permanent solution would be further down the evolutionary path of this project.  About an hour of testing connections and wire management I was able to fully re-assemble the enclosure and a final test to mark the completion of version 1.0 of what I have titled OMGWTFBOOOM.

What I have right now is a surprisingly capable powered Bluetooth speaker.  My plans for the future of the project is to re-surface the outside of the enclosure to introduce a little bit of weather resistance, replace the soft cover on the front with a metal grate of some form, and add wheels cause this thing is understandably a bit on the heavy side.  Additionally, I would like to put together a better method for delivering power and including a battery for some true portability.  Some more upgrades down the road would be an auxiliary input and some USB power for extra functionality.  Last, I would like to add a paint-job to complete the cosmetics.  But as with all projects, this will be a full evolutionary process.

Know the Content

The landscape of the world today is vastly different from when I was growing up.  Having a viewport to the internet in your pocket has changed the game.  In my adolescent years the internet hadn’t really come of age, and for those that would argue, it’s access wasn’t nearly as ambiguous as it is now.  Technology in general is a requirement today, how to interface, operate, and fundamental concepts will dictate your personal and professional life.

With its prevalence we must do everything we can to make sure that our children have every advantage as they grow, which means introducing them to technology so that they can succeed in the world as we can only imagine it will be when they reach adulthood.  Being an IT professional means that I have an affinity for technology.  I used to say I am the purveyor of all things cool in the world of tech.  I’d like to think that I still am, though there are some boundaries to what I can obtain which leads me to make more careful considerations on what I purchase.

Pulling back to the parenting theme; We as parents have a role in our child’s life that is more impactful that we can fathom.  We are the ones responsible for the introduction of technology to our children, and with it the internet.  As powerful as it can be I think a base level of understanding needs to be in place before ever setting a child loose on the internet.  It can be a great learning opportunity, but it can also be the first step down a slippery slope or even downright dangerous.

Consider this; There are lots of things in the average home to drink.  You’ve got water, milk, juice, alcohol, etc.  Sure, you immediately dismiss alcohol as an option, but do you limit the amount of juice you give to your kid?  Milk?  Water?  What if you give them Juice too often?  It isn’t inherently bad, but have you seen a child deny water in favor of juice?  This is curating the content that your child consumes.  Basic interacting with the internet and technology is similar.  There needs to be a balance in place so that basic skills and learning techniques are still present.

With that analogy in place just how much internet are you willing to just blindly hand over to your child?  Let’s face it here, there’s some applications you can get for mobile that are educational, but they are the beginning of the avalanche.  How long before the application becomes boring?  Now on to Netflix, or YouTube, or any other of content delivery networks?  While trying to understand how I should introduce technology into my son’s life, I had to take a few more steps back to appreciate where my understanding of it came from.

Largely technology and the ambiguity of the internet have lead many to take it for granted.  It has become my realization that more and more people don’t understand where we came from.  What did we do before the smartphone?  What did we do before the internet was always on?  I really think that missing out on physically reading a book, writing things down on paper, or interacting face-to-face with someone is incredibly detrimental to a developing child.

I think it’s tough to appreciate an always on fast internet connection without knowing what it’s like to have to tie up a phone line for it.  It’s difficult to understand the pathways that have opened because we can transfer data across the globe in seconds rather than days.  This is not to say that I will subject my son to a decade of internet at the painfully slow 56k (or worse).  But I want him to know how to talk to people, how to read a book, and how to write.  Additionally, I want him to be able to learn at will and not rely on an internet connection.  If all goes according to plan the light I’ll have to tell him to turn off in the middle of the night will be a flashlight, not a smartphone or tablet.

What I am trying to convey here, is I want to give my son, at the very least, a Cliff’s Notes version of technology history before introducing him to all that the internet and technology has to offer, because I want him to understand how powerful of a tool it can be.

Choose Your Babysitter Wisely

Time to address the elephant in the room; Technology.  When is it applicable to hand over to your child and when should you misdirect with the toy truck? 

A good half of parenting is observation, both in watching the child and what other parents are doing.  Observation being something that I have always been good at.  Then picking apart someone’s actions from a sociological standpoint has always been something that interests me.  So, I tend to see the why of how someone reacts a lot quicker than most.  I think this is where my initial judge of character comes from. 

One of the biggest trends in parenting that I’ve seen today is when in a restaurant handing off the smartphone to keep the child entertained / quiet.  I’ve seen this happen with children ranging in ages from 2-well into their teens.  Admittedly the latter half of that tend to have their own smartphone.  While I can relate that it is a very easy way out, I’ve got to wonder if it’s more harm than help. 

I’ll set the stage for you.  Two parents and their child walk in to a restaurant.  They get seated and the kid is fidgety and inconsolable, one parent frantically pulls out their phone, unlocks it, opens an app and hands it over.  This is a desperate attempt to keep the kid quiet, it’s got an added side effect of the parents can now speak to each other unhindered by the interests of their child. 

How did we get here?  When did the smartphone become the replacement for the pacifier?  Well, it’s not a new trend.  Parents used to be able to turn on the television and plop their children in front of it for time to themselves.  There is an unfortunate downside; the parents have caved once on this, they will cave again and again.

Throughout a child’s developmental life, they begin everything with mimicry.  You never really appreciate just how much of this happens until you see it happen.  Hearing my son trying to repeat something I said while not consciously censoring my language was an eye-opener.  That’s not to say mimicry is only going to show you what you need to do differently, I also remember my son sitting in a high-chair with a spoon tapping the surface of his tray and then bringing the end of the spoon to his face.  My wife and I thought the behavior was odd until we pieced together that he was mimicking us eating.  But I’ve gotten off topic. 

When you realize the parroting is happening you really need to be more conscious of what you do in front of your child.  How often do you check your phone?  How often is it out in the open?  This is where you really need to lay the groundwork for what is acceptable, and bear in mind that if you want your child to follow the rules, you must adhere to them as well.  Otherwise, you will be fighting an uphill battle against a fundamental learning method; mimicry. 

The other side of this is what else will your child mimic?  Take our family in the restaurant from earlier; did they just open YouTube, or some other video service?  Do you think the child is going to mimic what they see there?  They absolutely will.  Maybe not with as much conviction as they parrot their parents, but it will happen.  Additionally, I must ask, does the child understand at this point what is morally and socially acceptable for them and what isn’t?  Does the child comprehend just how powerful the device is that they hold in their hands? 

This post is going to be at least a two-parter, on one hand to keep the length of the posts consistent, and to try and maintain a consistent train of thought.

Insights on Security

As anyone in the IT world can tell you, security is by far the biggest concern in any organization.  With the prevalence of Crypto-Locker RansomWare variants, on top of the already innumerable amount of malicious software in the wild, it’s enough to keep anyone up at night.  In less than a decade IT had to shift its mindset from the traditional “I’ve got anti-virus, and it updates daily” to a more aggressive stance, parsing logs and network activity, and behavioral tracking.  The problem has become that traditional Anti-Virus isn’t enough anymore.

I remember fondly being the go-to for what Anti-Virus / Anti-Malware / General Internet Condom to use that is the “best” where I would take in to account performance weight against effectiveness, and choosing the software package that hits the best ratio of performance to functionality to Price.

Now, since the internet is so ubiquitous and there are developers everywhere that view it as their sole purpose to exploit vulnerabilities in computer systems.  Regardless if their intention is to promote security or something more nefarious, it is nearly impossible to stay ahead of the game, even in my small environment.

Anatomy of an attack

To better understand how to best secure your environment you first need to understand how an attack happens.  Knowing the fundamentals of an attack lay the groundwork that will put you in the best position possible.

  1. Reconnaissance
    • Just as we’re doing here, an attacker is going to gather all the possible information about their target that they can.
    • A determined attacker will utilize any means necessary to gather information on the target, including going through waste.
    • Other methods include network scanning, determining open ports and services on your network and what the responses are on those ports
    • An attacker may also do some online research about the target, looking for job postings, or even mirror the target’s web-site to get information about the structure of the network.
  2. Assessment
    • With reconnaissance done, it’s time to assess the information.
    • Comparing responses from the network scan against published vulnerabilities and determining what the most effective mode of entry is going to be.
    • Determine the direction of the attack, go to the target, or coerce the target to come to the attacker.
  3. Exploit
    • The attacker has gathered what the attack surface is, determined the vulnerability, and how it is going to be used.
    • During the exploit different payloads may yield different results, and the prime directive is to gain “persistence” or permanent access.  It is possible that to avoid detection after access is gained an attack may lay dormant for some time.
    • Once access is gained an attack will typically “pivot” to some other local resource that is more valuable (think Database, File Server, User Data)
Mitigation

Prevention is an idea of the past, so get that out of your mind.  Mitigation is attainable.   Knowing how an attack plays out makes it easier to take the steps necessary to secure your environment.  The saving grace of most attacks is that they are done with Free or Open Source tools, meaning you can perform all the steps of an attack yourself.  Particularly important are the Reconnaissance and Assessment steps (you can also exploit as well and then re-perform Reconnaissance and Assessment from within the network if you’ve gained access.)

Once you’ve gathered appropriate data, you now have something to work with.  This will give you a view on where your vulnerabilities are and how to mitigate them, be it through software patching or updating firewall settings etc.

So, you’ve secured your network, you’re good now right?  Nope.  All securing your network does is mitigate attacks originating outside the network and attempting to gain access.  There’s still attacks that get the target to come to the attacker.  These attacks are the hard one to protect against because they can come from almost anywhere, and all they usually require is user interaction.

Some methods to mitigate reverse attacks.
  • Instate application white-listing
    • Rather than pick what applications to block pick which applications are able to run
  • block executable content wherever possible (temp directories, user profiles, etc.)
  • Train the human element to be mindful
    • Far and away this is the toughest hurdle.  Users will always be the weakest link in IT Security.
    • Training should include:
      • Email best practices
      • Removable media (USB keys, CDs, DVDs etc..) should not be trusted
      • Authorized vs Unauthorized personnel
      • What information is pertinent to the conversation and what isn’t

While security is an ever evolving topology an IT professional should be pragmatic and vigilant in efforts to not become the victim.