The Home Lab

Learning professional skills at home with unprofessional gains

After years of toying with the ideas of how I want to do homelab things at home, I finally had a scope of what I wanted to do and how I wanted to implement it. Professionally I am 90% a Microsoft / Windows guy. I dabble in .NET and administrate a mostly Windows environment. I speak all the terms pretty well but there’s this other 5% that I know dangerously little about, and I feel like it’s something I need to stop making fun of and just get to it.

The Scope

I’ve maintained a media collection for quite some time, cause it’s legal to own a digital copy of physical media (That’s my story and I’m sticking to it.) and being able to stream stuff locally eliminates issues with connection speed. I want to make the platform around that media work better and be smarter. Ideally the frontend service will need to be universal, it must work everywhere, on the TV in the living room, on mobile devices, inside a web-browser, etc. It also must be quick. If it’s a hassle to get to the media, I’ll be less likely to use it. If I’m less likely to use it, my home users will be even less likely to want to put time into it. Lastly the interface points need to be easy, I’m here for the complicated setup so that further interfacing with it is easy.


That seems simple enough I have a couple options. I’ve looked in to Plex, and well it just put a sour taste in my mouth from the start. Then I saw some things around Emby but hadn’t really given it a fair shake, so last on the list was Jellyfin. The interface is pretty quick, and it incorporates quite a bit of metadata that doesn’t really take much effort to get in to place, so the whole platform has a polish about it that makes it really attractive.


Now, I’m not going to settle for just serving the media. I, of course, need to incorporate a better way to bringing media into the library (let’s face it, I’m going to want more media…). So onward to find ways to automate fetching media. These are all well known tools and since I don’t really have to interface with them it’ll be simple enough to set them and forget them. On the opposite end of the stack is JellySeerr, a search engine of sorts to hand terms to the well-known tools to make the magic happen and to integrate with Jellyfin.

Okay application stack out of the way, how do I serve it? Jumping on the whale late here, I’ll use Docker, of course. Since my professional life has me constantly looking at processes and identifying the failure points, I saw an immediate need for more than one host, cause the best part about redundancy is redundancy.

Seems like I have a rough shopping list, time to get into it.

The Hardware

Now, I’ve operated some form of storage on my home network for quite some time, I’ve got redundancy built in to the arrays so disk failure isn’t going to stop me. But I only have one NAS… this may change in the future but for now it’s just going to be a weak link in the chain. Media takes up a lot of storage and replicating that across multiple NASs would be prohibitively expensive at this juncture.

The docker hosts; I have some criteria, since these will live in the rack behind my computer chair, they had to be quiet, and preferably small. I was holding out to capitalize on surplus hardware when my office did a hardware refresh but that didn’t happen. I had seen something around using Micro Form Factor desktop PCs to perform these tasks, and I knew this was going to be my solution. Off to Ebay to find a few identical machines for my “server”. I found a really good deal on some Dell Optiplex 3040 MFFs, they had a decent but not fantastic CPU and half the RAM that the board could handle and as expected no Hard Drive. I got 4. The RAM and SSDs came from amazon, 4x1TB SSDs cause they’re dumb cheap this time of year, and enough RAM to max out all the boxes. The irritation came when I had to order the external power supplies, not something I was used to in the desktop space, but it makes sense when you realize that MFF PCs are really just laptops with better cooling and a bit more room for I/O. The boxes came with WiFi capability which got my gears turning but ultimately, I opted to just remove it. SSDs and max out the RAM and I was ready for OS.

The Software Part 1

Something I hadn’t really considered up until this point was what OS did I want for the Docker hosts. I had previously used Debian, and I had some recommendations for RHEL, but I opted to go with Ubuntu Server for the most universal Linux experience, I may catch some flak for it, but at the end of the day I have a working product. Got the OS installed on all the machines and made sure that I did everything identically on each one.** Once each machine was ready for production I was off to take on my next consideration; which flavor of containerization/orchestration do I want to use? The two products I considered were Kubernetes and Docker Swarm. After not much research I figured that Kubernetes was going to be too much for what I wanted to do so Docker Swarm it is! Setup was dead simple I had the cluster up and running in no time. Adding in Portainer for management and I had a pretty solid foundation to work with.

Without getting fully into the gritty details I have the following containers to stand up; Jellyfin, JellySeerr, Radarr, Sonarr, Transmission, Jackett (prowlarr is probably going to replace this). I want them to be able to run on any host at any time, so I had to come up with a way to make the storage the same on each host. After learning how mounts work in Linux, I opened up an NFS share on my Synology and mounted that on each host. Now regardless of where the container is running it can see it’s configuration data and do it’s job.

Some quick configuration to get all the services talking to each other and it was time to test. I picked a movie I hadn’t seen and requested it through JellySeerr, which in turn handed the search over to Radarr, which grabbed some information from jackett and sent the request to transmission, great right? Once completed Radarr moved the file into my media library and we were ready to rock.

Trouble in paradise

This story isn’t without a little peril though.  I found that Radarr and Sonarr weren’t fully doing what they were supposed to and after a couple of restarts I found that they were really struggling to work with their internal database (which really really really dislikes being shared over NFS) so it was time to figure out how to have persistent storage available across the cluster.  First suggestion was to use Ceph, but that really requires having an available disk to dedicate to the application and these MFF PCs really don’t have that.  Next was to figure out GlusterFS which looks like it can slot in on top of an existing file system, you know like that 1TB SSD I have in each of the hosts?  Another really simple setup and solution that only required a little bit of work to get functional.  Though adding in some options to wait for Gluster to start up before mounting the storage didn’t really occur to me, I added it in so should give me persistent replicated storage available to all 4 hosts.

The Software Part 2: The Search for Curly’s Certificate

Now that I have all the services working, it was time to condense them in to a stack, for easier deployment and replication should my entire cluster fail. Exposing Jellyfin to the outside world I realized a couple of issues. First; Port mapping from my router to the cluster doesn’t really allow me to do proper load balancing or failover to point at different hosts. Second; I should figure out a reverse proxy to hide all the services behind, so everything is secured encrypted and happy.

Not knowing how to approach the first one I started to try to fix the second. I had trouble getting Traefik configured right to properly handle inbound requests and forward them to the appropriate service, so I opted to use the gui for nginx built in to the Synology for the time being.

After doing some more research on the subject I found that there is a linux application KeepAliveD that will enable all the nodes in the cluster to use the same virtual IP and through VRRP determine which host answers on the address, and allowing any node in the cluster to fail without taking all the services with it.

The End of Our Story

So, now I have this semi-complex setup to acquire new media and to consume said media in a friendly way, that flexes some principles that I use in my day job. In the near future, I want to utilize some internal process monitoring and external monitoring so I know if the services go down. I’d also like to take a better look at Traefik to move the reverse proxy into the docker environment. Not too shabby for someone who doesn’t know Linux.

** If only there was a tool for this, that could make sure that my environment stays homogenous and all commands are executed the same way every time on multiple machines…

OMGWTFBOOOM

Now, I know what you’re thinking, ‘What the hell kind of title is that?’.   Well, I have been busy lately with kids and work and my usual environment for writing has changed for the better but is no longer as conducive to me getting any writing done.  So, now I am picking a different time-slot to put together something for reference when I come back to change things.

To properly set the stage I need to give some background.  I am part of a local group of IT folk who tend to do things together, on our Slack channel someone posted that they have a set of old home theater speakers free for whomever wants them.  I had designs to build good sound for my living room television, along with a more secure cabinet and whatnot.  I got the speakers, they were some high-end Polk Audio with solid enclosures, after removing the protective screens all the drivers appeared intact. I was excited.  Then I started shopping for feature rich receivers and the price made me shy away, never mind the amount of work it would take to build a cabinet to fit in the space and I would have no way to protect the drivers from the errant Hot Wheels car thrown by a child not getting their way, so the speakers sat in my office taking up space.

Fast forward to a few weeks ago I was going down the rabbit hole on YouTube (Google’s term) and I re-discovered people making their own Bluetooth powered speakers.  While watching one of the videos I happened to look over at the 6 speaker cabinets that I had just taking up space in the corner of my office.  Longer story short I had done some cleaning, my workbench was available, and the tinkering could begin.

First some research on what I had and what it was capable of; 1 CS1000p Powered Center Channel, 2 FX500I Surround Speakers, 2 RT800I Floorstanding Speakers, 1 PSW450 Powered Subwoofer.  Right off the bat I had keyed in on some terms that piqued my interest, specifically ‘powered center channel’.  I hefted this beast on to the work bench and now it was time for discovery.  On the back of the CS1000p, there were 2 screw down inputs one for mid/high frequency and one for low, an RCA jack for subwoofer, a switch for high level input on and off, and a traditional power input.  On the front I could see a total of 5 drivers, what appeared to be 2 woofers, 2 drivers for mids and a single tweeter in the center, there’s also a volume knob and a green LED.  I did not see a power switch of any kind, so I assumed that this was an always on sort of thing.

Next step was to test functionality, I broke apart a headphone cable to get leads that I could connect to the inputs on the back, I powered the speaker up, connected it to my tablet, and nothing.  The speaker powered (evident by the green light on the front) but nothing was coming through the drivers.  Well, this may end up being a little more difficult.  Time to break out the trusty screwdriver and let’s see what’s inside.

The first thing I noticed was that there were a bunch of screws on the metal back plate.  I pulled what appeared to be the screws holding the plate in place and was able to remove it from the enclosure pulling it away and disconnecting the cables attaching the guts to the drivers.  I was able to mostly identify what circuitry was at work here.  I had 2 channels but unsure which speakers they were attached to and some thinner gauge wires I assumed for the volume knob and the LED.  Still attached to the plate were a power conditioner, what appeared to be an amplification board, and a power converter from AC to DC power.  Upon closer inspection the plastic housing around the Subwoofer input and the Switch for High Level input looked like they either had some weird epoxy around them, or they had melted.  I had a hope for epoxy but turns out they were melted.

Now is the point where I must interject that I do not have very much electrical knowledge, but I usually have an approximate knowledge of what I am looking at.  I was able to identify the amplification chips and put together a rough idea on the path of audio signal.  I wanted to try one more time to power the components and push audio through, but the results were the same as before.  The parts on the back plate were going to have to be replaced.  Okay, let’s look at the drivers.  A couple hundred screws later I was able to get part numbers on the individual drivers and get what their power rating was.  I also found that there was a crossover board behind the center 3 drivers, which handled throwing the highs to the tweeter.  Some shopping and YouTube lead me to a replacement amplifier and Bluetooth module.  The video made everything look simple.

Being one to like having multiple opinions, and not wanting to throw a bunch of cash at a learning project I looked around to see what others were doing.  I then stumbled on a video where someone had found an amplifier board with Bluetooth built in (Hey, less wires!) and it was cheaper than the separate boards, I also found that it had a bunch of extra headers for expansion.  Almost like this board was built for this specific purpose.

The board in line for purchase is manufactured by Dayton Audio, it is a KAB-250v3 it delivers 50 watts to left and right channels (remember I have 2 channels inside the speaker) it’s a Class D amplifier board, with Bluetooth 4.0.  In the box it has the board and wiring harnesses for power, speakers, a Bluetooth pairing control button and an external Bluetooth antenna.  Also, in the box was a quick-start guide, which was just detailing out which headers are for what feature.

Right about here is where the anxiety set in.  In the past I have caused some irreparable damage to 12v electronics by being careless, so I wanted to be extra cautious, I had monetary investment into this project at this point.  Before moving forward, I wanted to make sure that the speakers and my amplifier would be compatible.  While on an errand with my wife we were close to a music store that sold pro audio equipment, I gave it a shot, talked with their pro-audio guy.  He gave a bit of a shrug and said, “I think you’re just going to have to be brave.”  He did clue me in to make sure that the impedance of the drivers matched up and to start low before going bigger.

Okay, moment of truth.  Time to test the Amplifier board with a single driver.  I pulled the driver out of the enclosure and put together a bunch of temporary connections, time to power everything up.  The outlets on my workbench have individual switches to turn them on and off makes it nice for instances like this where I can flip a switch faster than I can frantically unplug something.  I checked all of my connections to make sure I had continuity in all the places I should and then flipped the switch.  I don’t know what I expected, maybe some sort of startup noise, a flourish of non-existent LEDs, something.  All I got was a single green LED on the amplifier board and a little EM feedback or coil whine (I assume) on the driver.  I pulled out my phone and set off to discover Bluetooth devices and there it was “DAKAB”.  Initial pairing was a little shaky, but I really think I just didn’t have a handle on what the controller button actually did.  I got paired and time to fire up Google Music and hit play on a playlist and I immediately find that my Bluetooth volume was set to max.  But it worked!!!

I did a little happy dance, my wife laughed with/at me, and now it was time to test the rest of the enclosure.  I broke my temporary connections, reinstalled the driver, connected to the channel and re-ran the test, another success!  One more test, this time powering the whole enclosure and one more success! Now it was time to plan out how I wanted to tuck this all back in to the enclosure to make this fully function.

I opted to continue to use the AC to DC 12V 1.5A converter that I had on hand, just connecting it to the AC conditioner that was already present.  I used Butt connecters for all the connections cause a more permanent solution would be further down the evolutionary path of this project.  About an hour of testing connections and wire management I was able to fully re-assemble the enclosure and a final test to mark the completion of version 1.0 of what I have titled OMGWTFBOOOM.

What I have right now is a surprisingly capable powered Bluetooth speaker.  My plans for the future of the project is to re-surface the outside of the enclosure to introduce a little bit of weather resistance, replace the soft cover on the front with a metal grate of some form, and add wheels cause this thing is understandably a bit on the heavy side.  Additionally, I would like to put together a better method for delivering power and including a battery for some true portability.  Some more upgrades down the road would be an auxiliary input and some USB power for extra functionality.  Last, I would like to add a paint-job to complete the cosmetics.  But as with all projects, this will be a full evolutionary process.

Shall not be infringed

I’ll try to keep this as sane as possible; it’s difficult with some of the arguments being thrown around.  Let’s first set the stage, I am pro-gun.  All the firearms I own are legal.  I have a license to carry a firearm in the state of Texas.  I exercise that right a lot.  None of the firearms that I own have been used to harm another human being and to be clear, I hope that I am never put in to a situation where that would change.  I try to make it to the range on a regular basis to maintain a level of proficiency with the firearms that I own.

There is a narrative across media outlets that is hell-bent to demonize anyone like me.  Every time some of these people get screen time they revive tired old arguments that, despite any and all previous disproval or rebuttal, they claim goes unanswered.  So; let’s put things back in to context.

The term Assault Rifle:

  • Defined by Encyclopedia Britannica as; military firearm that is chambered for ammunition of reduced size or propellant charge and that has the capacity to switch between semiautomatic and fully automatic fire. Because they are light and portable yet still able to deliver a high volume of fire with reasonable accuracy at modern combat ranges of 1,000 – 1,600 feet (300-500 metres), assault rifles have replaced the high-powered bolt-action and semiautomatic rifles of the World War II era as the standard infantry weapon of modern armies.
  • Defined by Mirriam-Webster as; any of various intermediate-range, magazine-fed military rifles (such as the AK-47) that can be set for automatic or semiautomatic fire; also: a rifle that resembles a military assault rifle but is designed to allow only semiautomatic fire.
  • Defined by Google as; a rapid-fire, magazine-fed automatic rifle designed for infantry use.

 

That’s interesting, isn’t it?  Mirriam-Webster is the only one that defines a rifle that looks like a military assault rifle as an assault rifle.  In fact, looking through weapon classifications through the Bureau of Alcohol, Tobacco, Firearms and Explosives (ATF) there is no actual designation of an Assault Rifle except as it pertains to the proper name of specific firearms (AK-47, MP-43, MP-44 and STG-44) Everything else refers to what is defined above as a “Machine-Gun” or “Machine Pistol”.  Funny how the governing body doesn’t use the term Assault anywhere in regulations.

This raises 2 important questions;

First, who brought the term Assault-Rifle in to the narrative?  We harken all the way back to World War II where Nazi Germany coined the term Sturmgewehr (loose translation Storm Rifle, or Assault Rifle).  A more apt term to consider is ‘Assault Weapon’ which has been narrowed down to pretty much any semi-automatic weapon that looks like it could be military issue.

Second, why is a dictionary allowed to push political agenda?  It’s been my understanding that Mirriam-Webster is a de-facto standard for word definition, and now they’re attempting to alter the meaning of a word (or two in conjunction in this case) in accordance with social and political landscape.  I don’t know about you, but I think that should put Mirriam-Webster in the same league as Wikipedia.  Still mostly valid, but cross reference with another source.

Gun-Crime vs Violent-Crime

There is a willful disconnect between Gun-crime and Violent-crime when the statistics start flying.  There are always claims that after a ban on guns goes in to effect, gun-crime goes down.  Nobody likes talking about violent crime.  In fact if you compare the two, the only thing that changes after a ban goes in to effect is the weapon of choice.  The reality is usually somewhere in the middle, while I will concede that having a gun someone determined to cause a lot of damage has less hindering them if they have a gun, but far more devastating atrocities have been achieved without guns.

It’s easier to buy a gun than a …

This is where things get funny.  To legally obtain a firearm, one must fill out an ATF Form 4473, and pass an FBI Background check.  It’s neither quick nor is it easy.  Not to say that the 4473 is equivalent to the SAT, but it is comprehensive and falsifying information on the form is punishable by up to 10 years in prison and/or up to a $250,000 fine.  I frequently ask anyone who thinks purchasing a firearm is as easy as getting a coke to actually take a look at the 4473 form.

A gun’s only purpose is to kill human beings

Well, no, and in the same vein, yes.  There are some gun’s whose sole purpose is to be accurate for target shooting.  But that’s an outlier.  Of course, we’re talking about the AR-15, and similar rifles.   You’re right, it’s primary function is to kill human beings.  Unless you’re hunting small game, it’s not a great choice.  But the ownership of said rifle is usually not for hunting, it’s for defense, be it from a home invader or some other threat to one’s liberties.

 

After only scratching the surface that’s all that’s there it’s an argument between the two sides.  Since this is the United States, there must be two sides, and we must be polarized against each other to maintain our way of life.  Look the only way to truly put the whole thing to bed is stop the mud-slinging, and have an adult conversation about all the contributing factors.  Some takeaways from the narrative to consider; Most people want less murder (on both sides), most want to keep guns out of the hands of those who wish to commit evil.  Lastly there is not nearly enough education about firearms in the world today.  I don’t even pretend to have all the answers, I just know that we’re trying to achieve the same goal, maybe we should work together.

So preoccupied with whether they could..

I will be completely frank here, AI is a terrifying thing.  Sure, some of the more harmless implementations are cool, but if you look down the road AI is on, you’ll surely see the problems.  It’s great to conceptualize and implement an app on your phone that will automate tedious phone calls (See Google Duplex).  But considering what else is emerging in technology it’s not hard to imagine what the future will bring.  I want to believe that everything will be used solely for the benefit of all humanity, but I realistically know that’s not going to be the case.

So, let’s paint the landscape.  AI can be classified in one of two buckets; Narrow, and General.  Narrow can be defined as a machine (or computer) producing human-like results or decisions in a small subset of tasks (e.g. image recognition).  General can be defined as a machine (or computer) producing human-like results or decisions in a greater set of tasks (e.g. autonomous interaction with the world).  To be clear most of the implications of AI that make it to the news are narrow.  Some examples; would be Siri, Google Assistant, and Cortana, these applications are good at listening to your voice and returning results that you are asking for.

By in large Narrow AI is mostly benign, until of course you start applying it to teach General AI.  Some other terms thrown around when talking about AI are Machine and Deep Learning.  Both terms are essentially a method to teach machines or computers how to make decisions on their own.  A third term that pops up is Big Data which is just an extremely large data set.  Developers and companies will use Big Data to teach Narrow AI.  You’ve probably seen the captcha images that ask you to select the portions of the image that have a specific object in it, these are usually used for machine learning to teach a Narrow AI.  If you feed a trillion images to a fledgling AI that do or do not have a cat in them, eventually the AI will be able to recognize any picture’s feline content.

Tin-Foil-Hat Warning.

Now, the scary part; We’ve been made comfortable with the fact that we’re identifying roads for self-driving vehicles, but what we haven’t even thought of is all the surveillance data collected by the NSA.  This is largely our behavioral information, and since it was collected in a method we would think of as unethical, it’s not a stretch to think that the data will be used in an unethical manner as well.

Then you’ve got the big names collecting an absurd amount of information on people.  Facebook, Google, Amazon etc.  These companies are using that data to build AI.  Unfortunately, these companies are in business to make money and fiscal partnerships are bound to happen.   The prompting of this particular post is just one of those partnerships; a partnership with Google and the Department of Defense (Project Maven).  The idea is to assist our unmanned drones better identify their targets.  What nobody is saying out loud is this is attaching an amount of fire power to AI.

Let’s move on to the truly terrifying; AI doesn’t have the conscience of a traditional human and will follow orders given to it so long as it is adhering to programming.  Meaning that an AI will act on instructions that a soldier simply will not.  Coupled with the people in control of the instructions may or may not have the best intentions in mind.  With both sides of the political argument throwing around collusion with an outside source, and some notable people of power having their data / information leaked or taken over.  The technology is cool, and I’m curious to see what we can achieve with it.  The problem is, of course, who’s driving it?

 

Know the Content

The landscape of the world today is vastly different from when I was growing up.  Having a viewport to the internet in your pocket has changed the game.  In my adolescent years the internet hadn’t really come of age, and for those that would argue, it’s access wasn’t nearly as ambiguous as it is now.  Technology in general is a requirement today, how to interface, operate, and fundamental concepts will dictate your personal and professional life.

With its prevalence we must do everything we can to make sure that our children have every advantage as they grow, which means introducing them to technology so that they can succeed in the world as we can only imagine it will be when they reach adulthood.  Being an IT professional means that I have an affinity for technology.  I used to say I am the purveyor of all things cool in the world of tech.  I’d like to think that I still am, though there are some boundaries to what I can obtain which leads me to make more careful considerations on what I purchase.

Pulling back to the parenting theme; We as parents have a role in our child’s life that is more impactful that we can fathom.  We are the ones responsible for the introduction of technology to our children, and with it the internet.  As powerful as it can be I think a base level of understanding needs to be in place before ever setting a child loose on the internet.  It can be a great learning opportunity, but it can also be the first step down a slippery slope or even downright dangerous.

Consider this; There are lots of things in the average home to drink.  You’ve got water, milk, juice, alcohol, etc.  Sure, you immediately dismiss alcohol as an option, but do you limit the amount of juice you give to your kid?  Milk?  Water?  What if you give them Juice too often?  It isn’t inherently bad, but have you seen a child deny water in favor of juice?  This is curating the content that your child consumes.  Basic interacting with the internet and technology is similar.  There needs to be a balance in place so that basic skills and learning techniques are still present.

With that analogy in place just how much internet are you willing to just blindly hand over to your child?  Let’s face it here, there’s some applications you can get for mobile that are educational, but they are the beginning of the avalanche.  How long before the application becomes boring?  Now on to Netflix, or YouTube, or any other of content delivery networks?  While trying to understand how I should introduce technology into my son’s life, I had to take a few more steps back to appreciate where my understanding of it came from.

Largely technology and the ambiguity of the internet have lead many to take it for granted.  It has become my realization that more and more people don’t understand where we came from.  What did we do before the smartphone?  What did we do before the internet was always on?  I really think that missing out on physically reading a book, writing things down on paper, or interacting face-to-face with someone is incredibly detrimental to a developing child.

I think it’s tough to appreciate an always on fast internet connection without knowing what it’s like to have to tie up a phone line for it.  It’s difficult to understand the pathways that have opened because we can transfer data across the globe in seconds rather than days.  This is not to say that I will subject my son to a decade of internet at the painfully slow 56k (or worse).  But I want him to know how to talk to people, how to read a book, and how to write.  Additionally, I want him to be able to learn at will and not rely on an internet connection.  If all goes according to plan the light I’ll have to tell him to turn off in the middle of the night will be a flashlight, not a smartphone or tablet.

What I am trying to convey here, is I want to give my son, at the very least, a Cliff’s Notes version of technology history before introducing him to all that the internet and technology has to offer, because I want him to understand how powerful of a tool it can be.