Rack mount hardware is almost always expensive, loud, and power hungry. I just have never seen the point of building a home lab like this.
A single ATX desktop can do almost [1] everything a homelab at a fraction of the cost & power consuption. I think a lot of the reason for homelabs/server hardware was to get access to more CPU cores, now that 8+ cores are very cheap it is actually cheaper to buy a new consumer desktop than it is to run an old server.
What makes even less sense is wanting use to use software like vSphere or ESXi since its about 10x more complicated than just using virt-manager/QEMU. It's like using an excavator to dig a fire pit. Server hardware & software makes sense when it's not your home because then you do need a remote access tool like iDRACK. (There are DIY options if you just need something for personal use).
That said if you enjoy it as a hobby (or your homelab is actually a business thing) then go for it.
> Rack mount hardware is almost always expensive, loud, and power hungry.
Buying old rack-mount server hardware for home use is almost always a mistake. Old server hardware may feel cheap when you see an old dual-socket rack mount server on eBay with hardware that was fast 8 years ago, but you can probably meet or exceed the performance with something like a cheap 8-core Ryzen.
Rack mount servers are also exceptionally loud. Unless you love the noise of small, high-RPM server fans, you don't want rack mount server hardware in your house.
And don't forget the power bill. Some old servers idle at hundreds of watts, which will add up over the several years you leave it running. 24/7 server hardware is a good example of where it makes sense to be mindful of power consumption.
> What makes even less sense is wanting use to use software like vSphere or ESXi since its about 10x more complicated than just using virt-manager/QEMU.
I disagree. ESXi is actually extremely easy to use, as long as you pick compatible hardware up front. The GUI isn't perfect, but it's intuitive enough that I feel confident clicking around to accomplish what I need instead of looking up a tutorial first.
I've always ignored advice even people say somethings too hard or not worth it. And I pretty much never regret it.
I absolutely regret trying to get a used rack mount server running.
The combination of steep learning curve from workstations to server hardware, plus parts that were failing but tested ok, made for an extremely difficult path to troubleshooting and getting it running right.
And that's before you get to the quirk's of getting it to boot and installing an OS and drivers and software.
I love it now that it works, but it easily took 100x the time (yes 100x) and probably 2-2.5x the total expected cost getting it to that point.
Not counting the additional AC unit I installed to keep it (somewhat) quieter.
I usually expect one or two aspects of my projects to have unexpected roadblocks, but for this it was issues with what seemed like every single step.
My experience was buttery smooth. It was plug it in, replace the drives with new SATA spinning for bulk and SSD for fast storage, install proxmox, and I had my first apps on it within a few hours of starting. This was first on a Dell RX720 and later on an HP DL380 Gen9.
I got complete servers from decommissioning projects and they just worked. In 5 years, I’ve replaced a SAS controller battery backup unit on one of them.
The plural of anecdote isn’t data and all that, but if you buy complete gear that just aged out, it worked on the last day they used it and is very likely to work on the first day you use it.
The fan noise and power draw is annoying. Running a house full of VMs (I’ve got about 20 containers plus VMs), it pulls about 290 Watts per the meter. That doesn’t feel outrageous on the power side and is certainly convenient. (It’s about $500/yr in power.)
I have used consumer hardware and enterprise hardware in my rack. There’s pros and cons either way.
Consumer hardware is cheaper, more power efficient, and quiet.
But, there are a few reasons I have switched to enterprise hardware:
* remote management and redundancy features make it more reliable if I’m traveling and want to remote in to do something
* some software works better with enterprise hardware features (i.e. special disk controller modes)
* I feel like my development experience is closer to what I can expect in production
* 100+ gigs of RAM on one system without a big upfront expense
* the occasional PITA of working with enterprise hardware helps me to understand what I might be expecting out of infrastructure team in production, or design ways to make their life easier
> And don't forget the power bill. Some old servers idle at hundreds of watts, which will add up over the several years you leave it running. 24/7 server hardware is a good example of where it makes sense to be mindful of power consumption.
My rack has an always on laptop for applications that always "need" to be running. I then have an arduino in my office that's sole purpose is to wake on lan or power on via UPS until the servers are online when I turn my switch to the "on" position, and put them to sleep when in the off position. Any servers still on after 15 mins in the "off" position just get halted.
Once I did that and the friction of going on/off was so low my power consumption went way down.
Latest Ryzen's CPU perf and max 128GB ECC UDIMM should be enough for most users, but it still lacks homelabby features and flexibilities. I need more than 16x PCIe lanes that supports ACS and IPMI, so I go for EPYC build. It's great except idle power consumption. I with they improve it to Xeon level, but maybe chiplet architecture (8CCD and massive CCD) isn't good about it.
It was my initial thought since X399D8A-2T mobo (X399 but has IPMI) existed for TR 2000. But they changed chipset for TR 3000 and TR 3000 don't offer 16 core SKU (it's enough for me). I also found that even TR 2000 build isn't much cheaper than EPYC because it needs ECC UDIMM that rarely sold cheaply rather than ECC RDIMM. So finally I go for EPYC Rome.
>What makes even less sense is wanting use to use software like vSphere or ESXi since its about 10x more complicated than just using virt-manager/QEMU.
I always just assumed it was as a learning experience. When I was in my 20s, I didn't try to get Apache, qmail, and bind running because it was practical for me, I wanted a marketable skill. There are lucrative jobs out there for people who know these technologies.
That was my experience as well. Home labs seem to be cyclical for a lot people including myself. Started with a big overkill rack to learn technologies, and now I’m down to a 10 inch desk rack that is ATX case size with just a few things to run my network and small VMs.
The application side is a more understandable. There are lots of reasons to know and use apache, postfix, etc.
As far as learning goes, if you really want to work in enterprise IT then vSphere is good to know but if you are willing to learn things on your own then you might as well learn kubernetes, docker, etc.
Knowing things like vSphere is cool & perhaps useful but it also hides the how things work. If you want to know and understand things it is better to stick to the open source & interact directly with KVM and Xen. Like you wouldn't use cPanel to learn how a LAMP stack works.
What if you just want to spin up servers in an easy to use platform so you can use them to other things, and not learn the integrals of virtualization?
Then KVM is still probably the move. Install Ubuntu, then install Cockpit for a web interface to manage KVM and Docker. If you want a little more depth, Proxmox is also good but there’s some mild learning curve there.
Best to avoid vSphere/ESXi in that case. I learned using qemu and was able to step into many roles immediately, including some VMware ones. The Linux/qemu/Xen ones pay better.
Absolutely, rackmount servers tend to be very loud and often power-hungry. But if you think of rackmount as a form factor, it can make some practical and aesthetic sense.
The IKEA CORRAS Birch Effect "rack cabinet" in my living room currently has a nice virtually silent GPU compute server based on a consumer/gaming PC in a 4U short-depth rackmount case, a mostly passively-cooled Supermicro Atom server in which I've replaced the PSU fan with a Noctua, and a sinewave UPS. (There's an ongoing evolution of gear over time, including more exotic stuff, and most toys I might play with in the future will also fit this form factor.)
Aesthetically, those stack on the bottom of the cabinet, a Birch Effect shelf sits on top, and the plastic OpenWrt WiFi router with all the pokey antennae sits safely in the shelf cubbyhole. (Once I need detached WiFi APs, I'll probably build a 1U or 2U router with opnSense/Linux/*BSD. And I have a discreet slide-out rackmount console for if I ever get a deeper cabinet.) I like to think it looks like understated black home AV that's not out of place in a living room, which is better than a tangle of assorted non-rackmount PCs and UPS.
Practically, besides the tidy organizing and uniform cooling airflow, I have some rack posts I can use if the stack of gear gets too large/tricky for pulling individual boxes without disturbing the others too much. Also, when I lived in a dodgier student apartment, fastening a bunch of rackmount gear together with security-head screws seemed a good way to discourage a burglar from walking off with my data.
> Rack mount hardware is almost always expensive, loud, and power hungry. I just have never seen the point of building a home lab like this.
Rack mount is just a form factor, so that generalization doesn't make sense. You can get pretty much any capacity systems you need in a rack case, including small cheap ones.
When I got tired of having systems in various different case sizes and a mess of wiring and switches etc, I moved everything at home to rack mounted.
Now literally everything is one big box (16U rack), with just power and network cables going in. All the computers, network switches, wiring, UPSs, KVM switch and monitor. Very tidy, very clean.
The computers themselves are cheap, quiet and efficient, since I don't need any monster servers at home. A couple are Atom-based (<20W total), one Celeron and one desktop AMD-based (forgot model). Same level of hardware I had before rack mounting, but now a lot neater and more organized.
True, if you build something inside a rackmount case then it can be whatever you want. However, from what I could tell empty rack mount case & rails are still a bit of a premium. Were you able to get rack mount case w/ rails at decent prices? Seems each one would have to be at least a 2U to fit a normal ATX PSU ( I guess you could use an ITX or small PSU but those can be more expensive for the same wattage).
How did you handle the system & case fans? Did you rack cases have the standard fans and were you able to configure those to be slow/quiet?
> A single ATX desktop can do almost [1] everything a homelab at a fraction of the cost & power consuption.
So this is true, but the iDRAC/iLO on my big, loud server has a virtual KVM feature that lets me lazily sit at my desk and click buttons and install my server operating system of choice. It saves a lot of space and effort compared to flashing a USB, going and plugging in a keyboard, (mouse), and monitor, and going through the whole deal. I'd wager that's one of the big things that would compel me to buy a rack server. I recently built a nice ATX desktop, fitted with a 5950X and everything, and I found that the PiKVM project [0] does a pretty good job at replacing that "integral" part of the server for me (you can also look into an ASRock Rack PAUL [1], but good luck finding one for sale right now.
> What makes even less sense is wanting use to use software like vSphere or ESXi since its about 10x more complicated than just using virt-manager/QEMU
A lot of people (not me, I end up using libvirt/QEMU as it suits my needs) buy homelabs to work towards having hands-on experience for their system administration job, which uses ESXi/vSphere. It might also be for working on getting certifications from VMware, in which case they really don't have any choice but to use ESXi on their servers.
> Server hardware & software makes sense when it's not your home because then you do need a remote access tool like iDRACK
Now, I addressed this earlier (laziness), but these BMC things are very useful—you can monitor the health of various components of your server, and I believe even update the BIOS without stepping out of your chair. It makes administering a homelab much easier, and even the Pi-KVM, a DIY option, I'm pretty sure, doesn't have monitoring features. Plus, those DIY solutions require wiring stuff into your ATX motherboard, which can get janky and might put off people who want a turnkey solution.
No need to mess with PiKVM or add-in cards. It's a server board with KVM management that works out of the box with Ryzen processors. It might need a BIOS update to support your 5950X, but it will work.
Yeah, I know about those ASRock boards, but they're more expensive than the rudimentary Pi-KVM solution I have right now (a 35 dollar Pi and a 12 dollar HDMI capture dongle; I would use wake-on-lan for powering the board on… if my MSI board's WoL worked). Also, not the one you linked, but the newer B550 ASRock Rack boards are impossible to find for sale—the only place I could find the B550 boards were on wisp.net.au, and I don't live in Australia or New Zealand so it wouldn't be cost effective. Perhaps I should've opted for an X470 board, but it was "older" so I was put off.
Yea that's true. ESXi/vSphere can be very relavent although I found that it was easy enough to learn on the job. The real complicated stuff of vSphere probably isn't going to come up in a homelab but if experience is necessary to get the job then it's worth it.
pi-kvm looks very nice. I would really like to not have to use iDRACK or pay the license fee.
Normal ATX motherboards do lack a lot of features. I'm not sure why none of the normal mobos don't just use LVFS [1] to update the BIOS but luckily they can read the update file directly off the vfat EFI partition so pi-kvm would solve that. I think every modern ATX motherboard also supports UEFI network boot so you could setup a simple DHCP+iPXE server for onboarding machines.
The lenovo “tiny” hardware recommended in the article is really ideal. Its essentially laptop components in a micro case (no hid/battery/screen), they even are powered by a laptop style external DC adapter.
They are affordable, quiet, powerful (modern x86_64 with basic gpu) and light on power usage.
The downside to these small form factor kit PCs is that then you are very connectivity limited. You can't use it to build a NAS directly or GPU connected VMs, etc.
They are quite good as a cheap thin client that you use to access your more powerful hardware. As hardware ages it tends to lack some of the nice feature's like dual 4K@60Hz output, thunderbolt, etc. so having a new but cheap/lowpower machine helps.
Thats true. This model has just one small expansion slot so you have to be clever. But its fine for general purpose compute.
Yeah realistically NAS is out, but thats probably ok. I mean NAS is not a great fit for most general purpose computers. In a pinch you could go usb-3 jbod or something. But personally I think it’d be better to go with either specifically storage oriented hardware, or a scale out filesystem on top of a cheap cluster, something like odroid-hc2.
Personally I like to keep things compartmentalized even at home. So my NAS is a dedicated (off the shelf) system, and the lenovo mini servers mount it via NFS/CIFS.
This isn't always true. I got a 24-port Aruba 802.3at PoE switch with FOUR 10-gigabit ports, for a grand total of $120.
> loud, and power hungry
Again, not always true. Enterprises do care about power a lot of the time.
I highly recommend the Ubiquiti stuff for home lab use -- most of it is pretty quiet.
If you buy other rack-mount hardware, try to buy at least 2U hardware, the bigger fans are much quieter than the 40mm fans in 1U equipement.
If you must buy non-Ubiquiti 1U equipment, you can usually change the fans out for Noctua fans.
> A single ATX desktop
You can build an ATX desktop into a 4U case. I highly recommend the SilverStone RM42-502 for about $250 on Amazon. It takes standard components including standard ATX power supply, standard ATX, micro-ATX, or mini-ITX motherboard, standard fans (or even a CorsAir liquid cooler), it's basically a standard case. If you use quiet components it will be quiet. My ATX rack mount PC is not even noticeable unless I'm running up my GPU doing machine learning stuff.
There are much cheaper cases available as well, but the SilverStone case is quality, and will last you forever, you can just keep building new PCs into it for as long as ATX/ITX exist.
One of the advantages to building your PC in a rack mount configuration is that it's very easy to stack multiple PCs along with your network routers, switches, NAS, UPS, in one nice rack that's easy to move from apartment to apartment in one piece, and all your cables and connections stay nice and tidy.
It's also ideal if you play with a lot of smaller devices. For example if you want to have a cluster of 10 RPis, a rack mount solution is great for keeping the ethernet and power cables tidy, and it isn't going to be loud or any more power hungry than if you had them spread out across the table.
You can also 3D print rack mounts for non-rack equipment, just to keep them tidy.
I have to pay PG&E rates for power ($$$) so I'm a big fan of lower-power hardware for a system I'm going to leave on 24/7, e.g. my 25W-TDP 1U racked 8-core ECC-equipped Atom server built on this board: https://www.supermicro.com/en/products/motherboard/A2SDi-8C+...
> Rack mount hardware is almost always expensive, loud, and power hungry.
It really depends on the particular hardware. I recently picked up an R720 with 128GB of memory and dual E5-2670v1 for $450 on eBay. It idles at 120W (about the same as my brand new Ryzen 5950X desktop). It is not much louder than my old air-cooled 8700K consumer PC. Of course, it's not much faster either, and it's definitely slower than my Ryzen.
I bought it to learn how to use iDRAC and practice ZFS with 10x $20 1GB 10k RPM SAS drives. Also maybe to give Proxmox a try. All of my practical home-prod stuff runs on an old i5 desktop, not my rack servers.
The issue is that power savings stuff is often bad on Linux... especially on older hardware. 'Idle' in this context means running with near zero load since a server cannot suspend or hibernate. In theory you could setup hibernate/suspend + WoL but that tends to not work well.
Some power saving features will really only save 5-10W and can make PCIe cards unstable, poor hard drive performance, laggy keyboard & mouse, etc.
In particuler, GPUs often have poor suport for the normal power saving features. I just pulled the dedicated GPU (GTX 760) out of my old machine that is now a headless server and that saved ~20W. What's weird and bad is that that 20W was being consumed without a graphical interface even running.
>What makes even less sense is wanting use to use software like vSphere or ESXi since its about 10x more complicated than just using virt-manager/QEMU.
I disagree. Assuming your example of a single ATX desktop, ESXi really is easy to setup and modern versions provide a graphical web client. This assumes you're staying away from vSAN, vMotion, and iSCSI storage.
but, assuming you are learning the vmware stack because of employment. Not knowing things like vsan, ISCSI and vmotion makes your lab nearly worthless.
Learning vsphere and esxi properly requires atleast a decently sized cluster, especially if you start throwing NSX in the mix.
Forget even iSCSI as the goalpost, a lot of places still use fibrechannel drives and controllers and FCoE makes sense mostly once you’ve hit at least 10 GbE which is a bit pricey for the sake of learning while it’s basically hot garbage in professional environments worth a career in. Half the point of vSphere in a professional capacity requires use of several nodes such as different HA options, how to trunk networks, distinctions between different LUN abstractions (RDMs in physical v virtual compatibility modes) and an endless parade of the ways different vendors’ SANs and physical switches can completely mess up and ruin your entire month if you’re not pedantic about every random switch or flag in configuration. All these other vendor minutiae being so important rather than the general concepts as a practitioner is part of why I don’t do VMware professionally anymore and went quietly to SRE janitoring in major cloud providers and chalked those days up as time in IT. It’s just much more practical to learn AWS than to pore over the Cisco Nexus documentation unless one’s career really is in the lower infrastructure levels of tech.
There's a lot to be said for 1L form factors, which were included in the article, which are basically the size of a Mac Mini.
I have an HP ProDesk 405 Mini, with an AMD Ryzen 7 PRO 4750GE (Zen 2 based). 8 cores, 16 threads, with a base clock of 3.1GHz and boosts to 4.3GHz. Includes 8 Vega 64 GPU cores. Tossed 64GB of memory into it and a 1TB Samsung 980 Pro. Supports AMD Dash for basic LOM.
Idles at like 12-14W. Always silent. When stuff with the 5750GE hit they'll be better still. It's a great (and cheaper!) alternative to something like a Threadripper 5970X (when it hits in Novemberish) desktop if you wanted to cluster a few and have a low-power Apache Spark cluster that can actually rip through things. I think the only downer would be the lack of 10GbE support unless the next round of 1Ls offer 10GbE cards as options.
I had more space in my network rack than near my desk, so I bought a cheap Rosewill rackmountable ATX case, and rebuilt my old desktop into it (since I pretty much only use my laptop these days).
But I agree that buying old Dell servers for home use is rather silly at this point.
Right? Obviously, everyone should do their thing. But I suppose the tiny tiny "issue" I might have would be - I kind of feel like this reinforces the idea that "having server things in the house is big and complex."
So I encourage everyone who does this also tells the newbs, "I mean, you could also just slap linux on that old computer in the corner and do 95% of what I"m doing here, BUT MINE WILL LOOK COOLER."
Agreed. The startup I used to work for (acquired by Intel) both before and after acquisition had a good deal of lab hardware. Between evaluation boards and rack mount hardware our server room was insanely loud - like "it's 25M away and behind a closed door, but you can still tell when the lunchtime QA and benchmarking runs kick off".
There's no way I could live with that in my house, or even a fraction of it.
I agree. My setup is very much like the setup shown near the end of the article (the one that consists of a couple Synology NAS boxes and what looks like a few Mac Minis or other small form factor PCs). I have a file server with a decently large storage array and a few Raspberry Pis and other small electronic gizmos, some of which connect to my wifi, and some which plug into a PC via various cables (ethernet, USB, etc.)
The only thing out of this that consumes any amount of power worth mentioning is the file server/storage array. I haven't measured how much power it uses (I probably should), but I'm able to minimize it by allowing the disks to go to sleep and the CPU to run slower than when I'm actively using it.
I've never felt limited by this setup at all, but, then again, my home lab isn't really my main hobby.
You really need the separate physical hardware a home lab is not just a bunch of VM's
One of the best things about doing my CCNA courses at a CISCI academy was having 20 switches and routers to play with and find those areas where it does not always work out the way the books say they should.
Yes! Companies refresh Dells pretty often and you can find an i7 for not much money on eBay. Buy two to double up the RAM. If you know the right people at a company, you could get them for free even.
Home labs aren't about millions of hits. It's a playground.
A single ATX desktop can do almost [1] everything a homelab at a fraction of the cost & power consuption. I think a lot of the reason for homelabs/server hardware was to get access to more CPU cores, now that 8+ cores are very cheap it is actually cheaper to buy a new consumer desktop than it is to run an old server.
What makes even less sense is wanting use to use software like vSphere or ESXi since its about 10x more complicated than just using virt-manager/QEMU. It's like using an excavator to dig a fire pit. Server hardware & software makes sense when it's not your home because then you do need a remote access tool like iDRACK. (There are DIY options if you just need something for personal use).
That said if you enjoy it as a hobby (or your homelab is actually a business thing) then go for it.