The state of homelab tech (2026) (Friends)
The Changelog: Software Development, Open Source
2hr 2min Jan 24, 2026
Techno Tim joins Adam to dive deep into the state of homelab’ing in 2026. Hardware is scarce and expensive due to the AI gold rush, but software has never been better. From unleashing Claude on your UDM Pro to building custom Proxmox CLIs, they explores how AI is transforming what’s possible in the homelab. Tim declares 2026 the “Year of Self-Hosted Software” while Adam reveals his homelab’s secret weapons: DNSHole (a Pi-hole replacement written in Rust) and PXM (a Proxmox automation CLI).
Changelog++ members save 8 minutes on this episode because they made the ads disappear. Join today!
Sponsors:
- Depot – 10x faster builds? Yes please. Build faster. Waste less time. Accelerate Docker image builds, and GitHub Actions workflows. Easily integrate with your existing CI provider and dev workflows to save hours of build time.
- Tiger Data – Postgres for Developers, devices, and agents The data platform trusted by hundreds of thousands from IoT to Web3 to AI and more.
- Notion – Notion is a place where any team can write, plan, organize, and rediscover the joy of play. It’s a workspace designed not just for making progress, but getting inspired. Notion is for everyone — whether you’re a Fortune 500 company or freelance designer, starting a new startup or a student juggling classes and clubs.
- Fly.io – The home of Changelog.com — Deploy your apps close to your users — global Anycast load-balancing, zero-configuration private networking, hardware isolation, and instant WireGuard VPN connections. Push-button deployments that scale to thousands of instances. Check out the speedrun to get started in minutes.
Featuring:
- Techno Tim – Website, GitHub, LinkedIn, Mastodon, X
- Adam Stacoviak – Website, GitHub, LinkedIn, Mastodon, X
Show Notes:
People
- Techno Tim - Tim Stewart’s website, YouTube channel, and documentation hub
- Crosstalk Solutions - Chris’s channel, mentioned for building custom Ubiquiti API tools
Virtualization & Infrastructure
- Proxmox VE - Open-source virtualization platform for VMs and containers
- TrueNAS - Enterprise-grade open-source storage operating system built on ZFS
- HexOS - Consumer-friendly NAS OS built on TrueNAS (in development)
- Proxmox VE Helper Scripts - Community-maintained scripts for easy LXC and VM deployment
Self-Hosted Software
- Paperless-NGX - Self-hosted document management system with OCR
- Paperless-GPT - AI-powered enhancement for Paperless-NGX using LLMs
- Ollama - Run large language models locally on your own hardware
- Open WebUI - Self-hosted web interface for interacting with local LLMs
- Plex - Media server for organizing and streaming your personal media library
- Home Assistant - Open-source home automation platform
- Pi-hole - Network-wide ad blocking via DNS filtering
Document Intelligence & RAG
- Dockling - IBM’s open-source document parsing library for AI/RAG pipelines
- PaddleOCR - Multi-language OCR toolkit for document recognition
AI & Agents
- Claude - Anthropic’s AI assistant, used for homelab automation in this episode
- OpenCode - Open-source AI coding agent (mentioned as potential homelab tool)
- Model Context Protocol (MCP) - Protocol for connecting AI models to external tools and data
Networking
- Ubiquiti - Enterprise networking gear popular with homelabbers (UDM Pro, UniFi)
- Tailscale - Zero-config VPN for secure networking between devices
Container & Orchestration
- Docker - Container platform for packaging and running applications
- Kubernetes - Container orchestration for managing containerized workloads
- Fly.io - Platform for running containers close to users globally
Monitoring & Observability
- Grafana - Open-source analytics and visualization platform
- Prometheus - Open-source monitoring and alerting toolkit
Security & Authentication
- Bitwarden - Open-source password manager (self-hostable)
- Authelia - Open-source authentication and authorization server
Databases
- MariaDB - Community-developed fork of MySQL
- Redis - In-memory data store for caching and messaging
- PostgreSQL - Advanced open-source relational database
Hardware Mentioned
- Intel Optane - Ultra-low latency storage drives (discontinued but prized for ZFS special vdevs)
- NVIDIA GeForce RTX 3090 - GPU used for Plex transcoding and local AI inference
Concepts & Techniques
- ZFS Special Vdevs - ZFS feature for accelerating metadata and small file operations
- PCIe Bifurcation - Splitting a PCIe slot to support multiple NVMe drives
- Medallion Architecture - Bronze/Silver/Gold data lake pattern discussed for document ETL
Something missing or broken? PRs welcome!
Adam Stacoviak
\[00:00\] Well, friends, we're back. It is a new year. Not a new Tim, same Tim. Get your hat on Tim. I heard you got some strife on the internet recently.
Techno Tim
Yeah.
Adam Stacoviak
You took your hat off and you started to just have your non-hat Tim going, you know, and a slight uproar. What happened there?
Techno Tim
Yeah. People freak out when I don't have my hat on. In my last video, no glasses, no hat. I broke my glasses and I didn't have a backup pair and I thought, yeah, I'll go no hat too. And so like, you know, you got this crazy person on Tim's channel that kind of looks like him and sounds like him, but it doesn't really look like him. So yeah, people get really confused. Every now and then I do it on purpose though to try to throw off the algorithm for YouTube because I don't know. They're like, maybe we target people who like glasses and now we'll target people who don't like glasses, you know. So the backward hat, you know, I've done that forever, but I've noticed some people are like, take your hat off you're inside, you know. And so sometimes I switch it up because maybe the algorithm will now target those people that think that. Right. So you never know, you know, those games you play with the algorithm.
Adam Stacoviak
Well, of course, man, you got to A/B test. A/B test. Your A/B test, you know.
Techno Tim
Yeah, I mean to. Yeah, A/B test. Yeah. And then a C. And then you take, your C is your B and your next. It's a constant.
Adam Stacoviak
Yeah, fighting for ears and eyeballs, you know how it goes. Ears and eyeballs. Well, we love these people out here listening to our podcasts and our content. We just go on this journey because we're just nerds. We can't help it. We just have to pursue the inevitable, I suppose. And somehow put ourselves to that pain slash pleasure and share it.
Techno Tim
That's right. And that is what you call what we do.
Adam Stacoviak
Man, did you think we would be where we're at right now last year, Tim?
Techno Tim
No, no, not at all. A lot's changed. Even I feel like for homelab, it's changed even more. But yeah, a lot has changed.
Adam Stacoviak
\[02:03\] What do you think has changed? I mean, it's obvious, but I want to hear your own words. What do you think has changed for homelab in particular?
Techno Tim
If I could sum everything up in one word, it'd be availability, just availability. And that goes a few ways, you know, availability in parts. It's very difficult right now to get your hands on server parts. Oh my god. Whether that be used server gear or motherboards, just CPUs. It's very hard to get your hands on those because, you know, I suspect that most of these companies have contracts with really big companies. And so your onesie-twosies orders, or even your massive orders from stuff like Newegg aren't as big as say, you know, Microsoft or something like that. It's really hard to get your hands on server grade hardware.
It also, that also has been true for the secondhand market too when it comes to CPUs, motherboards and everything like that. I mean, homelabbers have always been used to paying the homelab tax. Well, let me take this back. We used to not pay the homelab tax. We used to get the used server gear free, take it away, really cheap, you know, type of hardware. Then people started realizing, hey, there's homelabbers out there and I can make money off this. All right, it still has some value. I guess I should put it that way. It has a lot more value than it used to because, you know, people are using this stuff in their home for homelabs, which was awesome.
Secondhand market, you know, it was great. And now we're at a point where not only do we have the homelab tax because people realize that they could start making money off their secondhand gear. Now you can't even find it. And so that's one big change I see is just availability of server parts, you know, and the same goes for RAM. As you know, prices are through the roof. If you can even find it, you know, prices are through the roof. You're paying, I don't know, double, triple. I'm scared to even look anymore of how much RAM costs. Hard drives too. Hard drives up. I was looking earlier today and I paid $159 a year ago for a 14 terabyte hard drive refurbished. That same one now on eBay, if you could even find it, you know, is almost $100 more.
Adam Stacoviak
Really?
Techno Tim
Yeah. Anywhere from $70 to $100 more. And, you know, storage is just gone through the roof too. It's easy to find storage, but you're paying a lot more than you were before.
Adam Stacoviak
What about CPUs? CPUs the same?
Techno Tim
It is the same. If you can find them. Yeah. Secondhand CPUs, they're expensive. I mean, most people aren't, I mean, let me take this back. If you're building a server with server grade hardware, a lot of people are buying them used. We can't afford new ones. Or don't want to afford, I should say, in some of the cases. But even those secondhand ones are through the roof, because people are still getting a lot more life out of them. Or they can't buy the ones that they want to upgrade to, you know, the latest whatever EPYC CPU. Most of those are allocated to some big customer. So even the midsize customers can't upgrade because they can't get those. So they're not releasing that gear to then trickle down to the rest of us. So it's tough for CPUs. CPUs too. Yeah, DDR5, CPUs, hard drives, motherboards. I feel like the only thing that's really cheap right now are cases and enclosures because no one can build them. No one can build them. So I think like, you know, enclosures are really cheap right now because they're like, please build something, but we can't.
Adam Stacoviak
So, I saw who was it? Gamers Nexus was talking about cases recently. It was about four months back saying the cases were actually up because of tariffs.
Techno Tim
Yeah, yeah.
So then there's that. Then there's tariffs on everything, which is, you know, whatever percentage increase across the board for everything. There's that for sure. But a lot of this has to do with, you know, just the AI race that's going on and, you know, build, build, build, and DRAM prices are through the roof because there's a shortage, shortage of hard drives. There's a shortage of everything because everyone's building data centers right now. So yeah, there's definitely tariffs across the board on everything. But this is like beyond that. This is like beyond hard drive prices, RAM prices, GPU prices. I mean, through the roof, GPUs are another one. Like you can't even get your hands on them. So, you know, if you bought one four years ago, you're pretty lucky. You know, if you bought a 30-series, I still have my 3090. It's actually back there as something I'm testing with.
Adam Stacoviak
Yeah, right at the beginning of COVID, right at the beginning of COVID. I thought, how am I, I don't want to pay $1,300 or $1,100 for a GPU. Like I thought, you know, am I really going to pay this price? I'm so glad I did.
Techno Tim
Wow.
Adam Stacoviak
You know, I've had it for four years. You know, I paid retail for it. And so, you know, if you think, you know, the 4090s, 5090s, they're more expensive than that, too.
Techno Tim
So I bought my 3090 last year for $300 less than your retail price.
Adam Stacoviak
Wow. Yeah. Yeah. Yeah. Yeah. Yeah.
Techno Tim
Because that's when probably the 5090s were just coming, just about to come out.
Adam Stacoviak
That's right. 4090s were out. Yeah. 5090s were being announced and everyone was dumping them. But now it's like, you can't even get your hands on them. But yeah, that's a good deal. That's a good deal.
Techno Tim
Yeah. It's a good GPU. I may have some fun things happening on that GPU as we speak right now. Training a RAG system. Just doing some RAG right now.
Adam Stacoviak
Cool. Man. I was just looking up that stuff, too. Yeah. Yeah. Yeah. So I thought you would say that. Glad you mentioned the hardware shortage because that's key. But I thought you would say this your availability remark would have been not unavailability, but abundance of availability in terms of capability. These, you have minted homelabbers that can now tend their lab garden, so to speak. That is right.
Techno Tim
That is right. That is right. So that's my second piece as the explosion. So then my second piece to availability, that was the other side of the coin. And I'm not just saying that, it's in the notes right here, is the explosion of self-hosted software that we can now run at home. Like it's incredible. And so that's my prediction for this year. This year is year of the self-hosted software. We can't get hardware. We've got to make do with what we have. And so this is the year for software.
And it's the year for software for many reasons. First of all, we have way more capabilities at home. Like I've been running Ollama at home, Open WebUI, you know, to play with models and do chat. I've even done some coding assistance with some agents. That stuff's fun. Models aren't as good as the open ones. All right. I'm sorry. Models, the open models aren't as good as the closed models, the ones you pay for. Right. Obviously, or you wouldn't be paying for them. But they're good enough to do a lot of tasks, especially just, you know, like you were mentioning RAG and stuff like that.
Like I've been playing with Paperless, Paperless-NGX, Paperless, and Paperless-NGX is a self-hosted document scanning solution. So if you think about, if you think about, you have lots of documents that you want to store, you want to store on your own hardware, you know, they might be private documents, whether they're I don't know, financial statements or subpoenas or marriage license. You name it.
Adam Stacoviak
Yeah.
Techno Tim
Whatever you have that's private that you keep, just think of what you keep in My Documents. Think about keeping that on your own servers and then being able to scan those documents and then getting metadata and data about those documents. That's kind of what Paperless does in a nutshell.
Well, now all of these not really sidecar, but these kind of sidecar solutions are starting to pop up where you can feed them to a model and get better data out of those. So right now, Paperless-NGX, it's super cool. That's actually my next video. So you're getting a sneak peek. And that's why I'm so like, yeah, that's why I'm so gung-ho about it right now.
No, Paperless uses traditional OCR. So optical character recognition. And for the most part, it's okay, right? It's okay. It's way better. It's faster than humans. But it is nowhere near the accuracy of a model that's been trained for vision. And so this is kind of the next evolution in OCR. It's not use optical character recognition. Use a model that's been vision trained or multimodal is what they're saying now, where you can feed it text or feed it image and you get text out.
And so I've been playing with this thing called Paperless GPT and Paperless AI, which hooks into Paperless. And now I can scan documents, scan images, and get high fidelity data out of those images. So for example, I scanned serial number on one of my devices, you know, took a picture, scanned a serial number, OCR did terrible. It got like made in Japan, right? It's about it. Serial number wrong. Everything was wrong. You feed it to an LLM that has been trained with vision, like a super small one from Ollama. And everything works perfectly. It even was able to figure out that the FCC trademark, their little FCC logo actually said FCC, even though it was
F with circular C's inside of it. So it's really cool. It's really cool solution, self-hosted solution.
That's where I'm thinking that this year is the year for software, not only because people are making all of these awesome solutions to self-host, it's that people have a lot of assistance now to get those ideas, to make them come to fruition. They have agents to help them. Like I was just talking to a guy the other day who just built the piece of software he always dreamed of, but never had the ability to do it because he's not a developer. He's not a developer.
Adam Stacoviak
And say what you want about that code and whatever. I'm a developer too.
Techno Tim
Yeah. Good.
Adam Stacoviak
Because I think that's, that doesn't work. This is a problem. That's good code. This solves my problem that didn't exist before this moment. That's good code.
Techno Tim
That's right. And so people who are driven by results or who want, you know, code is just means to an end for a lot of people. You talk to product developers. You talk to people who are in IT, but don't do code, but have these ideas. Those are the people right now who are creating these awesome solutions. And finally, being able to get those ideas out of their head.
So anyways, long story short, I just talked to a guy who built this whole solution on top of Ubiquiti's API. It's actually Chris from Crosstalk Solutions. He built this whole solution on top of Ubiquiti's API. And it's like the solution he's wanted for years, but never took the time to do it or paid a developer to do it.
So anyways, we're seeing lots of that now. And I'm hopeful. And it's super exhilarating to see these ideas coming out because when you have people who are generally software developers, I don't mean to buck people up, but a lot of times there's, you know, deep focus on some super technical solution, looks perfect, runs perfect, structured in a certain way. But now I'm seeing these solutions by people who think way outside of the box and they're just trying to solve a problem. And I get to see how they solve that problem. And it's really cool to see because like, I might not have approached the problem the same way that they did. And so I get to see like a new perspective on coding.
So I don't know. That's the other pieces of availability. So I think this year is the year for self-hosting software, for open source software or for solutions in general to be able to run on your homelab because a lot of people are just going to be okay with the hardware that they had for a couple of years. And people like me who love to self-host stuff are always looking for like the next container app to run on your server. I'll fill you on that front there.
Adam Stacoviak
I mean, I've been scratching little itches. That's how I'd say it. Question on Paperless is, can it handle books? I suppose this does the OCR and or the vision version of that. Books are cool too because one thing I'm doing is I'm trying to figure out how to get knowledge out of certain books I only own in paper that I can't even get in digital. Like, there's a lot of books that are just not like that. And I think, and great, I have this book, it's on my bookshelf over there. I've had it for years. I paid the author. I've paid the publisher. It's my copy. But I'm not going to go pick it up because that's the old way. Not that I don't read books. I still read, okay. I still read.
Techno Tim
But you can read good. I can be good, you know.
Adam Stacoviak
I was trying to build a little center for people who can read good. But I'm not there yet. I'm not as good as you like me yet. School for ants.
Techno Tim
That's right. That's a good clip. That's a good clip.
Adam Stacoviak
So can you do books with this Paperless world?
Techno Tim
Yeah. So you can. You can because you can have multi-page documents. So you could. I mean, you would have just basically had this whatever 300 page document. Right. And then you could feed that to the LLM, which has vision.
And yeah, it should be able to parse it all out. Like, Paperless by itself should do pretty good on books as long as you get a good scan on it. But I'd still feed it to the LLM anyways for it to use its vision because it's going to be, you're going to go from whatever 80% accuracy, probably a lot lower to like 90 high 90s accuracy.
OCR in general, you don't realize how bad it is until like you actually try to scan something in the real world. And you're like, oh, yeah, this used to be amazing, but it's not amazing anymore because we have vision-based LLMs that are amazing. Yeah. So yeah, you absolutely could. So I'm thinking, okay, you would scan it. You would get it in Paperless. It would put it in PDF form. And that's the thing that's cool about Paperless too. It tries to get everything in a PDF form.
So you would basically get it in a PDF. I assume the reader that you're using uses PDFs too or maybe uses EPUB or whatever the weird extension is that I can't think of. But I think if you got in a PDF, that would probably be good enough, I think.
Adam Stacoviak
Yeah. Yeah. My preference is to Markdown. I want to get things to Markdown.
Techno Tim
Yeah. Yeah.
Adam Stacoviak
And I got some solutions I'm working on around transcription and just really pure good stuff, which is the same. And that's my next goal is to be able to transcribe with really good accuracy because there's a lot of jargon out there and whatnot. And I'm close. I'm like 98%. I was just 99% there. So I got some curiosities in that front.
Techno Tim
It's so funny. You mentioned that too because I've been going down this rabbit hole on document scanning. This is kind of how it goes when I start researching the video or something that I'm doing. Like this is like the third rabbit hole within this whole Paperless thing.
You know, I learned that there are solutions and people probably know this. I don't because this isn't the world that I live in. But for document scanning, there are solutions out there that prepare your documents for AI. And so it will take say, a document and identify and break it up into its parts so that you can feed it to like what you're trying to do, a system for RAG. So it will understand a title, a footer, and all of these pieces of the document and not just the text itself.
So there's two solutions out there. One's called Dockling, which is from IBM and it's open source. And it takes any document you want, whether it be MP3 PDF Excel and will break that up into its parts and then feed it to your LLM so that you can do RAG against it. The other one is Paddle. And so Paddle is another one, Paddle OCR that I don't want to say does the same thing because people who know this stuff are going to be like, it doesn't do the same thing. But for me, from the outside looking in, it's a solution trying to solve the same problem where it's trying to not only get the data out of the document, but lots of metadata about it too.
So those are two solutions that might help you. And I say that because you're saying you want everything in Markdown, that's going to help you big time because if you scan a document that has a table and you do OCR against it, the text you get out isn't a table, right? And so same with even an LLM. So what you want to do is use Dockling or Paddle to do the transformation or the recognition of the individual parts. So if you took a picture of a table, workbook table, Excel table, then your output could still be a table, but in Markdown.
And so this is like the next, I don't know, I feel like this is like on the frontier of document scanning. And anyone who's doing document scanning in the industry, they're probably like, this has been around for four years. This is coming from like a web developer who does infrastructure at home. And so this stuff is new to me.
So those are two things I would look into. And I get to mention them in my next video because they're pretty cool. But I think that these two solutions, Paperless AI and Paperless GPT are trying to solve that thing. And the funny thing is Paperless GPT can hook into Dockling to do that for you too. So it's getting wild, man.
Adam Stacoviak
I was just thinking about architecture there. So if I, maybe you'll go here with me, Tim, I've been thinking about ETL pipelines. I feel like the world is an API. The world is a CLI. And the world is an ETL. And that means extract, transform, load. And I feel like that's exactly what you're doing there.
So if I were building that pipeline, and I were, you know, using Paperless and I was behind the scenes in your little nerd research lab there, whatever, I would want to keep the original images. And the reason I would want to keep the original images, I would want to extract whatever the purest original copy of it would be, which would be an image, right? Let's not take the transformed version of it. Let's extract literally what we get from it. The raw data. So let's have a raw layer.
That's, if you went with the, I think it's the medallion process, I believe it was called, you got bronze, you got silver, you got gold. And so the bronze layer would be this original raw layer. And so that would be simply images of every page you have. It could be the simple image of your serial placard, or it could be all the pages of the book. Store those in the raw copy as an image. Boom, you got that. That's your bronze layer.
Then the T comes into play. The transform comes into play. You say, okay, let's now take all those images. And this is great as technology or models change or vision models get better is you can go back to that original raw source. It's almost like how they do mastering for films. They go back to an original film that was shot on film, remastered for 4K, but they're going back to those original slides. And so that's kind of the same process. I would want to get original image though.
Techno Tim
Oh, I agree. I agree. I like it. This is similar to like meta fields, you know, in developers and APIs when you scrape stuff. It's like, let's pull out the stuff that we can use and put it in our API. But oh, by the way, we're going to have this meta field that has everything we found to begin with. Just in case we need to come back and process it a little bit later, a little bit better.
Adam Stacoviak
Yeah. We'll leave it on the table if you don't do that. You put on the floor. You're not capturing it. So you tend to throw away in that process to get to the pristine, you throw away what was not really that good to you. In the ETL world, you want to keep that original raw source now in so far as that it does hold value.
But if you go back to that original raw, if you need to ever, as your technology changes in the transform layer, well, then you've got lots of things you could do that goes back and gets more accuracy. If you can't get full accuracy, now if your score is 50 out of 50, you're 100 out of 100 in your quality score and your raw is not really in the way anymore, we'll throw the reference pile, but I want the images. I want the images so that when it comes down to that table, I can actually have the LLM examine the image of the table and then the Markdown we get from it and be like, that's good. Let's go.
Techno Tim
Yeah. I like it. Man, ETL is taking on such a different, I guess, a different perspective. Last time I was talking about ETL was SQL, trying to pull data out of one database and put it in another. In the perspective of this, it really makes a lot of sense because LLMs in general are best effort, best guess every time. That best effort, best guess is going to be different every time and it could be better in the future or it could be worse. Who knows? But saving the source image, yeah, that's awesome. It sounds like a fantastic way to treat analyzing images like this.
Adam Stacoviak
The pipeline is medallion and I said, it's bronze is the base layer, which is your raw. Your silver, which is maybe an augmented version of that that's been cleaned up a little bit.
Then you finally transform it in the final layer in your gold layer, which could be your production database, for example, or your production layer. You've got that first layer raw, that middle layer where you're evaluating things, maybe you're doing some joins, maybe you've got multiple databases, and the final transform is in the gold layer, where you're taking maybe two or three different databases or two different data sources and you're merging them in production.
Techno Tim
That's cool. Yeah. Yeah. That's cool stuff.
Adam Stacoviak
Let's go back to Chris from Crosstalk, and unleashing, let's just say Claude. Claude Opus 4.5 on your UDM Pro or whatever, if you're Tim, Techno Tim, maybe you've got the latest greatest, I don't. Tim, I'm roughing it over here. I got to buy my own things. I know you probably buy your own things, but you get gifted a lot of stuff. I don't mean negatively.
Techno Tim
No, I mean, you get the fun stuff. I'm envious.
Adam Stacoviak
So here I have my UDM Pro that's not even the special edition. It's just the one that's not special, and so that's what I'm using.
Techno Tim
Yeah. Yeah. I wouldn't worry about it. At the end of the day, their software continues to evolve, and so you're getting the latest and greatest everything, even though your hardware might not be up to stuff. That's the cool thing about Ubiquiti in general.
Adam Stacoviak
It just does like light dances and stuff. You got all your RGB stuff, bro. I mean, when I put on my beats and I take my dance break, I want it to dance and do a light show for me. I just don't have that capability like you do.
Techno Tim
Oh, yeah. No, no. Or you could play Snake. Do you see that video? Someone playing Snake on there?
Adam Stacoviak
Yeah. It's pretty wild on their display.
Techno Tim
That's right.
Adam Stacoviak
So this world, something that happened recently with me, one of our neighbor friends came by, one of my son's friends came by, and he brought his Switch. His Switch 2, as a matter of fact, after Christmas. It was one of his presents. And like anybody who's inviting somebody with a device into their home, where do you think I said? I said, you got to be on my guest network. Well, for some reason, they just couldn't get on. Like the authentication happened to the Wi-Fi network, all things checked out as good, couldn't get DNS.
And so I thought it was my newly homegrown Rust project, which is called DNS Hole. So I rebuilt Pi-hole in Rust. If you didn't know this, it's not available yet. dnshole.dev in the future very soon. I'm waiting for one or two more things to happen before I can do that. But right now, even as we speak, my DNS is being resolved by my own DNS server that has fully replaced what Pi-hole is. I think you'll love it when I can release it. As a matter of fact, I'll share it with you soon if we can.
All that to say is that he couldn't get on via DNS. And I'm like, gosh, DNS Hole. Maybe you messed up here. It was not DNS Hole. Okay. It was, DNS Hole was perfect. You know what it was?
Techno Tim
What's that?
Adam Stacoviak
It was VLANs, man. It was my VLAN rules.
Techno Tim
Okay.
Adam Stacoviak
So of course, I popped out Claude. And I'm like, what's going on here? Because I couldn't figure it out on my own. And I'm like, gosh, why didn't you just pull out Claude and let it just log into your Ubiquiti and just check out some things. And so through investigation, it turns out I had some jacked up VLAN rules.
And while I was in there, I was like, you've done this all wrong. You know, this is great work. You've done a bit like, you got old rules. You got these rules that conflict. You got this one rule that does nothing. You got this whole set of rules. It doesn't make any sense. Can I fix a few, please? Sure, Claude, please, please help me out.
Five, ten minutes later, beautiful VLANs near you again. All the routes great. He's on the internet. They're playing and having fun. They're playing Mario Kart and life is good. So I mean, like, that's the world we live in, Tim. I can't even get my VLANs right, but Claude can.
Techno Tim
That is awesome. So I haven't used Claude in that way. Like, to be honest, I haven't used Claude all that much. I use Copilot with models, you name the models. But no, that's interesting. So did it do it through the CLI?
Adam Stacoviak
\[28:52\] Yes.
Techno Tim
Okay. Yeah. Got you.
Adam Stacoviak
Well, those are like, it has my, I've got an SSH key. So I'll SSH into my UDM Pro.
Techno Tim
Got you. So it's you.
Adam Stacoviak
Yeah. Yeah. Yeah. So logged in. It could do. Listen, I'm okay. Hold your seat. It reads the Mongo database directly. It updates the Mongo database directly.
Techno Tim
I know that's, it's not cool. Production. Don't read the database.
Adam Stacoviak
Yeah. But I've done it before and I was confident. And then it will trigger whatever it does to let the UI catch up, essentially, like the cache layer that's in the UDM Pro, whatever. Just because you change the database, it doesn't mean that the reads come back quickly. You got to sort of re-cache the cache kind of stuff.
Techno Tim
And oh, yeah, man, it's so cool.
Adam Stacoviak
It will log in to the UDM Pro via SSH as if it's you. Or in the case of Chris, which I'm sure he did or was thinking about doing, is you can use your own, you can use the API, the UI API, or you can just log right into it and just SSH around, just CD directory and, you know, like you're on a system, like you're a sysadmin, no different.
\[30:03\] I think that's such a wild world. I think that's what's making homelab more special with me. Proxmox has got a little more fun. Tim, if you— I'm about to show you some things. Okay. I have a CLI built called PXM, stands for Proxmox. And in a one-liner, Tim, I can have a brand new Ubuntu machine running. I can specify the IP address. I can specify the CPU, the RAM, and the disk. It already has my SSH key. And literally in less than 10 seconds, it's reporting the IP address back to me via the CLI.
Techno Tim
Yeah, yeah.
Adam Stacoviak
And one liner later, that same— I can do PXM info and then whatever the VM ID is. So PXM info 104, for example, and it reports back to me. SSH user is Ubuntu at whatever IP address, all that good stuff, whatever the details are of that machine. And moments later, my agents can be building on brand new infrastructure. And that is awesome.
Techno Tim
Yeah, no, it is awesome. This is exactly what I'm talking about. This is exactly what I'm talking about. It's like AI and agents in general are just letting people get these ideas out of their head and tinker way more and go way deeper than they used to before.
And this reminds me of when I went from IT to a software developer, I went from using other people's tools to using my own tools. And for me, that was like a light bulb. I was like, I don't need the UI anymore. Give me an API or even a CLI. And I can figure it out. And for me, that was just like a light bulb went off. And it was just like this moment where I was like, I felt like so much freedom, you know, to be able to build software that I wanted.
And so now, it's just so awesome to see like other people being able to do that, you know, take that step from using other people's stuff to using my own stuff. And so now I feel like— like you, you know, I mean, would you have ever written that thing out in Proxmox, you know, five years ago? Maybe. But it would have taken a long time.
Adam Stacoviak
It would take you way too long. I wouldn't have the time. It's just too daunting of a task to do.
Techno Tim
Exactly. Because it's really time. It's not necessarily ability.
Adam Stacoviak
And I suppose it's probably both time and ability. But yeah, I would have just never tackled it because it had just been too hard of a mountain to climb, really. Because I mean, even with the augmented AI tools, it was still hard. I mean, it didn't get easier. It got easier to move faster and to get past the hurdles. But gosh, I had to solve so many problems. I forgot so many ways to deal with— how do you store the image on Proxmox? Well, that's kind of obvious to most people, but like getting through the whole lifecycle.
And that was one of the first things I built with Claude. So I've learned a ton since then. So I want to rebuild it. I want to go from— because now I know what the tools should do. And before I was trying to make this— I don't want to try to make it. I was just trying to explore really. And now I know exactly what I want it to do and what I don't really care that it does that I just don't need. And so I would just wouldn't waste my time on that part of it.
Because I was trying to make this— I guess, I didn't want to have to log into my Proxmox machine every single time and navigate the web UI and click all the things. And it's not that it's a bad UI. It's just that that's just— that's not the way. And you're more— you know, I wanted to be able to do a CLI version of it. I wanted to get JSON back and feed it to my agent. And now that's all possible, really.
So have you played with or heard of this latest thing, which is called Ralph Wiggum?
Techno Tim
Ralph Wiggum? No. That name sounds familiar, though.
Adam Stacoviak
From the Simpsons.
Techno Tim
Okay.
Adam Stacoviak
Yeah. Gosh, why is it called Ralph Wiggum? I forget. I think it's because they just keep trying despite setbacks is how I— if I can phrase Ralph Wiggum is keeping the loop going despite setbacks.
And so I believe it was, you know, I don't have it here. I was going to try to figure out who actually created— I think his name is— I don't know, I can't remember, but it was somebody who discovered this loop essentially. So you essentially keep feeding back the loop of the input output that you would normally do with your own typical Claude scenario, which is, you know, you entering the prompt, it doing something and returning some sort of, you know, response back to you and doing work in between.
Well, they have found this way to create this Ralph Wiggum loop. So that you can essentially define a pretty clear instruction set. You might call it a spec or a spec, but they actually just call it prompt.md. And so in this prompt.md, which you would feed into Ralph— can do a loop. It could be a small loop like, you know, build this one part of the feature end to end and you just go until it's done.
Well, the reason why I'm telling you this is because I feel like now if I— now that I know what it could do and what it should do, I would want to— and if hardware was more available, I would be more inclined to do this, but I would build a test subject hardware machine. That is Proxmox. And then I would just set loose this— now that I have a pretty clear vision. I would set loose this thing on that machine. Just have it build this Proxmox redo, I suppose potentially, you know, because I'm just trying to get the value out of it, not so much the pristine code. Sometimes that's the value part too and you enjoy the process, but just for an exercise, because I want to automate Proxmox.
Why not do it via this Ralph Wiggum loop, which is just do it until it's done and you can kind of give it repetitions. You can say, okay, do, you know, one, you know, one version of it— not a version, but I think like tries. I forget what the terminology is for it. Let me see if I can find it real quick. It's like one iteration, two iterations and you can specify because you have, you know, only so many dollars to spend. I don't want to spend more than 20 bucks on this feature, 10 bucks on this feature. So either spend 20 bucks on the feature or, you know, 10 or 15 iterations until you get to some result. And then I'll come back and examine it. And to run it again, all you do is just run it again. It's like— I don't put it in that way. It'll just go back and do it again. Like that's such a cool world to be in, man.
Techno Tim
Yeah. No, it is a little homelab garden with that kind of loop. So cool.
Adam Stacoviak
Yeah. Yeah.
Techno Tim
No, that is for sure. Because yeah, a lot of times, you know, agents will stop and I know exactly what you mean now because, you know, you could tell it to do something to completion, but it's either going to stop or check in or do something, you know. And I know, I know the prompts that I give— I have a lot of my prompts saved because they're, you know, annoying to keep explaining to AI, you know, like, you know, fix all unit tests and run all linting and do all this, you know, do these things. Don't do these things. Go, you know.
And to be able to not be a human in the loop anymore until the very end is pretty cool to think about. It's like, no, you loop, you figure it out. You do so many iterations of this piece of software or you do one really good iteration. And let me see the final result. At that point, you're just like a director, you know what I mean? You're just like, you know, a director.
Adam Stacoviak
You see, that's right.
Techno Tim
Yeah. Give me, give me something.
Adam Stacoviak
That's right. A parent. Go, go clean your room. Don't come back until it's all the way clean. And you know what? I'm going to check under the bed. I'm going to check in the closet. So, you know, make sure you don't put stuff under the bed and in the closet. And when I come back in, you know, an hour, it better be done.
Techno Tim
Right. I'm not a parent, but, you know, I remember those days, you know, thinking— thinking I think I can outsmart my parents by hiding everything.
Adam Stacoviak
Yeah. Yeah. Yeah. Yeah.
Techno Tim
They're pretty clean though, you know, and they don't clean up after themselves. I mean, not yet. Well, generally, unless, you know, something— yeah, I won't go there, but yeah.
Adam Stacoviak
\[38:17\] What is the centerpiece of your homelab right now? Like, what are the centerpieces? I imagine Proxmox and TrueNAS is still there in the center. Unifi hardware is obviously probably part of the center. What's around that center? What are you building on?
Techno Tim
It is. So this kind of goes into, you know, another one of my predictions for this year too is— if we could ever build anything— is one big box. I think people are going to return to this one big box idea. Where they're— yeah, only because things are hard to get a hold of. And, you know, while you might be able to get a hold of lots of little older machines, you know, the one-liter machines, you know, to do, you know, clustering, I feel like now that things are so scarce, people might be going back to one big box that's your storage, that's your AI, that's your compute, that's your, you know, virtualization, that's your NAS, that's your everything.
And it's kind of the way I've been going too. So, I do have my TrueNAS box. It's one big box, you know, has a video card, has RAM, has 10 hard drives, you know, GPU, all that stuff. And so, that's, you know, not only my NAS running ZFS, but it's also where I'm running my applications now too. So, I've moved a lot of my applications on my NAS.
Adam Stacoviak
Yeah. So, I've been doing this— you know, where you were doing stuff with containers, you had to do that sort of sidecar load, which I did follow. But I didn't get the same results you did. Maybe is that the way you're doing it with that whole— you have to create the YAML file that it knows about it in the app container world?
Techno Tim
Yep. Well, if you're talking about TrueNAS, yeah. So, I— so you don't have to use the YAML and do it that way. I do, because I want YAML, because I'm a developer, but also it's a lot— I'd much rather edit YAML than fill out a form any day, even if you could give me, you know— yeah. And so, I'm also— because then you get CLIs and you get help from AI, you get all the stuff you get with YAML. And I can do it in VS Code. So, there's that too.
So, yes. So, I'm now running my applications on top of my NAS. So, I've always gone back and forth. Like, you know, do I want my NAS to just be a NAS and just be storage or do I want my NAS to be an application server too, and then run those applications on top of my NAS. And so, I've done both. And I'm still kind of doing both, but for the most part, what I'm calling now my home production is on my NAS.
And so, my home production, you know, I've gotten a little bit wiser over the years and a little bit crazier, but, you know, I have a home production now. And my home production is where the services are that need to be up. They need to work, or I'm going to hear about it, you know. That's Plex, that's my NAS, that's, you know, whatever else I have running, which is a lot of stuff.
And when I say, I'll hear about it, I don't just mean my wife, because she will say something if Plex is down, because we record a lot of stuff. And if it doesn't record Survivor on whatever night— I hear about it. So, that needs to be up. But also, you know, alerts and stuff I have set up and running too. So, you know, my home production, that is right.
Adam Stacoviak
Yeah. I've been getting so deep in there.
Techno Tim
Oh, yeah.
Adam Stacoviak
I've been getting so deep in Grafana and Prometheus now. And it goes back to what you're saying. Like, you know, I do a ton of— well, I've done a ton of Grafana and Prometheus in the past. A lot of observability stuff, you know, in the enterprise world, a corporate world.
But at home, I was always kind of like, man, that's a lot of work. That's a lot of work to get that going. You know, now that I have help from, you know, LLMs, it's work I want to do because while I could muddle through it and spend a week getting, you know, scraping, working on one machine, you know, the trade-off just wasn't there. And so now that I can scrape metrics on one machine in about 10 minutes, the trade-off is there. And so it's worth it to me.
And so, yeah, tons of tons of— I'm monitoring everything now. I have metrics on everything now. You name it. You name it. And I'm going to show off some of this pretty soon in my homelab tour. I do every year. It's coming soon.
Both the hardware and software? Everything I host and run?
Techno Tim
Excited. Yeah. So that's coming soon. But I, you know, I need to make it really good for you to, you know, see— you know, people have certain expectations. And some reason it's always got to be a little bit better. Or people will be like, oh, that's what you did last year. So, yeah.
So I'm running my applications on my NAS now. And there is a reason for that, not just because I want one box, but ZFS is a really, really good file system. And when you layer on stuff like, you know, caching, and then metadata, special vdevs and separating out your metadata and putting that on ultra fast storage. And then putting, you know, your app data on fast storage too, you get this really good bulk storage that can perform like NVMe storage.
And so that's been my idea, this kind of crazy idea that I have. I'm putting 10, 14 terabyte hard drives in an array, you know, I'm doing striped vdevs kind of boring. But and then on top of that, I'm, you know, augmenting the things that need to run fast like metadata lookups and app data. And putting that on super fast storage. But anything in that ZFS pool can also use it. So it's kind of a tiered approach to storage, which is RAM. RAM is going to be the fastest. Then I have this, you know, special vdev where I can put files on there too that are below a certain file size. And then I have bulk storage.
So my idea is run hybrid, hybrid ZFS, you know, all of my video editing goes on there. But also all of my databases still go on that pool too. And I still get, you know, NVMe-like performance for most of the things that I'm running. So it's pretty cool. And so it's been a challenge for me to get that working. So that's why I'm doing it.
And mainly because, you know, I always look at it like this. You know, if Plex is running on one machine, which it used to, and my media collection is on another machine, now I have two chances for it to be down. If I reboot my NAS or I reboot my application server, right? And so now I've doubled the chances of that service being down in my home. And so I co-located everything onto one box. So now it's like, well, you know, if the NAS is down, that means the apps are down too. But my NAS should never be down. And if it's down, we have big problems. So yeah, that's kind of what I've done.
I still have a Kubernetes cluster at home. You know, I still have three Proxmox in a cluster running Kubernetes. And I still have that on mini machines. That's kind of my homelab test kind of lab where I test stuff before I actually run it in production, which I have a co-location too where I'm self-hosting Proxmox. And another Kubernetes cluster there that's running technotim.com— plug for my website. Like, that's all self-hosted in a co-location on hardware I own running on Proxmox in a cluster running on Kubernetes that I maintain myself.
So, you know, my home kind of cluster is kind of a test bed for that too. So yeah, I actually run and manage three Kubernetes clusters. I have a lot going on, but it's fun.
Adam Stacoviak
That's a lot of stuff to run. I was going to ask you about your clusters because I went the route you did a while back when you built that cluster from the NUCs. I don't leave about one because I can only afford one, but you bought three. And I thought that was cool. And I think that's where you ran your Kubernetes. Then you— or you ran Proxmox on high availability there. One of the two.
Techno Tim
That's right. Yeah. So I don't have Proxmox in high availability. I have them in a cluster. I'm like— I get it while people run Proxmox and need to do that.
Adam Stacoviak
Yeah, because like, I don't need HA VMs. I push that further down— to the right, to the left. I don't know which way it is. I push it further down to the services and then run HA services, right? Like, I don't need an HA Kubernetes node.
Techno Tim
Right. You know, I just build more nodes. It's the whole cattle approach where I'm like, you know, it's great for, you know, if you have a single VM, something super important that's old legacy app that you need to run and it needs to auto migrate somewhere. But Kubernetes, you know, as you know, you don't worry about that. You just worry about the services— you run three replicas. And if one node goes down, one node goes down. So that's the approach I take. I don't have Proxmox in HA. It's just clustered. So I have one UI and I can migrate stuff easily.
Adam Stacoviak
You got your TrueNAS, which is running your home production services.
Techno Tim
That's right. Yep.
Adam Stacoviak
You got your colo, which is in a data center.
Techno Tim
That's right. And that's all of my public facing stuff.
Adam Stacoviak
Yeah. Which makes sense because you want being with there and, you know, maybe no firewall poking stuff like that. All of you can probably use Tailscale or something else to do that.
Techno Tim
Yeah. Yeah. No, I used to— I used to host it out of here. No problem. You know, update DNS dynamically. It was all fine. And I could today, I just wanted to kind of expand it. I had an opportunity from a local person here in Minneapolis to join their colo. And I thought, hey, why not? Why not? You know, let me do some super fun, you know, site to site networking stuff and backups back and forth. So pretty cool stuff.
Adam Stacoviak
What do you run your Kubernetes clusters then? These applications? What do you run on there?
Techno Tim
So Kubernetes clusters. So I have the other applications— anything from Discord bots, websites. So I host, you know, my own documentation site that's, you know, multiple replicas. I have my own link site. I have some APIs that I run— two or three APIs, because I have this mobile app that I use that I built many years ago that's still running. And this mobile app then has, you know, APIs, which then needs, you know, databases. Databases are in there.
Other people's websites, like, you know, my brother has a website I built for him, you know, my other brother has a website I built for him. So those are hosted in there. A whole bunch of— I'd have to look, but just random stuff. But it's basically web dev stuff, a lot of web dev stuff.
Adam Stacoviak
And is Proxmox mainly just skunkworks stuff like lab stuff then?
Techno Tim
\[48:31\] Well, those are— so Proxmox is actually the host. My Kubernetes nodes are VMs in Proxmox, right? So I'm not running Kubernetes bare metal.
Adam Stacoviak
Okay.
Techno Tim
I'm running Kubernetes as virtual machines. So I have nodes that are Kubernetes nodes running on Proxmox. And the Proxmox is also running some LXCs, like DNS, Postgres. So I have Postgres in a cluster running on an LXC, because, you know, kind of mixing that into Kubernetes while it does work, depending on your IO can go bad really quick. And I don't have a ton of IO.
And then, yeah, LXC— I'd have to look at the list. But I use LXCs too. I was always kind of against them. I know they're doing containers now too, which aren't really great. It's kind of like a hack up how they're doing containers now. But I am using LXCs for small things. And when I say small, I mean, like, I don't need a full OS for them. Right. Most of the time, I don't.
Adam Stacoviak
It's interesting. The way you're using TrueNAS because I was always in the camp of let my NAS box just be a NAS box. But I'm kind of bummed because it's a Xeon CPU. It's a ton of RAM. And so I look at it. I'm like, well, you're not doing really much. I mean, like a file server is not taxing. As an enterprise, maybe with like thousands of users. That's the box you want.
And so I've never really been happy about that scenario that I'm— but then I'm like, you know, what, one problem, one issue, it's a NAS. I don't want to conflate what's on there because if I start putting applications on there, different things on there, though, within the uptime may go down or I may have an issue that is not NAS related. So that may have been like thinking like NFS mounts.
So why what made you want to put applications there versus just NFS mounts?
Techno Tim
Yeah. So NFS mounts are great. You can get into some trouble with NFS mounts like SQLite.
Adam Stacoviak
\[50:34\] Yeah. That's terrible.
Techno Tim
But— now I don't know. I just— you need to have storage, you know, for like a database application, you want to be closer to the actual storage for sure.
Adam Stacoviak
Yeah.
Techno Tim
Yeah. And it's not even just the latency piece. It's the locks and everything like that. Like NFS just doesn't handle it so well. Like the latency I can kind of get over. But like SQLite in general, you'll have these locks that are like locked that you can't unlock and you'll get corruption and stuff like that.
But NFS mounts were great. Yeah. I mean, you got to figure out permissions and stuff like that. You know, and I went down that route, you know, too for a while. But then it's like I have to back up. I have to like, you know, take care of applications over here, you know, set up all those mounts and do all that stuff. And then also still, you know, care and feeding for that— NFS mounts and snapshots and keep that connection up. And again, you're back to, you know, you've just, you know, doubled your chances of downtime, you know, two— you know, it takes two, right? It takes two to make one.
So, you know, and again, like I've gone back and forth with this like so many times, you know, it's like this whole generalized versus specialized, you know, that you see all over IT and, you know, enterprise in general. I've generalized and specialized my server so many times that like I think I'm ahead of like corporate entities that do this with their employees.
And so I've specialized and generalized my NAS so many times with so many different things. But right now, I'm landing on this. And I think a lot of it has to do with two things. Well, actually three— one, TrueNAS ditched Kubernetes, went back to plain Docker, which was awesome, because I never would have done it if they were still running the whole Kubernetes bit. But they went back to just standard containers, quote unquote standard containers, you know, Docker images, Docker containers, like I should say.
Two, then I'm able to do it with the YAML because I'm not going to fill out their forms. Like if they ever take that away, I'm bailing, I'm going to find something else to do because just not me, like I don't want to fill out a form, you know, to be able to, you know, put in my environment variables, like with the .env file, I copy and paste them in there, like why should I have to do that in a form? I get why they exist because people aren't developers. But that's not how I want to manage my containers. A lot of that has to do with Kubernetes too.
And then, you know, the other piece is this whole like hybrid— well, it has a lot to do with what you just said is like, hey, you have this beast of a machine just sitting there doing nothing, you know. I feel like it's that meme where that guy's like poking that thing with the stick and he's like, do something, you know, that's kind of how I feel if my NAS is like, you know, watch the metrics.
Adam Stacoviak
Yeah, dude, tell me about it. It's funny. You say— it's funny to say that because now you should see my metrics because now I post all my app metrics on the screen like you can put little widgets, you know. My traffic, reverse proxy, you know, five or six megabytes per second, you know, which is, you know, not a lot, but you think like that's more than every day.
And then like, you know, I look at my— I can see my MariaDB database, you know, going— it's doing, you know, three, four hundred megs per second of queries and stuff like that. I'm like, yeah, this thing is like, it's humming right now. You know, if I look at the CPU differences, you know, you know, it's only a couple percent, you know, more than it was before. And so I probably have 50, 60 containers running on there.
And again, like the reason why I ended up doing this is because I figured out a way with ZFS to create this hybrid pool. I mean, people in production probably won't do this. I think it's cool, but, you know, to make my hard drives, normal hard drives, you know, be as performant and as responsive as NVMe. So not only the speed that you get, but as responsive. And that's— again, like I've layered in NVMe to handle all of the quick writes, quick files, quick access. And for anything else, it's going into RAM, which is another huge case of like, yeah, I want to store my applications on something that has tons of RAM. And if it's storing, you know, bits and blocks in ARC, which is, you know, ZFS's RAM cache. If it's storing that stuff in RAM, yeah, do it. I want it to. So I'm like getting like the super performant, you know, apps that are mostly reading out of RAM. If they aren't, then they'll read out of NVMe storage. And then worst case scenario, they read from a slow disk, which is worst case scenario, because I have all those tiers.
Can you tease with your— with your RAM, your vdev is— which is NVMe, and then your disks? What's— I think you're probably a person, some of your future videos, but—
Techno Tim
Oh, no, this is— no, I've done videos on this too. So my— so I have striped vdevs. So let's back up. So for bulk storage, I have 10, 14 terabyte hard drives. And of those 10, 14 terabyte hard drives, I'm doing striped vdevs or kind of mirrored pairs, where I have two that are in a pair that are mirrored. And so, which means I have 50% loss of my data. So you got seven terabytes essentially?
Adam Stacoviak
Yes. Per pair is seven terabytes. Yep, that's exactly right.
Techno Tim
And so, but I do that for two reasons. One is because you can, in mirrored pairs, you can stripe them across. So you're basically kind of getting like a RAID 10. I don't want to say a RAID 10. Think of it like a RAID 10 where, you know, you have these pairs, but the data is striped across. So you get the performance on reads and you get performance on writes, which is good for me when I add videos because, you know, I have a bunch of sequential reads and writes and I don't know where they're going to be.
But on top of that, you can also expand by pairs. And so, traditional ZFS, it's super complicated in expanding. And I know that they've been adding features to be able to expand vdevs and do all this stuff. But, you know, when I started my array years ago, I realized that, you know, buying two drives at a time is a lot cheaper than replacing all four drives with four different size drives just so I can get a bigger pool. And so, in the traditional sense with ZFS, it's kind of what you have to do is you got to plan up front and buy up front, but doing pairs, let me build incrementally and buy incrementally.
So anyways, that's my— that's a nice slow disk array. Then I've done this thing called special vdevs, which is basically say, hey, all of the metadata about that data, instead of storing it on the pool itself, move it off onto super fast NVMe drives. So if you end up in a folder, like I have some folders, I have thousands of files, you know, that can take a really long time for it to parse that metadata and retrieve that metadata because it lives on slow spinning disks. Well, if you have a special vdev, you move it off of there so I can look it up on NVMe.
Then on top of that, another thing you can do with special vdevs is you can say, oh, by the way, don't just store my metadata there. If you find any files that are, I don't know, below 64K, just put them on there too.
Adam Stacoviak
That's pretty cool.
Techno Tim
So if you think about it, like now— now any small file, which usually takes a little while to find on spinning disks, is now stored on there too. And so it's like I've just given my whole entire array a huge boost because now most of the stuff is handling— happening on NVMe.
There's one huge caveat to that whole thing. If you lose that special vdev, you lose all your data.
Adam Stacoviak
But I'm listening.
Techno Tim
Yeah. Yeah. Yeah. So you lose all your data.
Adam Stacoviak
You're definitely cowboying this. I mean, yeah, but you know what you're doing.
Techno Tim
Yeah, I've done the same thing— mirrored vdevs. So I have four NVMe drives, you know, and so two would have to die for me to lose all my data. Right. But then on top of that, you should have a backup, right? And so, you know, I am definitely going cowboy because I want the performance, but I think that's a— I'm being a careful cowboy. I don't know if that's possible.
Adam Stacoviak
But you got spurs on, man.
Techno Tim
That's right. You got your slingers.
Adam Stacoviak
Right. But my slingers are on safety. How about that? You know, where, you know, I'm still, you know, taking precaution. I'm still building in redundancies.
Techno Tim
If you're a rock with one NVMe, I was going to call foul on, but you've got four, of course.
Adam Stacoviak
That's right.
Techno Tim
True NAS fashion. So not only four, but you got them in pairs. So you can lose two of the NVMe. Now that— you also go with different brands, buy them from different places, things like that too, or just buy a batch?
Adam Stacoviak
So I— on top of that, there's these old Intel Optane drives. So Intel used to make these Optane drives. They're ridiculously fast. And they have like the lowest latency ever. Intel stopped making them.
Techno Tim
Yeah.
Adam Stacoviak
But these things will like outlive Earth. You know, like these things, like the read and write performance you get on them and the longevity and how many times it can read and write is like ridiculously long that I ended up buying four of them because of how fast and because of how
Techno Tim
\[60:01\] Responsive and because of how long they're supposed to live. So no, I bought two from, I think, Newegg. And then two more from Amazon, like later on. They're hard to find, but they're still the best drive for that specific use case. And so when those die, what I'm going to do is just replace them with Samsung, you know, consumer grade.
Adam Stacoviak
Yeah. I hope they never die only because they're the fastest thing out there. They blow away any NVMe that's on the market. How did you get to get four NVMes in this world? Because that's different.
Techno Tim
It is different. So I use an adapter, you know, PCI Express card. So traditional NVMes, this is going to get a little complicated too, but NVMes want to use four lanes of PCI Express where they can talk directly to the CPU or talk directly to something. I can't remember. I'm not a true infrastructure person. I just play one on YouTube.
Adam Stacoviak
So the mask is off.
Techno Tim
That is right. Hey, I coined the term infrastructure as a hobby. That's kind of my thing. IAH or something. That's what I call homelab because, you know, my actual career is, you know, I'm a software developer. Infrastructure is my hobby, but I love it.
But anyways, NVMes can use four PCI Express lanes. And in your motherboard, typically you'll have one big slot that's 16X, you know, PCI Express lanes. You have to buy a card that can actually split that out into four individual lanes. And then you can address all four individually. Right. There's also this thing called bifurcation. And so you need to make sure that your motherboard can do bifurcation. And what bifurcation is is exactly what I just talked about. It's able to split that 16X card up into four individual lanes, four 4X lanes. And your motherboard has to support it and the card has to support it.
Server motherboards, generally speaking, like Super Micro ones, a lot of them do. Desktop consumer boards generally don't.
Adam Stacoviak
Yeah.
Techno Tim
And so if you're going to do it on a board that doesn't support bifurcation, then you can buy a card which is really expensive that can do it for you on the card. And those cards are, I don't know, four, five hundred bucks, maybe even more. But if you get lucky and you have a server grade motherboard that does it, that's where you want to do it. So anyways, that's how I get four NVMes in one PCI Express 16X slot. And they're all getting 4X bandwidth for each.
Adam Stacoviak
So that's four of your lanes for your GPU. That's right.
Techno Tim
Yeah. So I have a lane for my GPU.
Adam Stacoviak
GPUs in the primary probably, right? Yeah. And maybe one of your secondaries which are bifurcating. You probably have a workstation or a server grade motherboard. I'm assuming you've got that ability.
Techno Tim
Yeah. Workstations are great. Workstation motherboards are a great hack to not have to go server grade and not go consumer grade. You kind of get that middle ground. You're still in the 600 to 800 bucks, maybe a thousand dollars for the motherboard. But that's because of the world we're in right now. But you do get the capability and you usually get ECC RAM availability as well versus non-ECC RAM, which you don't really need need in an SSD world. But you should have if you want peace of mind, I guess.
Adam Stacoviak
Yeah.
Techno Tim
Yeah, a lot of people will go back and forth whether you need it or not. People will say, well, if you care about your data, you do because if the memory gets corrupt, that's where it writes first. It's a source of truth.
Adam Stacoviak
Yeah.
Techno Tim
So in ZFS, it's less important. I'll say that because of how it checks and how it does its CRC checks on the memory and can verify the data. Some people still swear by it. Like, they'll say if you don't use ECC, you might as well not even do stuff. People go to the extreme. I'm not there, but I'm glad I have it. How about that?
Adam Stacoviak
\[64:05\] That's an interesting world. So I mean, when you think about a problem like this, so let's zoom out to, you know, not scare the homelabbers away, those curious folks. When you think about this problem, do you get out your big old whiteboard? When you're spec'ing this world out, how do you think about it? Because you're a YouTuber too. So you think about it probably in the story arc. And you also think about as a technologist, how do you map this and plan this and test this world? How do you do that?
Techno Tim
You know, first and foremost, like I've been doing this for a long time, even before YouTube. Like I've had a server in my basement since I don't even know. I'll probably date myself, probably like 2004 or 5, going back to like this old piece of junk that when I was in tech support, I asked my manager if I could take it home so I could learn about Linux and learn about Active Directory. And he looked at me like I was crazy because the thing was 10 years old already and he told me, yeah, you can take it, just take the hard drives out because it had data on it.
And so I've been doing this for a long time. I've had a server in my basement for a long time. So I generally think about, you know, what do I want to do? What capabilities do I want to have? Generally speaking, I want to provide file services. So I want a NAS to be able to do shares and store my files on. I don't want to store my files obviously on my system. I want to store them on my NAS because then that gets backed up.
I want to have some compute. And it takes very little compute to do most of the things you want to do. A lot of people will say, get server grade. But I could spin up this i3 right over here that's, I don't know, from five years ago. And this will like destroy any self-hosted container I throw at it and like laugh at it. So you don't need much compute at all. RAM is great, but again, like you don't need tons of RAM.
And so I just try to think of what services do I want to provide? For me, it's storage. For me, we record a lot of TV and stream stuff at home. So I think about Plex. Plex then brings in a video card. If I want to transcode, most of the stuff I can direct stream here, but when we travel or whatever, iPads, phones, they want to transcode. So having a video card in there that can do that transcoding for me on the fly is good. That's a decision point too. You could get by with Intel stuff, but then I start thinking, well, I want to run models at home too. So I kind of want to have shared infrastructure. So I want my video card to be shared too. So that's what I do now. Share my video card with Plex, with Ollama, with the stuff we just talked about with Paperless AI and stuff like that. Technically, that's going through Ollama. But even some of my, you know, I do some transcoding for some of my video cameras too.
So yeah, I try to find a video card that'll work in all those scenarios. And so I never want people to think like, oh, I need to go and buy this big thing before I can start homelabbing. It's never like that. Like if you have an old PC in your basement, use that first. Figure out what you want to do with it. Use it as is. If it had Windows on it from 10 years ago, wipe it, put Linux on it. And if you're scared of Linux, that's fine. You can put Windows on it. But I would say just try Linux, because you're going to find things are a lot more compatible.
Adam Stacoviak
Yeah, for sure.
Techno Tim
And just try it. I mean, you might find it sucks to be a sysadmin at home. I enjoy it. When things go wrong, my wife says something doesn't work. That's when I'm like, all right, something broke. I have a job.
Adam Stacoviak
That's right. That's right. This is why I'm back. One step from 10.
Techno Tim
Oh, that's right. That's right. But you know, I enjoy doing it. A lot of people joke on, a lot of my comments on my YouTube videos are, bro has a full-time job at home.
And it kind of is right, to an extent. But if you build things right, you've got things working and you know how they work and you document it in case things go wrong. And a little help for me, I like a lot of things are hands off. So you're focusing on the next thing you want to do.
Adam Stacoviak
Yeah.
Techno Tim
So again, like I don't have a recipe of how I do things. I just try to think of what are my base services I want to have? It's always going to be storage, streaming, and some compute, and some kind of transcode capability. And from there, I just try out a whole bunch of containers. I treat containers now, just think of like apps on your phone. I can spin them up with that quick. I can spin them up probably about five, 10 minutes, all said and done working with a proper certificate. But they're like apps on my phone. If I want to try an app, I try the app. Does it do what I want? Yes, it does what I want. No, it doesn't. There's this other app that I should try, and so I'll go try that. And that's kind of the world I live in now. My self-hosted services are basically apps at home for my home to use. When I say at home, I mean pretty much me, but there are some that my wife uses.
Adam Stacoviak
Yeah. And as long as I have a platform to do all of that, it really doesn't matter what parts I used because you can get by with so very little nowadays.
Techno Tim
They really can.
Adam Stacoviak
I don't think you have to go so big. I mean, like this is where it leads, okay? It's like, I don't know if you're a golfer, Tim, but that's what golf folks say as well. It's like, hey, you get invited to play golf. You're like, no, I don't want to go. And then somewhere during that first round, you're like, oh my gosh, this is the best game ever. I'm going to go buy all the clubs I could possibly buy. And golf tech, like any tech, is just limitless really what they can fine tune and dial in. And so if you've ever become a golfer, you start with like this small itch. And the next thing you know, you spend $10,000 your first year in some way, shape, or form. I'm being facetious, but you know, golf rounds aren't cheap, golf trips with friends aren't cheap, golf clubs are not cheap. And then you have to have special clothes or you want to have special clothes because hey, why not dress the part?
I kind of feel like that's the same thing with homelabs. Like you can begin with, like I did, I can remember the day when I got my Raspberry Pi. I can remember the day when I spun up the TrueNAS. Or actually it wasn't TrueNAS, it was just like whatever 45 Drives sent me because way, way, way back in the day, I want to say probably six or seven years ago, maybe eight years ago, they sent me an AV15 to try out and they said, hey, no, you know, you can just keep it, it's just yours just to have and play with. And this is when they were first launching that line to homelabbers or what would become homelabbers. And I was like, really? And they're like, yeah, we don't want it back. It costs way too much to ship it back to us.
And at that time, I guess hardware was just cheap enough that they're like, yeah, we don't even want it back. Just use it and enjoy it and tell people about your experience about it. I'm like, okay, cool. And so that today is my TrueNAS box. And it's got the Xeon Silver, I think 4012, I believe, the CPU in there, if I recall correctly, or 4212 maybe.
But I began, that was gifted to me. I didn't know what I was doing with it at first. But then I began on a Raspberry Pi and sort of the experiment. And so that's where I began with everything I ran on my Raspberry Pi. Now I wouldn't, because my, it's not my needs, like what I actually need for computers has grown. But my desire to play with bigger things has grown, you know, the playground. I can't just have the carousel on it. I got to have the slides. I got to have the swing. I got to have the rock climb. I've got to have the, you know, all the things. So my playground for homelab has just grown a little bit.
I'm curious though, when we look at maybe this potential dichotomy between TrueNAS and your desire to put all things there, which maybe somebody out there's having similar feelings. And then this word of Proxmox. I feel like I'm with you. I kind of want my TrueNAS box to do everything. But TrueNAS, the software, is not quite there yet. Where do you see, do you have this vision or purview into that world where you can see TrueNAS being this all-in-one big box software? Because it traditionally has been great for what it is, you know, ZFS, storage pools, a little bit of applications you need. But not a tremendous amount. I kind of want my Proxmox and my TrueNAS in one box. Is that how you feel?
Techno Tim
Yeah, I do. And you kind of can. So you could virtualize TrueNAS. So let's take your thing for instance.
Adam Stacoviak
I've done it for years. And it's totally fine. It's totally fine. It's totally fine.
Techno Tim
It's totally fine.
Adam Stacoviak
But don't you have to like map them to like weird drive IDs and stuff? Isn't it like weird in the uptime, I suppose if you had some major issues with your drives?
Techno Tim
No, so it's easier than you think. And so let's take for instance, you have that TrueNAS box. It's running TrueNAS. Just pretend you want to run Proxmox on it now. But then you create a TrueNAS virtual machine. And you give that TrueNAS virtual machine, you pass through the hardware of that HBA controller. You give it the whole piece of hardware. You say, nope, this hardware controller, hard drive controller, HBA is now assigned to this virtual machine. And what that does is now the virtual machine has direct access to all of those disks. There's no IDs to pass through because it thinks it's the true owner of the disks. And then you do that. Then your life is good. And then whatever.
Adam Stacoviak
That's one way to do it.
Techno Tim
That is one way to do it. But I agree. So let me put it this way. And this is not a dig to either product. They're better at some things. And so TrueNAS is leading as a NAS. They're leading with, I'm a NAS, but I could also do apps.
And I can also do virtualization, not that great. And that's my own opinion. But it can do virtualization. It's just not that great at all because it's not a hypervisor first. It's leading with that.
And when you think about Proxmox, Proxmox is like, hey, I'm a hypervisor first. Like, that's what we do. We do virtualization. You could install apps like LXCs, although I'll get into that. But you could install a whole bunch of apps and run them on the machine itself. But if you think about it as a NAS, not a great NAS experience. Sure, you can install a Samba server, assign it some pool, and then do all of the Samba config in the CLI and all that stuff. Figure out permissions. You can do all that. They're both capable of doing each other's job. I'm kind of like, I want something in the middle.
And there is HexOS, which is kind of coming. But that's kind of like some joint venture between some people and Linus. And TrueNAS or iXsystems creating this more, I guess, consumer-friendly version of TrueNAS. There's that coming. But I played with the beta. Looks pretty cool. It's going to be a facade on top of TrueNAS because they're going to use TrueNAS APIs and it's really TrueNAS under the hood. But that's something different.
I want the best of both worlds. I wish they would, I mean, they're not going to do this. But if I get to design the perfect NAS at home for me, it would be a TrueNAS-like experience for the NAS piece. And maybe even for the application piece. But give me the virtualization capabilities that Proxmox has, and the networking capabilities that Proxmox has. Or if I couldn't do that,
I would love for Proxmox just to run Docker containers. They are so, I don't know what's going on there. They're so against like running Docker on the host. And I know you could shim it in and do it yourself. But they've even got to the point where now they're like converting OCI containers, which, you know, Docker containers quote unquote, to LXCs, to then run as an LXC because they don't want to run true OCI containers like Docker. Like, I don't get it. Like maybe, I mean, I don't know, Docker CE. Does it cost money to... I mean, I don't know. I know there's been a lot of change in the Docker world in the last six years. I mean, they were almost a dead company.
Adam Stacoviak
And then, yeah, then they were revived. It was Docker 2.0. I talked to their CEO on a podcast here, in the same podcast you're on, a few years back. It's a fun conversation about Docker 2.0 where they went from zero revenue to revenue. And I mean, at some point, you have to protect your moat. Really, you do as a company.
Techno Tim
Yeah, I imagine it's probably licensing. It's probably something there. But as a user like you are, I want that world to marry. So figure it out.
Adam Stacoviak
Yeah, yeah.
Techno Tim
And Docker's done, like now that they're, you know, they have Model Runner. They're doing all this Scout stuff and scanning containers and really building up their Docker Desktop. And I'm just focusing on the Docker CE part. And maybe there is licensing around, you know, maybe you can't ship this with your own product.
But let's take that all the way out and just go down to like containerd. Like just get OCI images, Podman. I mean something, like at the end of the day, I don't want to run more...
Adam Stacoviak
Like more license friendly. Yeah.
Techno Tim
And so at the end of the day, I wish I could run OCI containers as first class citizens on Proxmox. And I, again, I don't know the reasoning behind it. I honestly feel like it has something to do with more of their strategy and how they're trying to be highly available and LXCs highly available. Though that, I don't think is there yet. And VMs highly available. I just don't think they want containers spinning up on the host itself. I get it. You can get around that. But yeah.
Adam Stacoviak
Do you mess with Fly.io, by any chance? The, we're all done in like prod Fly.io?
Techno Tim
No, I haven't messed with it.
Adam Stacoviak
It's the, I mean, if you love containers, then you'll love Fly. I mean, Fly is what we host changelog.com on. They're a partner of ours. We love them obviously. This is not technically paid. I'm not paid to love them. We just love them anyways. Fly is like that. You're running containers, right? You're running a container in production.
Techno Tim
Yeah.
Adam Stacoviak
It's Firecracker VMs. I'm not familiar with everything behind it, but Fly machines essentially, they spin up very fast. They run down very fast.
Now, I would love to have a version of Fly in my homelab. And it sounds like that's what you're describing there, which is I want an OCI container. I want as close to bare metal as I can. I don't have to spin up an Ubuntu VM to then throw Docker on to then launch my Docker container. I would like to just have the entire system be container friendly. It sounds like you're saying.
Techno Tim
Yeah. Yeah. I mean, similar to Cloud Run on Google or any of these things. You give it a manifest. You spin it up and it's running container only. And I mean, that's kind of what I'm getting with TrueNAS right now. I feed it a YAML. I got to create a dataset for it. I got to tell it where to put the data it's going to use. Then I use a Docker Compose file and I'm done. So that's kind of what I'm getting with TrueNAS now. I just wish Proxmox would just build that into their CLI or UI or whatever. So that people just don't have to do this. Well, I'm going to run an LXC and I'm going to run it as root. So then I can install Docker to then run an OCI container inside of this LXC container or pay the virtualization tax and run it inside of a VM. I just want as bare metal as possible. And I don't know. I'm sure Proxmox has a lot of reasons why they don't do it. Probably has something to do again with their strategy. It doesn't fit in there, but I just don't see how you could ignore OCI containers in general.
Adam Stacoviak
Yeah. Yeah, I agree with that. I do agree with that.
Well, the closest I've been able to come is the CLI I built, which in this case, it's got the cloud image, an Ubuntu or even a Fedora cloud image on the server already, which it uses that as its base, like you would. And so rather than going that whole route of creating the template and including the template to create a new machine, the CLI does a version of that through automation. Got you. And you're able to, through cloud-init, you're able to define the network, right? You've got all that there that uses the network, the SSH keys. And then everything else is the Proxmox API, which I really wish Proxmox and TrueNAS did a better job of documenting their API. It's just not, it's good documentation, but I just feel like they don't treat it like a first class citizen.
And the kind of developer I am is I want to play with your API. I want to build my tools on top of your system, not be forced to go to your web. I want to web you out. Like even with TrueNAS, like you're the same. I don't want to fill out a form to spin up a new thing. I would much rather automate through whatever layer it is, whether it's me or an agent, some sort of CLI. And then if the agent's using it, then it can just easily use the CLI I built. But I want to be able to automate those things on those kinds of systems.
And the closest I've come is that exactly, like PMX new VM, and then specify all these things, send it a template, which is super easy. And those are YAML files. Those are YAML files, like two or three YAML files to define a couple of things. And you're off to the races, you can define a minimal
Ubuntu brand new machine. Now, you could define it to be bigger, but I just have found that it's just more easy to layer on a post-install bash script than try to script it all. Like you get into Ansible Land, it was just really nasty. It got really error prone. So I was like, you know what? Forget all that. I just want to define a base VM. A base VM that is blessed with an IP address, with the RAM I want, with the CPU I want, with the disk I want, and protect it. It's got a dash P on it, which means if I try to delete it accidentally or my agent tries to, it's got to go through this whole entire dance and send across like documentation and social security numbers to fight that to delete a VM. You can't just accidentally delete a VM. You got to do some things.
But it's pretty, I mean, like literally within, so your 15 minute scenario that you defined earlier with a new container, less than 60 seconds, Tim.
Techno Tim
Yeah.
Adam Stacoviak
Maybe even 30. Maybe a minute to get the IP address back. It's got to like launch the VM, you know, the updates, right, from Ubuntu whatever, which takes time, and then actually launch the actual machine itself and then get an SSH key. That's the thing that takes time is the boot, the update, and then finally QEMU giving you that IP address back. But like that's all like instant.
Techno Tim
Yeah, that's awesome. No, it sounds awesome. I'd love to check it out. Cause yeah, again, when I said 10, 15 minutes, that's me figuring out, okay, data mounts, environment variables, like that's me, that's like research I had to do anyways. If I knew it, yeah, it still takes me five minutes, but that's not 60 seconds. I should look at doing that with TrueNAS too. It's just automating the show, what I'm doing behind the scenes.
Adam Stacoviak
It's not ready. It's ready to be used if you don't mind warts. It's not open source ready yet, because I think people have a different expectation of what that might be. I want to do one more rewrite because I now have more clarity on what it is, but I think we could layer on what I'm already doing with the idea of already saying, okay, there's TrueNAS on the system here. And there's a separate layer that you can spin up that second, that same VM, but then also say there's mounts elsewhere, there's NFS mounts elsewhere, whatever you want to do, to do your world, which you're talking about there. So yeah, I'm curious why you don't use Ansible though, because I have my, I have my post Ansible, like I have bash.
Techno Tim
Agents like bash. I just mean like, no, I know I get it.
Adam Stacoviak
But like I have my Ansible Playbook for new VM. It has 50, 60 tasks that are all there.
Techno Tim
Yeah.
Adam Stacoviak
And it does it intelligently. Like, hey, if I say stop the firewall service, well, if it's not running, it's not going to stop it. And hey, if I tell it to install this one package, it knows that doesn't run on this type of machine. So it's not going to try. Like I have my Ansible Playbook where I just click go. Anytime I create a brand new machine, I have a standard Playbook I run. And it's going to apply updates, update, reboot, install Zsh, configure Zsh with Robbie Russell, because that's what I like. It goes through the whole shebang of like, this is my standard VM. And I just don't even pay attention to it. It probably takes a minute or two throughout all the reboots and applying and installing packages. But after that, then it's ready for production.
Techno Tim
Yeah. So I think, I'll defend that by saying, I have never been, not for any reason, an Ansible guy. I just never really got into it. I didn't understand the world. I know what it does and how people use it, I get all that, client-server, I get the recipes, I'm not foolish, but I never got into it to need infrastructure automation.
And the lingua franca of agents is Markdown files and bash. Bash is everywhere. You have to add on Ansible on your client or on your server somewhere. And it's baggage.
Adam Stacoviak
It is for sure native Linux, right? So it may work for you, because that's been your history, right? But in that world, when you're trying to automate that kind of thing, I just tell the LLM, hey, I'm launching a new instance of Pi-hole. And here's the specifications for it. It will write a script. That script can be idempotent. And it will rerun it. And it's just as good, if not probably better and faster than Ansible.
Techno Tim
Oh, yeah, yeah, yeah, I'm not, I'm just thinking about like if you're building, if you're building a CLI, like at some point it has to be able to scale to different things. And so are you recompiling the CLI every time it needs to install one more thing or just writing a new script? How does that work?
Adam Stacoviak
Well, it has versions. It's written in Go. So it's got versions. And so if there's new features or a patch release, I'll patch release it and throw a new version out there, which you can then run PMX update and it will get the new version of it.
Techno Tim
I just mean like the source of truth for your list of things. Say like one day you want to install Zsh.
Adam Stacoviak
Like here's the beautiful thing, Tim. Those aren't in there.
Techno Tim
Yeah, that's what I was going to say.
Adam Stacoviak
Those things are in user land. You can define them. And so that's why I want to rewrite it because I sort of like, you know, I haven't been doing this as a day job. This is my little scratch, my little itch. And I really haven't, I probably haven't changed the code in three months, honestly. It does what I needed it to do. This Adam wants to change it to release it and make it better.
But the world I wanted to build was this really interesting CLI, but most of those things live in user land. And so have a separate repository that the world can define their own minimal Ubuntu, minimal Fedora, minimal Debian. You name it whatever you want to do. And all the patterns are there. And all you got to do is clone the repository, put it in a certain place, update your config. And when you run PMX new VM or whatever it is, and you can do dash dash template or dash dash package. And you point and you say the name, well, it knows that because config says all of your templates are over here. So you won't have to recompile the Go binary, the Go CLI to do that. All that will live in user land.
Originally, they did. Okay, I was a fool. Okay, I put those things in there. And I was going to make it to extract out. So I wanted to give people a good bootstrap. Well, then I learned, well, that's just probably not the best way. And so it's better to have a user land repository that people can commit to and update. And it can be pointed to by people. And sure, it's one more step to clone that repo down and put it in a config. But I feel like the trade off is better long term.
So that's where I'm at. I haven't gotten to the point where now I can put a lot of those minimal templates or even more expressive ones that use Ansible, which is totally possible. I just didn't want to do it. And I just wanted to get a base image and at least my agent on that image and be like, make the world. And it goes. And it makes the world.
Techno Tim
Yeah, no, that sounds awesome. No, it sounds awesome. I'd love to check it out. Cause yeah, again, it was like, I did Ansible only because that was way better than running CLI commands over and over and over and over or even writing one big bash script. Now that's changed because LLMs love bash. So now things have changed. But yeah, I should definitely revisit it. Yeah, I'd love an agent.
Adam Stacoviak
Yeah. Yeah, at some point, I'd love to hook up my Open WebUI, so my internal chat that I use with Ollama, and build, you know, use OpenAgent or something, is that new OpenAgent that's out? OpenCode.
Techno Tim
Yeah.
Adam Stacoviak
\[90:01\] Open Code. Sorry. Yeah, Open Code. Run that agent and tell that agent to do stuff in Proxmox for me. All via chat.
Techno Tim
Yes.
Adam Stacoviak
Like I just don't even want to run a CLI. That's what I'm saying. Like, I don't even want to run the CLI. Sure I could do that and expand the variables and figure out what it can't do. Well, the problem with it is that Proxmox has some challenges. Like it would have to navigate... The reason why I went the CLI route was one, I thought I wanted it for me. Now the whole world shifted to be agent first. Now I want to build an MCP. I want to build an MCP for Proxmox that the MCP understands the Proxmox API. So, hey, MCP talk. You speak Proxmox API. LLM, you speak human, but you also speak to the MCP. You know what I mean? So, now it's like, I speak human to an LLM that speaks MCP to that LLM kind of API to Proxmox and just do the stuff.
Techno Tim
You'll get to a limitation at some point though.
Adam Stacoviak
First layer in the middle there. First CLI is kind of the important piece. And then you layer on the MCP server on top of your CLI. It's an MCP. You can speak native Proxmox as necessary, like if there's an API using native API or the things that it doesn't do that you've taken from 15 steps to one command. Yeah. And you've got tests against it. Well, your CLI, in addition to the Proxmox API with your MCP server and your agent in Open Code or in Claude, well, that's the beautiful world.
Techno Tim
Yeah, man. Yeah, that sounds awesome. That sounds awesome for sure. Because yeah, the Proxmox binary, whatever it is, PXM, that can do so much. I mean, that's how people are doing even their Ansible scripts right now. They're like, you know, shelling into it, running a PXM command. So, yeah, it sounds awesome for an MCP too to be able to say that. But then that's a lot of words for me to type. You know, I'm getting lazy developer. Like it, rather like you have a CLI and...
\[92:02\] ...run a command. Like, do I want to sit here and describe, you know, what to build? No, just build a standard VM. Let me know when it's done. Call it this.
Adam Stacoviak
All right. Well, the determinism there's the challenge. You can do that. But you know, the reason why MCP came into play was because LLMs are non-deterministic. And you can say that, but every time you might get a different version of it, or if you're using Sonnet versus Opus now and like Opus 4.5 is the default, you know, per Anthropic and Claude. They want you to use Opus 4.5, which is non-deterministic. It may know that and you get great results every time. It's not the same path to Rome. The MCP server is what helps you determine and create determinism or even skills. So you layer on skills, MCP and CLI with a decent API, which is why we love Proxmox, to just bolster that. Make it more expressive, give me more in there. Better documented. Because if you give us those tools, we'll use your thing more. And I got to imagine if I'm an enterprise buying a support license, you know, that's how they make their money, right? Is how Proxmox makes money on Proxmox? Is support licenses?
Techno Tim
Yeah, that's how most open source, you know, are making it. Now they're starting to do premium features. It seems like that's like a huge trend now. It's like, oh, you can self-host the normal version, but then premium features. You know, it's a combination of I'm either going to be your SaaS, I'm going to give you extra features, or and/or I'm going to be your support.
Adam Stacoviak
Yeah. That seems how most open source projects are monetizing now.
Techno Tim
In large installations, I can see that.
Adam Stacoviak
For me, I would buy a support license as a means to sustain and just to get rid of that thing that pops up every single time, really. I would just be like, I would honestly give Proxmox 100 bucks a year just to get rid of that, you know, realistically.
Techno Tim
Yeah, there's a, yeah, I agree. I asked them one time, hey...
\[94:02\] ...do you have a homelab license? You know, they have a cheaper version of the homelab license, you know, that I could use so I could get, you know, legit updates versus bleeding edge updates because their price per core or whatever it was was, you know, kind of cost prohibitive for homelab persons. Like five, six hundred bucks, might be more now, per year. And I'm like, yeah, that's a lot. Not a lot for enterprise, but a lot for Tim-erprise, you know, so. And they were like, don't worry about it. You know, they were like, don't worry about it. Just, you know, use our latest updates.
Adam Stacoviak
Anyways, I say that because you can get rid of that nag screen really easy.
Techno Tim
You can't. Yeah, oh man.
Adam Stacoviak
So since we're talking so much about Proxmox, have you heard about Proxmox Helper Scripts?
Techno Tim
I did a video on it.
Adam Stacoviak
It doesn't matter. You should check out Proxmox Helper Scripts.
Techno Tim
It's your channel, Tim, and I missed this.
Adam Stacoviak
That's all right. That's all right. No, it's out of the... Going back to algorithm. You know, it's fighting for ears and eyeballs and, you know, and Google says like, hey, this does not deserve ears, eyeballs. And plus like, dude, there's no way you can keep up. Anyways, that is a collection of tons of scripts, single liners, you can run to do anything you want to do on Proxmox. All written in bash. There's one that's like get rid of nag screen that you can run. But there's also like the default script you run is so good. It'll like get rid of nag screens, set it to, you know, remove the enterprise repositories. You know, it'll do everything you want to do as a homelabber, which is give me bleeding edge updates because that's all I can get. Turn off nag screens because I can't really afford a license and, you know, disable some other things because I'm never going to use it. So really cool, really cool repository. It was actually made by someone who passed away and he passed it on to the community. So really cool website and a really awesome group of people.
Techno Tim
What is it called again one more time?
Adam Stacoviak
Proxmox Helper Scripts.
Techno Tim
Okay.
Adam Stacoviak
\[96:06\] Might be proxmoxhelperscripts.com. I'm not sure. But if you check it out, it's so awesome. I mean, it's worth looking at. Proxmox Helper Scripts, almost like a github.io website. It is...
Techno Tim
Search that on the web.
Adam Stacoviak
We'll link up in the shows, of course. I know we're getting close to our time, Tim. I'm happy to talk to you, but I go to view scripts. I mean, anything. So they basically built almost like an app store for LXC containers, which is pretty cool. Like, hey, do you want to install... do you want to install Home Assistant as an LXC container on your Proxmox? Yes. Click one shell script, you're done. Do you want to install... No, yeah. Do you want Ollama with GPU enabled? Yes. Okay. Run the shell script.
Techno Tim
Yeah. Go to Proxmox VE Helper Scripts. It's community-scripts.github.io. And that's what they've done is they basically built almost like an app store for Proxmox. Because they're able to either create an LXC for it or create a VM for you. Like databases. Like, hey, do you want to run Postgres? Yes. Do you click this button? Run the shell script, you know, and it's cool, man.
Adam Stacoviak
Yeah. It's really awesome. I did a video on it. I had so much attention.
Techno Tim
Yeah, man. It deserves so much attention. And it's not building VMs. It's building LXCs for the most part because that's what you want, right? You don't need a full fat VM to run, you know, MariaDB. You just need MariaDB running, you know, with a little bit of storage and a little bit of RAM.
Adam Stacoviak
Yeah. Yeah. Like, think about setting up, you know, Bitwarden. How hard that is, you know, or Authelia, you know, this does it for you and, you know, very simple shell script.
Techno Tim
Well, all right, then. I'm glad I talked to you, Tim.
Adam Stacoviak
Yeah, man. Like, Home Assistant, like, you want to run Home Assistant in an LXC container...
\[98:07\] ...you just want to try it out. Click this script, two seconds later it's running. You don't like Home Assistant, delete it, you know.
Techno Tim
Are you using this for those... You're probably not using this for those things, though. It sounds like you're using it just for the simple things, though.
Adam Stacoviak
I am using it for some stuff. I used it for Pi-hole. Set up Pi-hole as an LXC container because I'm like, yeah, why not? Like, do I want to run, you know, sudo apt install and do all this stuff? No. Do I want to run as a VM? No. So, yeah, so I've done it for certain things, for certain things in my lab. Yep. Redis, did I do it for Redis? I might not have done it for Redis because I run Redis in a cluster. I do run Redis as an LXC. I do run it in a cluster, but I don't think I use their scripts. But any time I, you know, any time you want to test something out, I mean, this is almost faster than trying it with a Docker container only because, you know, they fill out all the defaults for you. So you go from, you know, you go from running the script to actually running.
The challenge here and the one thing, the big takeaway is here is you don't run this script in a shell prompt. It's kind of weird. You have to run it from the Proxmox terminal web.
Techno Tim
Does it make sense?
Adam Stacoviak
You got to log into your web UI.
Techno Tim
Yeah, and open terminal.
Adam Stacoviak
Things might have changed since then. But if you run it in your shell, you know, you're not executing it, I think, as the right person. So, and that might have changed, there might... they might have made changes. But I know for sure, if you run it in the terminal from the web, it works. So yeah, set up, set up Grafana, set up Prometheus. It's all right there.
Techno Tim
Wow. Yeah, it's pretty awesome. Let us pretty quiet. I didn't even know this existed. I've been using Proxmox forever and learned something new every single day. What's left? What's left on your thought list? I know you made a list.
Adam Stacoviak
I did make a list.
Techno Tim
On your list, it's like...
\[100:08\] ...man, we can't end the show without talking about this.
Adam Stacoviak
No, it's my... A lot of it was just, you know, I don't want to say complaints, but you know, just the state we're in with how expensive things are. Nothing's available. But software is getting better. So that's a huge upside. And you know, people are making awesome things using a lot of tools and getting ideas outside their heads. A lot of awesome self-hosted stuff in the open source world. And it's... I love it. I love it. I went from this... I feel like I went from this drought of software. I don't know, a couple of years ago where I'm like, yeah, you know, I've seen all the Docker containers people run at home. So now I'm like, oh my gosh, there's this whole like new world of people building these new containers of things I can run that I've never even heard of before, you know, based on AI maybe, maybe not that... that like I feel like software's fresh because now I can try all these apps again. So yeah, these are like for me, it's like server apps. Think of it like that.
So I think this year's going to be a wild year. I would definitely encourage you to unleash Claude on your UDM, whatever you might have. And just say like help me just examine my VLANs, examine my rules and my profiles. And it might be like, you know what Tim, you're pretty good. It might be like, you know what Tim, I can help you here. You know, it's probably going to say like, well, why do you have these duplicate rules? Like I... I... there are probably rules in there that I either never deleted. Like I am not a firewall rule expert. I'll be the first to say it's like, I try it until it works. You know what I mean? And like it's always guessed for me.
Techno Tim
Well, let me just say that I set up my VLANs based on your video. And it was upset with me. Just saying.
Adam Stacoviak
It's operator. Just saying. And I really didn't know much about things. I was like, you know, I've been told, hey, if you run a network, you should have VLANs because you want your kids to be on one thing, which I totally agree. You know, you want your guests to be on another and I totally agree with that too. And then everything else is on this...
\[102:11\] ...trusted network, which I totally agree with. But then all this intermingling, I'm like, well, just because, just because it's an NVIDIA Shield, should I put it on my IoT thing? And then the answer is no, the answer should be, it should be on trusted. It just needs access to too much. You know, and there's other things too like, it's the dumb things that you definitely don't want on your trusted network on the IoT VLAN.
So the three VLANs I'm for sure I want is trusted, kids, and IoT and guests.
Techno Tim
Yep. Yep. Yeah. That's kind of where I settled too. But then I have all of my networking equipment, you know, on a separate one. Like it's, you know, it's on the default one. My trusted is its own VLAN. I do have one more and then cameras, all my cameras go on one VLAN. But yes, I agree. You got to think about like, it's not necessarily what the device is. It's kind of the role that it plays and how much do you trust it?
Adam Stacoviak
Yeah. And so some people might say, yeah, put my HomePod on IoT. I don't, I don't want my HomePod. You know, that's IoT. Well, if you're... If you give it access to your schedule and you're telling it to turn on your lights in your home, I think you kind of trust that thing, you know, enough to put it on your trusted network. And so that's the way I kind of think of it now is like, what does it need access to? And how hard is it to cross the chasm, you know, like Home Assistant. Like, I do trust that thing, but I don't want to write a thousand firewall rules for it to go talk to everything in IoT. So I put it on the IoT because I don't want to do the opposite.
So yeah, even my Xbox for a while. I was like, die hard. Like, no, you are IoT. But I'm just like, you know, I put my password into the thing and I play games. It's going to go on my trusted network because I don't want to cross the network just to do other things, you know.
Techno Tim
I think it's got to worry about if that thing ever gets circumvented. That's the main thing is like if it gets...
Adam Stacoviak
Yeah, yeah, it's all about limiting your blast radius, limiting your blast radius and how comfortable you are with that, you know.
Techno Tim
Yeah. I think this year, though, is the year...
\[104:15\] ...of AI in the homelab. I know it has been already for me. And I do lots of stuff across machines with Claude. I don't just do it on the single machine I'm on. I'm not only using it to build software or to build little itches and scratches and stuff like that. It really is like the moment, for example, when PXM gives me that IP address back, I take that info report. It gives me, which I've designed to be agent friendly. In that case, I'm the CLI or I'm the API. I copy and paste it into the agent and I say, here you go. Here's your machine. And then it logs in because it's got my SSH key. And it's like, okay, sweet. It's a brand new, you know, base image of Ubuntu. Let me build your world here for you. Here's the Ansible group you want right here. It stores it in, you know, in Git in the repository. It may be a deploy file or deploy directory. And we always make it idempotent. You know, so that way, if you want to rerun or something like that, or if it needs a different one, maybe a post or a pre-install. Who knows what? But that thing has just been so cool to just unleash like that. So I think this year will probably be your year too of AI in your homelab. And that's kind of fun. New worlds, new capabilities, Tim.
Adam Stacoviak
Yeah, I definitely need to unleash some agents here and try some of that too. I've been, you know, I do run a ton of AI stuff and I've been, you know, doing a lot of LLM stuff, especially. But agents, yeah, would be cool to turn some stuff loose in my Proxmox cluster and just say, go build some stuff rather than using an Ansible Playbook. Just go for it.
Techno Tim
Rather than one year later, Tim, we should talk in three months. Your, my prescription, if I'm your doctor... doctor's to caveat here, okay, is your prescription is go unleash Claude on your homelab network and do some cool stuff and come back in three months and tell me some tales.
Adam Stacoviak
Okay.
Techno Tim
Because I guarantee you, I guarantee you, come back a whole different...
\[106:21\] ...Tim.
Adam Stacoviak
I bet, you know, it's just like, again, like a lot of people probably going through this, like, you know, I've had the tools that I've used and I've designed the tools and I know they work and I'm comfortable with them and I write CI/CD pipelines. But maybe I should just kind of just give that to AI and tell it the outcome I want and not worry so much about how it gets there. Especially on the low stake stuff, you know, if you just got this little thing you want to do, what's the harm? You wouldn't have read it anyways. Who cares about the code? Why do you need a code review?
Techno Tim
All you need is a code viewer. Your VS Code is now... that stands for code viewer now. I'm just kidding. I'm just kidding. That stands for continue, continue, continue. Right.
Adam Stacoviak
Yeah, exactly.
Techno Tim
Well, I mean, you can also automate a lot of that stuff too to be in YOLO mode. So a lot of people will say YOLO mode for some things. That's kind of Ralph Wiggum, that loop there. The Ralph loop is, if you, if this is your first time hearing about Tim, you're going to hear a lot more about it soon because it's the beginning of what's going to come when it comes to a well engineered, like the doesn't remove you as an engineer, you stop to be an engineer. So be a good engineer with a good to-do list or a good spec. And I only shroud that Ralph loop on it. And, you know, if the code works and it passed tests and these different things around security, well, then who cares really in the end? Is it the best idiomatic Go? I mean, I kind of do if I'm maintaining this thing, but if the agent's maintaining it, and I didn't have the software yesterday, I need it today and now it's here and it solves my problem. And I don't have the time to maintain it anyways. It's like did the tree fall in the woods and you hear it anyway? Like it's that whole thing, you know.
Adam Stacoviak
That's really better. Well, it kind of related to that. One thing I did learn too is, you know, after having AI write some code is having it write tests too is super important. I've noticed.
Techno Tim
Yes.
Adam Stacoviak
So I've noticed like anything I want to keep, have it write tests because not only does it prove that it works...
\[108:25\] ...it's a good hint for it to understand how the code works. Just like humans, you know, any time I, like when I review someone's code, I'm like, where are the tests? Not because I'm asking where your tests are, because in your tests, I can kind of figure out like what you were thinking when you wrote this and what you're trying to do. So I've noticed that too is like anything you care about, you know, that you have AI writing, have it write some tests too. It's, you know, cost you what, some tokens, not really any brain power, but tell it to do it too because it's good. It just helps, you know, your agent understand in the future what it was doing before.
Techno Tim
YouTube.com. What is your... what is your... Do you, do you say what your YouTube is out loud?
Adam Stacoviak
Just Techno Tim. I usually just, yeah, just people Google Techno Tim. It's technotim.com now. Huge, huge change. No more technotim.live. It was that, you know, I paid the squatter. Now I'm big time. I have a dot com now. I had to put tons of redirects in place. I was like redirects everywhere. Actually, I did them on the edge on Cloudflare most of them. But then I had to like find my old links. Then I have link shorteners. Then my email. And then I set up aliases and cut that over and a lot of DNS stuff. I actually did it in about a day. I had help, for me. I did about a day. So it wasn't that difficult. It was just a lot of things to remember. A lot of things to remember to do. And still, I think I still broke something. But at least I didn't, at least I didn't lock myself out of my own email. So if you ever changed domains, it's a lot. Do it as early as possible or never.
Techno Tim
Oh my gosh. It's like renaming something. It's the worst. Especially if, yeah, it's just the worst. Pick a good name from the beginning. Do what you can to never have to change your domain. Ever, ever, ever.
Adam Stacoviak
Yeah. Yeah. Yeah. Never name your business after a street or a product. Unless it's...
\[110:27\] ...like main street. I said, I noticed that with so many, you know, local businesses. It's like, you know, we're, I don't know, we're Whatever Boulevard, but they're not on Whatever Boulevard anymore. It's like, oh, wait.
Techno Tim
We actually have here in Dripping Springs. We have a Mercer Dance Hall. And it used to be on Mercer. And it's not anymore because Mercer's real estate got more expensive. Now they're on like Route 12.
Adam Stacoviak
Yeah. See? Exactly. You're not Mercer Dance Hall anymore. It's like, where do you go to Mercer Dance Hall? Route 12. Everybody knows Mercer Street here. You know, it's like, well, I'm here on Mercer. Where's the dance hall at? Either it's... it's not anymore. It's like, yeah.
Techno Tim
Yeah, that and a product like, you know, yeah, or like a price, you know, Dollar Store. They're not even dollar store anymore. It's like Dollar 25 Store, you know, but they're still Dollar Store. That one's kind of generic, but you know, if it was like, Everything's a Dollar or Everything's $5 and it goes up to 10, then you're in big trouble.
Adam Stacoviak
Well, there was another store called something Five. Oh, Five Below?
Techno Tim
Five Below. Yeah. You can go in there and buy things about five bucks. That is right.
Adam Stacoviak
What is it? Yeah. Now I get it. The color is blue and they make it like it's cold, but that's right. Like it's, you know, it's a single on tundra now. You know, come on.
Techno Tim
You know, that is right. Yeah. Yeah. Yeah. You better keep a five below in there. You know, temperature wise because, you know, not everything's below $5.
Adam Stacoviak
That's right. That's right. Well, everyone, technotim.com. Check that out. Thank you, Tim, for just exploring this fun world of homelab with me every year. But my prescription is go away. And instead of coming back a year later, come back in three months and tell me about your new world. I want to see Tim's new world in three months when you just unleash AI even more so, these agents on your homelab. I'm here for it, man.
Techno Tim
I love talking to you too. It's always a pleasure. You give me so many ideas. And now I think I have a lot of ideas.
\[112:31\] I want to do something right after this next video. But yeah, I'm excited. I'm glad to be here. And it's always nice talking to you, man.
Adam Stacoviak
Yeah. Same. Same, Tim. Same, Tim. Good seeing you. Glad you're well.
Techno Tim
Bye, y'all.
Adam Stacoviak
Bye, friends.
Techno Tim
Bye, friends.