Hey everyone, it's January 10, 2026, and looking back, it's been almost exactly a year since I unboxed that UniFi Cloud Gateway Max on January 17, 2025. What started as a quick upgrade has turned into a quiet game-changer for my setup—mostly because life got in the way and the homelab sat mostly idle for months. But now that things are calming down, I've had time to reflect, optimize, and even lean on AI to guide the rearchitecture. This ties right into my ongoing experiments with local LLMs and AI workflows (Ollama evals, GPT-4o vs. locals, etc.)—using chat models not just for quick answers, but to frame real decisions when time is short.

The unboxing was low-key: excited phablet-typing at low battery, but the real story is the upgrade path. I kept the old Cloud Key Gen2 Plus (UCK-G2-PLUS) since UniFi Protect was already set up perfectly—cameras, NVR, all humming. Instead of a full migration (which bombed on my first restore attempt—network reconfiguration fail, thumbtack reset drama), I just adopted it into the new Gateway Max as a pseudo-dedicated NVR. Smooth controller integration, offloads video processing, and keeps the main gateway focused on routing/firewall/visibility. Pro tip: Document everything—VLANs, firewall rules, port maps. I learned that the hard way.

No AI for choosing between Cloud Gateway Max and Ultra—that was standard research, heavily influenced by Evan McCann's excellent comparison charts. The Max's compact form, 2.5Gbps ports, and solid IDS/IPS headroom for home/SOHO fiber made it the clear pick for low upkeep.

But when it came to the deeper homelab rearchitecture—storage pools, ZFS tweaks, degrowth from power-hungry enterprise gear—I threw the initial prompt at several big chatbots: Perplexity, Grok, Copilot, and Gemini. Gemini Pro (specifically Gemini 3 in that session) stood out with the most comprehensive, context-aware response that built directly on my details without needing much re-prompting. It suggested ditching RAIDZ1 for striped mirrors, provided copy-paste commands, and spotted pitfalls I missed. We can dive deeper into why Gemini won (and how context size played a role) in a follow-up post—I'll link it here once it's live. For now, the key takeaway: AI framed the whole pivot perfectly, saving me hours during a quick sous vide wait (because why not?). Project done, celebrated with a nice steak and a glass of Argentinian Malbec—pure instant gratification.

Looking back at this past year, the real starting point of my homelab rearchitecture wasn't the hardware changes or the final ZFS commands—it was a single, focused chat session with Gemini Pro (Gemini 3) in November 2025, about half a year past the time I wrapped-up the Proxmox VE three-part series (May 2025). I needed to solve a classic homelab dilemma: my existing ZFS pools (basin, pond, stank) were mismatched RAIDZ1 setups that throttled random-write IOPS for heavy log ingestion in Wazuh and OpenCTI, while the MD1200 DAS was a noisy, power-hungry relic I no longer needed for capacity. The goal was a cleaner, faster flashpool using striped mirrors, zero-downtime migration of all LXCs/VMs, and future-proofing without buying new drives.

Instead of spinning up n8n (general workflow automation with 300+ nodes, great for complex pipelines) or Flowise (LangChain-based low-code builder for RAG chatbots, vector search, chunking, and multi-agent flows), I just pasted my full hardware context into Gemini and let the conversation flow naturally. Modern LLMs' context windows (Gemini 3 Pro supports up to 1 million input tokens, though practical chat sessions often operate effectively in the 32K–128K range for speed) meant it remembered every detail across turns—no re-explaining hardware, no chunking strategies, no vector DB setup. I could discuss concerns ("Striped mirrors will sacrifice some capacity—worth it?"), refine aspirations ("Prioritize reliability for self-hosted sec tools over bulk storage"), and get grounded suggestions with built-in reasoning.

It felt like having a knowledgeable homelab partner at 2 a.m.—when no one else in the house cares about RAID trade-offs or ashift values. I argued points, it pushed back with logic, and it architected the end state first (flashpool as the target), then handed tactical steps—including the key storage migration I used to move every LXC (from Pi-hole to Immich) to rpool via Proxmox's "Move Disk" feature (GUI or qm migrate CLI).

The biggest value here isn't just code generation (though the copy-paste commands were spot-on)—it's the shift from "build your own RAG pipeline" to instant, conversational architecture. In the past, for a similar deep dive I'd spin up n8n or Flowise, requiring nodes, connectors, chunking strategies, vector DB setup, and testing loops—hours of plumbing before you even get useful output. With modern LLMs' massive context windows and native capabilities, I skipped all that. No pipeline setup, no chunking debates—just describe the problem, argue concerns, refine aspirations, and get grounded suggestions with search grounding if needed. It feels like a 2 a.m. homelab buddy who remembers your entire setup across turns, no one else in the house needs to care.

This usability leap has opened doors for more efficient "code vibing" homelab projects—iterating fast on configs, scripts, or even full services without heavy orchestration. (That's fodder for another post: what's up with Antigravity, Cursor, and Claude Code in the agentic coding space? Early 2026 vibes are wild—Google's Antigravity as an agent-first IDE with autonomous planning/execution via Gemini 3, Cursor as the refined VS Code AI accelerator, Claude Code for terminal-deep reasoning and autonomous refactors. More on that soon.)

Here's the key part of that brainstorming flow (lightly edited for readability, but preserving the back-and-forth feel—full history attached for reference). I've pulled directly from the conversation to ground everything in truth: Gemini's suggestions were tailored to my described future requirements (high-IOPS for sec tools, single-node simplicity, power/noise reduction), and it emphasized migration to rpool first as a safety net.

My initial prompt (dumping hardware + goals, as in chat):

i'm rebuilding my tech stack and plan to host most everything in Proxmox VE via LXD as much as possible. this is currently in my plan: Shuffle aka Shuffler, Wazuh, OpenCTI, MISP, AIL Project

Gemini's response (key excerpts—framed the strategy, split into native vs. Docker-in-LXC based on my requirements):

This is a highly feasible plan. You can host nearly 100% of this stack in LXC containers on Proxmox... Prioritizes the Proxmox VE Helper-Scripts... Type A: Native LXC (Best Performance) Tools: Wazuh, MISP, AIL... Type B: Docker-in-LXC (Easiest Maintenance) Tools: Shuffle, OpenCTI...

Wazuh (SIEM) Recommendation: Use the Community Helper Script... Command: Run the community script command in your Proxmox shell to spawn a dedicated "Wazuh" LXC... Allocate at least 8GB RAM...

Shuffle (SOAR) Recommendation: Docker-in-LXC... Use the Docker LXC helper script... Caveat: Shuffle uses Opensearch. You must set vm.max_map_count=262144 on the Proxmox Host...

And so on for each tool, with detailed methods.

Follow-up prompts/responses (iterating on Dockge, Podman, CasaOS—Gemini argued against Podman for my stack, suggested Dockge):

User: I also previously used Portainer CE but can forego that if there is a simpler solution... I've also started to experiment on Podman...

Gemini: This is a common pivot... Switch to Dockge... Why avoid Podman for Shuffle/OpenCTI? Networking Complexity... Socket Binding... Documentation... Revised Plan: ... bash -c "$(wget -qO - https://github.com/community-scripts/ProxmoxVE/raw/main/ct/docker.sh)" ... curl https://raw.githubusercontent.com/louislam/dockge/master/install.sh | sh

On TrueNAS/CasaOS/MD1200 pivot (Gemini suggested PCI passthrough for TrueNAS VM, warned on MD1200 issues like fan noise):

User: I'm also interested in Casa OS... re-install TrueNAS Scale as a container but physically tie it to control my Dell MD1200 array...

Gemini: This part of your plan needs a course correction... CasaOS: Skip it, redundant with Dockge... TrueNAS Scale: Must run as VM, not container... PCI Passthrough for HBA card... Warnings for MD1200 + SSDs: Fan Noise (The "Jet Engine" Problem)... 6Gbps Limit... Interposers...

Core storage rearchitecture (the big one—Gemini suggested migration first, then destroy/wipe/create based on my goals):

User: ...the script did not seem to detect the previous installation

Gemini: ...Before doing anything, stop the container and take a manual snapshot... Verify your Current Version: /var/ossec/bin/wazuh-control info... Manual Update Sequence: apt-get update... apt-get install --only-upgrade wazuh-indexer -y... etc.

Later turns focused on pool states:

Gemini: pool: basin... config: raidz1-0... (full before states as extracted)

After user shared zpool status: Suggest striped mirrors for flashpool, with exact create command matching my devices.

Migration: "Use Proxmox's Move Disk in the GUI to relocate containers/VMs to rpool first—zero downtime—before destroying old pools."

Full commands from Gemini (as I executed—always test/back up first!):

Bash

# Migration (per container/VM)
qm config <ID>  # Check current disk
qm migrate <ID> --targetstorage rpool  # Or GUI: Hardware > Disk > Move Disk

Bash

# Destroy (post-migration)
zpool destroy basin
zpool destroy pond
zpool destroy stank

Bash

# Wipe
for disk in sdc sdd sde sdf sdg sdh sdi sdj sdk sdl sdm; do
  wipefs -af /dev/$disk
  sgdisk --zap-all /dev/$disk
done
partprobe /dev/sd[c-m]

Bash

# Create flashpool
zpool create -f -o ashift=12 flashpool \
  mirror scsi-36b083fe0dffbfa002d21cca4172514ec scsi-36b083fe0dffbfa002d21ccbb187797e4 \
  mirror scsi-36b083fe0dffbfa002d21cf9227b92ed4 scsi-36b083fe0dffbfa002de694ac1f246e22 \
  mirror scsi-36b083fe0dffbfa002de694ab1f06447c scsi-36b083fe0dffbfa002de694ab1f10ce50 \
  mirror scsi-36b083fe0dffbfa002de694ac1f16b6a1 scsi-36b083fe0dffbfa002de694ac1f1e4a2c \
  mirror scsi-36b083fe0dffbfa002de694ad1f2bbbbf scsi-36b083fe0dffbfa002de694ad1f3282b5

Bash

# Tuning
zfs set compression=lz4 flashpool
zfs set atime=off flashpool
zfs set xattr=sa flashpool
zfs set recordsize=64k flashpool
# Quotas for sec datasets
zfs create flashpool/sec-stack
zfs set quota=4T flashpool/sec-stack

Bash

# Register
pvesm add zfs flashpool --pool flashpool --content images,rootdir --sparse 1

Before/after pool states (direct from chat—Gemini echoed my zpool status, then suggested the new config):

• Before (basin example): raidz1-0 with three drives, ONLINE.
• After (flashpool): Five mirrors, ONLINE—matched my SSD count for striped performance.

This collaborative vibe—argue, refine, confirm—turned a potential headache into a sous-vide-session project (steak + Malbec celebration still the highlight). No RAG pipeline, no chunking debates, just instant, context-aware architecture.

The payoff? A clean flashpool rebuild, MD1200 shutdown (kept as optional expansion), and a lighter, more reliable homelab—all sparked by one chat that felt collaborative, not mechanical. Sous-vide steak + Malbec celebration? Still unbeatable.

In a follow-up, we'll compare why Gemini Pro edged out Perplexity/Grok/Copilot here (context retention, native tools), and tease more on emerging coding agents.

Drop your own 2 a.m. AI partner moment below—what's the wildest late-night homelab win you've had with chat alone? 😄

Let me start by saying that I've figured out a repeatable way to get around this issue which had stumped me all week! And, here's the proof:

Let's back up a second and talk about what happened and what got us here. Previously, these things happened (not in any specific order):

• An urgent message was received one day from someone at home that the "internet was dead". That situation was true for about a fully day overnight since it required an ISP technician to come inspect their fiber hardware. I was able to get some very minimal IoT smart devices working but nothing that needed the "cloud" worked.
• At an average of about 11-cents/kWh the servers typically hit at $30~50/month prior to summer on-set and more than likely, due to their garage location in North Texas hea, would register more if I did a new Kill-a-Watt measurement. As it is, they're running at a noisy full-tilt instead of the normal 20% speed.
• A mix of both ARM64 and AMD64 machines in a Proxmox VE cluster is not the most optimal way to run things due to inconsistencies of storage media and other dependencies.
• Recently de-clustering my high-availability setup due to the resulting heat and aforementioned cost of running a #homelab and job function change.
• Wanting to find a way to setup Frigate and leverage the Coral Dual Edge TPU and PCIe 1x adapter but finding all current market options too costly or power hungry

These are our current cast of characters: old laptop hand-me-downs from my now no-longer teenage son in the forms of an Acer Aspire V5-122P (AMD) and an Acer Aspire E5-576 (Intel with QuickSync). These had previously been pre-installed with Windows 7 and I had then flashed with the latest Ubuntu Desktop at the time of being received.

I then followed the appropriate hardware install instructions for the official Home Assistant Operating System Generic x86-64 which called for enabling UEFI Boot and disabling Secure Boot (because HassOS uses Buildroot and a customized version of Linux).

Let me digress here and mention that about two-(2) weeks prior I had been looking into converting at least one of the Nano Pi R5S into Home Assistant hosts but wasn't getting very far since I had to figure out how to merge separate Buildroot processes, one for HA and the other to incorporate the Rockchip drivers.

There are some older instructions out there that talks about setting EFI and all that but in my experience those options only came into play when Secure Boot was enabled initially in-order for you to basically reset the options to factory defaults. But if you find yourself in a bind, then feel free to try it out just to remove variables:

I initially decided to go full steam ahead and use the Aspire E5-576 as the main HA unit and, also, wanted it to be running Frigate, Jellyfin, Plex, and a whole host of different things to make full use of the overpowered hardware and Intel Quick Sync. However, the way HassOS is envisioned right now is that it needs to boot pristine without any other things loaded with it during the back-end Docker boot-up. Anything more than that then the installation will complain and you will get the Unsupported nod from Home Assistant observer and that would not be a happy experience, trust me:

So, I pivoted and decided to use the Acer Aspire V5-122P, which only had 500GB of storage, to instead be a sort of head-end and call the Quick Sync enabled machine remotely as a service. Those of us dabbling with AI automation will recognize this as using Agents that do their own autonomous activities.

But, that wasn't before I had already been hitting my head on the wall no matter what I did as a work-around to get past booting into this frustrating message:

I've been all around the support groups and discussions and even found a bug report, which did give me a clue!

Within those workarounds someone talked about renaming labels and partitions and that may be true in some situations where the installation has progressed to completion. I'm only repeating the post here because it taught me a little about getting a separate terminal out of HA without breaking it further: Ctrl-Alt-F2

However, the whole thing about workarounds and renaming labels falls apart because there is nothing to fix, and by that I mean that is what if the partitions and all that underlying stuff may have failed in various ways and there's nothing to operate those fixes on. The key for me was noticing the Dependency failed part of the boot process:

This, my friends, was my aha! moment, the part where latent pieces of the boot and file structure might not have been correctly expunged from the system.

Whether it is a bug or not is debatable. What is important is that removing this part of the information is the key. So, back to the recommended Method 1 of the official HA installation, before I restored the HassOS image I decided to wipe the disk with zero's as well as remove all partitions.

However, as you can see that was going to be more than a seven-(7) hour process for a 1TB disk:

I would not fault you, dear readers, if you haven't had any digital forensic background to understand the GUID Partition Table (GPT) But, it should suffice to say that overwriting that minimum 1% (one percent) of the disk would be more than enough to clear out the data and disassociate most of the disk information from mapping. Back in the days of boot sector viruses this place of the disk was always under contention because it presented a way for malware to load prior to the operating system and circumvent security protections and many other shenanigans!

To proof my theory, I decided to do the same data clearing for the 500GB disk on the Acer V5:

Once completing the image restore you should get a layout similar to this:

The only things to do now are to shutdown Live Ubuntu session, disconnect all external drives, connect any special hardware, boot-up, and (optionally) restoring from backup.

I'm going to give it about a week of live testing for functionality as well as looking at the power consumption from the wall. If things are stable then I might just shutdown the virtualized version and stick with this laptop version.

The next step of the puzzle is how to make use of the space beyond what the default HassOS image allocates by default for itself, approximately 30GB. I'm going to try to tackle that in a follow-up:

There is a mismatch somewhere in terms of what HassOS reports compared to how things are actually mapped on-disk. Here's what GParted reports from a Live Ubuntu session:

As can be seen, and similarly claimed during the installation process, the hassos-data partition did, in fact, expand to the rest of the disk. Why then is the built-in Disk metrics only reporting about 10GB left of free space? If you happen to come by this post and have an explanation, feel free to add it in the comments.

As for the Aspire E5, I'm currently leaning towards making it into just a full blown Docker or Podman host. The reason being that if I installed Proxmox VE on top of it then I'd have to contend with the extra hypervisor overhead on an already system memory challenged system. A simple Ubuntu or Debian Bookworm bare metal server installation might net me just the right amount of processing utility that I need.

See you next time.

Welcome back, it's literally been a minute since I published Part 2 of this series where I found a fix for the Pi-hole NTP sync errors.

I don't think there's going to be a separate need to discuss blogging on Ghost via mobile. It should suffice to say that copy-pasting images works better on Chrome mobile mode; you can get better native desktop-mode experience when using Firefox. The capability to flip back-and-forth seamlessly is a game changer for speed edits. I'm not sure about the experience using an actual tablet like the Google Pixel Tablet or a Chromebook like the Acer Chromebook Plus 14" (CB514-4HT-359X) but that's going to be one of the next experiments. My final edits have been on the old MacBook Pro, whenever it's still got a charge.

Part 3: Proxmox VE Helper scripts meets XDA Developers

Let me start with the elephant in the room, why use LXC instead of using the existing Docker VMs in my current stack? Well, let's talk about that.

I've been running MISP and OpenCTI for the longest time in a cluster for high-availability. Specifically, ElasticSearch can become unwieldy when you try to give it the best possible scenario when you've got limited rack space expansion and a power budget cap. Essentially, because of this I'm considering a re-do and clawing back resources these projects have taken over in resources. Plus, I've recently had some bad spells with old Portainer versions that over time has given me a bad taste in the mouth. And then, from the last time I had touched edits on Docker did Podman enter the picture. So far, in the past few months this might be the direction I take if only so I can learn more about it's pitfalls and the fact it is what you would get to use in a locked-down Red Hat enterprise image. There's alot to unpack there, but my point is that I don't take this decision lightly. There's going to be a learning curve and trade-offs to be made. I'll have to be OK with that. So, this is why we're going back to individual containers in the form of LXC instead of full VMs if we don't have to.

Now, with that out of the way, a brief note on XDA Developers. Hailing from the Philippines, I've been on the cutting edge of mobile computing since back in the Nokia days. If I couldn't get new stuff locally then I could always hop on a plane to Singapore and visit Sim Lim Square. And, on one of these trips is where I got hold of my first O2 XDA (and a bunch of Havaianas). Mind you, at around this time I was sporting the latest Nokia 9500 Communicator. I was ready for another form factor, let those parts sink in. Rooting and hacking bleeding edge mobile devices is how XDA and I crossed paths. Years later after transformation I really enjoy the self hosted section topics for my use case.

About Proxmox VE Helper scripts, I discovered this group while looking to experiment with lightweight versions of my homelab projects. What I would do was learn from the scripts and incorporate or improve on them for my specific use case. Now, however, I plan to use several of the templates to get things restarted based on availability and if something gets mentioned over at XDA.

I've already showed what a basic phablet install experience and experiment looks like over in Part 1 of this series. Here's where I've landed so far:

There's a big part of myself that cringes at the auto-numbering since my original schema was to map IP to VMID. If it matters that much later then, in theory, it should just be a matter of a backup and then restore to a specific VMID. Well, in my mind, that's how I'm coping with the situation.

In Part 4, we can discuss about the use cases and groupings of what's been installed. Or, I may just launch a new series based on specific group topic projects linked stemming from here.

Happy trails.

Strike while the iron is hot? Well, I'm going to do a version of type on this phablet until it powers down because its now at 11% and hopefully not loose any content. I guess that is part of the curse when you've got the ideas and time is measured in either bars of battery or wifi strength.

In Part 1, I talked about success in getting Pihole setup on ARM64. There might be some other lightweight resourced projects to install or migrate later, perhaps Uptime Kuma?

Part 2: Pi-hole NTP sync sidequest

There was an NTP setting that needed to be addressed related to permissions, which by some reports is a bug or a setting that doesn't make sense depending on how you have Pihole set-up:

Navigate to Settings, and notice the green Basic toggle:

Click on this and you'll enter Expert mode:

A new option will then appear called All settings:

Within All settings navigate to the Network Time Sync tab and disable most of the settings:

The most important edit is zeroing the setting for ntp.syc.interval:

Save your settings and you will then notice the error alert disappear:

With that out of the way, onward to Part 3.

Yes, I'm still doing my edits from the Flip4!

It's been a minute.

We'll, several months in fact since getting most of the Lab situated. ​The Unifi gateway has proven to be a useful addition to complete the end-to-end visibility for my network. I've also walked myself back from over-engineering the access points, meaning to say I've held back deploying all of the nanoHD APs and instead forced myself to find the house layout and leverage the Unifi Designer page to map out some better configuration options given the limitations of where I can drill or line some CAT6a.

Yesterday, I looked at my simple stack of containers and decided it's about time to put back some of the tools that I've been missing given my current job function use cases. I still have MISP and OpenCTI running and they're due for a version refresh. While Pihole and Adblocker get updated, they're no longer part of my network's flow and I'll rectify that when I transition to using the new firewall zones feature in the latest Unifi firmware. Home Assistant is running but the Frigate integration hasn't been done until I decide on a spare machine to run the Coral TPU on. Smartthings needs a configuration update since I no longer use or host a Blue Iris integration server and set of IP cameras, I've peppered the place with cheap Wyze cameras, instead.

For today's topic, the main goals I had were:

1. Install something that works on both of the NanoPi R5 installed with Proxmox VE on ARM64
2. Comb through the recent self-hosting posts from the XDA Developers site and select stuff that aligned with my past and future usage goals
3. Decide to use individual LXC containers and leverage those that exist in Proxmox VE Helper scripts site
4. Forego the temptation to use Docker or Podman, but Install fresh containers for future experiments
5. Do all of this using my Samsung Galaxy Fold4 phone while testing the new U7 Pro Outdoor with omni-directional antennas I just installed the day before

Part 1: NanoPi R5 with Proxmox VE ARM64

This previous project is sort of a heartache for me. While I did get Proxmox VE installed, my ventures into installing a working VM and a basic OS have been unstable and mainly unusable probably due to system requirements. And, this is probably where the beauty of LXC shines due to it simply using what's already there.

The helper script for Pihole will fail and complain to you and suggest you use a different installation script. No need, since ARM64 is a supported architecture:

And so, let's see if we can get Debain installed copying the basic specs from:

But, we're going to use the default Proxmox VE container template to create it:

This got created fine:

We now have a running Debain LXC to build upon!

Let's use the one liner installer:

But first, get curl installed and update packages:

Configure the easy defaults:

Finalize checks:

Browse to the web address and enter credentials:

Update the rules:

Inspect alerts:

This NTP sync error has a fix and sort of controversial in its implementation. So, I'll post the specific steps I had to do separately.

Stay tuned for Part 2.

... and we're back !!!

Today just so happened to be a standard month of setting-up shop in the new digs. Most of the boxes have been unpacked except for the holiday stuff. I don't think we knew how much decorations we'd amassed over the years and that already included having gotten rid of all the almost decade old disintegrating stuff (which had actually been great for Halloween, by the way).

The only ISP offering fiber in my area had already installed their hardware before we moved in by drilling a hole through brick straight into the middle of the living room wall which limited my options unless I wanted to make another one to feed out from the ONT modem onto CAT6A. As much as I would have loved to forego a separate server room it felt like a blessing of silence when the Mikrotik router was turned off.

And what did it take for me to do that?

I was hoping you'd ask. But first, here's my server rack hastily recreated in the garage just to get this blog back up and running:

Yes, I know, all those spaghetti cables!

What you don't see is a CAT6A shielded cable snaking 60-feet across open ground from the living room through brick. Instead, there's a black CAT6 cable coming from the UniFi 60W switch down-linked from the PoE injector of the UniFi NanoHD access point into the Mikrotik CRS328-24P-4S+RM switch. Apparently, this works via the magic of how UniFi configures Wi-Fi meshing on access points at the moment and it results in what would have been just power to instead the AP becoming a network media bridge.

I'm not going to complain, the alternative is to pre-order the recently announced official UniFi Bridge, which is currently out of stock. I'm probably still going to order one when it becomes available given some network stability due to the signal having to travel through layers of brick to the detached garage. Or ... I might just consider some kind of upgrade to WiFi 6E/7.

It is pretty interesting starting from scratch. I am now realizing that I had over-engineered (of course!) last time by surrounding all corners of the house with access points, even if I did turn down their transmit signals. This sort of explains why all my IoT devices still preferred to connect to the UniFi 6 Mesh that I'd placed on the roof rather than the nearest access point. So, really, in terms of access point coverage less is more and only add if you really must.

In my next post I'll talk about my experience with finally getting a real UniFi Gateway onto the rack and the status of the rest of the gear as a result of that network hardware change. I'll be posting and update towards progress on hosting this Ghost blog platform on ARM64 as a result of my previous experiment, too!

Until then, Happy New Year of the "Dragon"!

This little journey started the night before Thanksgiving as we were packing for a long-ago planned trip to Japan.

As is my usual wont, I found myself puttering around restlessly with the HomeLab trying to eek out a little bit more from the little bits of hardware I had accumulated since my most recent personal research project. The PCIe board for the Coral Dual Edge TPU had arrived but my previous attempt at installing it was hijacked when I accidentally ended up shutting down my Proxmox VE node that was hosting several of my Docker projects – including about one(1)-TB worth of OpenCTI metadata that hadn't had time to settle and get written correctly to disk. How was I to know that the hardware backplane on the Dell R720xd it was hosted on was going to bork and make my ZFS cluster disappear into the ether?

That unfortunate incident wasn't all too bad. At this point having all that data just happened to be not as critical or immediately usable for work stuff and I'd already rebuilt the setup several ways enough that I've got a working project fork of OpenCTI to my own to help ease the recovery and deployment process. Then, too, I've been back to having my router, switches, and WiFi on physical machines such that my whole house network and its entertainment side of the equation are not affected. But, now, I did have a pair of FriendlyELEC NanoPi R5s sitting cold and unused and just begging to be put back into use – not as FriendlyWrt travel routers but as Docker hosts. In fact, I mused, could I run an ARM64 version of Proxmox VE on these things?

The bigger plan, of course, was if it would also be viable to to install Proxmox VE onto a high performance AArch64/ARM64 server and see compute gains on something like Apple Silicon.

It started well enough. I was mildly surprised to know that on the latest 2024-10-16 firmware release it literally said "Added Proxmox VE system". What luck!

Per the instructions, I downloaded the latest Official image from the link and used the Flash Official OS to eMMC option via TF Card.

Further perusing lead me to the "Getting Started with Proxmox" page. I used balenaEtcher to write the file onto a 32-GB SDXC card (the multi-OS version takes up about 16GB decompressed), inserted it into the TF card slot and plugged-in the power brick. And then watched from the old spare monitor connected via HDMI as the system kept booting into a loop-cycle for a few hours before I gave up, powered the whole thing down, took a nap, then left for the airport to Japan.

Returning from Nara then Osaka via Haneda to the Narita airport a few days ago, I suddenly wondered if the power brick that I'd used from the UniFi Flex Mini was the culprit. My assumption was that it provided 5V/2A but to my surprise it was actually only rated for 5V/1000mA and the boot loop immediately righted itself once I plugged in the power to an Anker Prime Power Bank. Basic TF to eMMC installation took less than a minute for each SBC, after that:

I made sure to add all the pve8 repos and do an initial update of everything to bring up the Proxmox VE version from 8.2.x to 8.3.0

But, there's one other problem, I wanted ZFS. Thankfully, I was able to find the source of the Proxmox-Port that FriendlyELEC had used to build their image and it had part of the installation instructions.

In the case of the NanoPi R5s, I was able to find the header information and kernel that matched it from:

Here's that download:

I gave it a go with just the headers, initially:

And discovered that ZFS wasn't loading:

Checking the available kernels, I noticed that 6.1.57 wasn't even one of the available options:

I'll save you some tinker time and say make sure to install the Linux kernel headers and images first. Then, reconfigure zfs-dkms for good measure, and then reboot:

And, now this error which was sort of weird:

This didn't fix things all the way and I found a reference to try and do a dkms autoinstall which would help bake-in the missing ZFS modules into the kernel. I tried two-(2) different approaches to see which sequence works best but in the end they both worked.

And profit!

Once the ZFS pool(s) are created to your liking then you can try some of the tips and tricks to create dataset folders. Because we're dealing with ZFS, it is these dataset folders that you can map from the Proxmox VE Datacenter -> Storage options to contain whatever image types you prefer.

My initial impression based on a quick test, from a usability standpoint, its easier to create an ARM64 virtual machine inside a standard x86 host machine than the other way around. That may simply be due to the limitations of the SBC hardware that I have on hand.

So, in summary, this works thanks to the magic of Linux and hardware cross-compatibility, and all the hard work poured over this Proxmox VE fork. I'd really love to see it officially adopted and it might have even happened if large enough volume installs merited official developer support. As it is, Apple no longer makes servers; while Intel and AMD have finally caught-up to Qualcomm since I last wrote about it a few months ago.

If I do find an Apple Silicon to play around with, you can be sure I'm going to give this build another try and update y'all. Happy tinkering!

The Rabbit R1 was introduced during CES 2024 and it was probably one of the promised products that helped make the the term Large Action Models (LAM) more widely known compared to mainstream Large Language Models (LLM) and, to a degree, the future iteration which is Large Agentic Models (LAM).

This got me excited since this would essentially be a physical example of what an LLM stand-alone device could be. Yes, I knew it would be buggy. And, so, I pre-ordered as soon as it became possible to do so. I'll get to the unboxing later below.

Rabbit notes what their definition of what a Large Action Model as:

Whereas, the newsletter Towards AI published their mapping comparison last March of how these currently map in their estimation:

However ....

I think some earlier example of agentic model reasoning

In a paper published on Arxiv towards the end of January 2024 talked about LLM Situational Awareness. One of the examples was a generated image and an action-flow based on what was seen, including a all to restock whatever was needed for refill.

The thing is, back in 2013 a similar function was available on some refrigerators as reported on CNET: The LG Smart Refrigerator know what you have, knows what you need. But, maybe that was probably a low hanging fruit choice example. The other examples in the paper were more interesting, such as a potentially to automatically call emergency services during life threatening situations. I doubt my smart fridge, 10-years old technology as it is, would do that. But, perhaps an SOS from a smart watch? Or in the case of vehicle accidents then the onboard computer (the old OnStar emergency services comes to mind). Since everyone has a phone, then Android location services?

My point here, being that any new use of GenAI should be showing improvements from things that already exist on many consumer devices.

Once the Rabbit R1 arrived, I let it sit on the table top for a few weeks. I gave everyone the chance to either love or pan it, as well as the developers a chance to fix the pre-release bugs they could. Today, I decided to review it. What are weekends for, right?

I found it odd that Rabbit's service link calls out to a link called VPN Proxy. Maybe its to circumvent or hedge against some network filtering? Whatever it is, the result was a wait-time to get to the log-in screen for each service.

In the case of Door Dash it wouldn't even let me in because there were "numerous logins from my IP". Well, uh, oh! Me thinks that its possible that the use of that transitional VPN could be the culprit? I'll try to give it 24-hours and if it still does the same then that more or less confirms my hunch.

(to be continued!)

Its literally been a week since I walked into a Costco and took notice of the electronics area. This is the normal store layout: beautiful 8k and 4k OLED displays first followed by some quick-selling items (that day it was golf shirts), turn the slight corner and you have the laptop display right next to the tablets and phones. I normally skip this area quickly since I have a "phablet" (when was the last time you head that term used?!) in the trappings of a Samsung Galaxy Fold4 and a factory replaced Pixel 7 Pro (ever since the Pixel 6 Pro met its photographic hardware demise) and there is nothing in terms of a mobile carrier combo that the service people in that area can offer me better than what I, already, have.

However, something caught my eye that day. I must have been thinking about the projects I was doing and combination of hardware that are in this blog. Were those actual Copilot+ PCs on display? Not just one, but two!

Make and finish

I stepped closer and lifted each in turn, testing for the weight and sturdiness of how I might use it in day-to-day.

The keyboard on the Microsoft Surface Pro immediately dropped away. Ok! It felt almost exactly like the first time I held an iPad with its keyboard case more than a decade or so ago. Lighter. Plastic-like or flimsy is what came to mind. Somehow, my expectation due to the look was something akin to a Lenovo X1 Carbon – but, it wasn't. This thing will probably break in pieces if dropped on the floor together with that 16-hundred dollars and change.

The Asus Vivobook felt more familiar as a laptop. Well, actually, wouldn't it be more fair to compare this to a Chromebook? Unfortunately, even if it was plugged with lights underneath the keyboard no amount of fiddling with it could turn the display on. No audio or visual sensory feast here. It was simply "dead, Jim."

Out-of-the-box experience

When you put something on the floor display make sure it has every chance to impress. The only thing that seemed to have been done to the Surface Pro was make sure it was connected to the official demo Wi-Fi connection. Apart from that there was this rectangular widget-like window that I had to partially configure and accept EULA's in behalf of the next user after me. In truth, it doesn't seem like anyone had touched this thing since it rolled onto the floor if I had to do the initial set-up. After all the hype leading to and from the Microsoft event I was expecting fireworks, balloons, and cotton candy. If this was a State Fair in Texas then give me something fried and dipped in butter with a beer on the side! But, no. None of that.

Let's take it for a spin

There was no settings that I could configure (nor did I want to sign-in my own account). But, there should have been one set-up. Regardless, I was going to test it the usual way.

In the succeeding screenshots the fuzzy ones are from the Costco display and the clear ones are from Copilot in Windows (preview) on this desktop that I'm writing on.

This is a solo shot but if memory serves it was similar on the Surface Pro:

Now, let's ask the standard question:

How about some web search capabilities? Lets look for a blog on local AI usage from a local (this blog). This tests the location access rights the OS and app asked permissions for, as well:

As you can see it can't find the information that a simple Bing web search would:

And when somehow it tries to later on stay on topic locally for "Texas" it then fails in the overall topic context of technology and goes religion. Granted, such a link does exist if you've recently been watching some episodes of the 3 Body Problem. The standard non-Copilot+ PC has no issues with this question or context.

Here are my attempts to steer the Surface Pro Copilot+ PC to the right direction with clues that the LLM should be able to pick-up but, here, it continues to loose focus on the context and just spit out answers to the most recent question.

Honestly, my Amazon Alexa and army of Google Chromecast can do this better in spades and those aren't even upgraded to use Anthropic or Gemini, respectively!

Copilot with online strings attached

I mentioned early on that the WiFi was connected to the demo access point. While making sure I was getting the best experience via a normal speed test, I tried a few queries on this so-called Copilot app experience and was greeted with a need to be online for things to work.

What?! I mean, whatever happened to the point of having an NPU chip on this thing? How could a device that is 5-years newer be worse than what my home-built 12-core (24-threaded) AMD Ryzen desktop PC is able to do with the Copilot preview on the task bar?

I'm going to have to try sometime later, perhaps in Part 2, and install Ollama to force this Surface Pro to work as it should. That being said, it appears that is the same experience that PCWorld reported about the same time a week or so ago.

Bottom line: When offline, the current generation of Copilot+ PCs are dead in the water for anything AI related.

Game on

Let's not even go there? But, since you're here I might as well mention one more time that my AMD Raden 5700XT from 2019 gives me around 19.51 TFLOPS whereas the Surface Pro barely eeks out 4.6 TFLOPS. That's simply horrible for any serious gaming in terms of rendering framework and similarly what was, also, reported elsewhere. The fair but mismatched comparison of the Surface Pro would be against the same Qualcomm Adreno line-up performance of the Samsung Galaxy Fold4, but in a laptop/tablet-ish format.

Bottom line: No Cyberpunk 2077 for you!

Not blown away

It's ok. But, like I just showed, these demo floor units and the way Copilot+ PC (and AI PCs that are coming out from AMD and Intel later this year with more serious APU applications) need to think about the user experience if they're going to try to show the benefits of what generative AI can do.

Bottom line: Will the real Copilot+ PC with fireworks blazing please stand up? Please stand up?! Right now, this is meh.

Outtakes

This is where my testing ended. I didn't want to hog the demo units any longer than any casual buyer. However, as I stepped back to give the next person in line a chance to try is when I noticed there was no one wanting to do so.

Not one kid. Not one teenager. Not one curious geeky adult like me. What?

The Competition

In other news, Apple with the Apple Intelligence implementation which upgrades Siri on the iPhone 16 (partially backported to iPhone 15 Max) and MacOS Sequoia with OpenAI built-in comes out later this year in the Fall.

The way things are going I'm pretty sure Apple is going to go hard on AI functionality in every device they have so it is going to be interesting to see if demo lines are busier come that time.

Some two-(2) years ago now, I posted a a video about a bunch of UniFi/Ubiquiti hardware that I'd suggested for purchase to better manage my research team's various network use remotely with the idea that there was no guarantee I would consistently be able to access it physically due to whatever circumstance. The choice of a more user-friendly GUI was a big plus since I've been trying to teach, upgrade, and pass-along some intermediary networking skills as we all move into cloud native architectures.

Now, I actually use these prosumer devices myself and I have, in fact, detailed some of the quirks of using UniFi WiFi products in the blog. However, for the office research lab needs and IT policies, it was to be lined all the way. So be it.

And then, the pandemic happened and that was immediately a return on investment in terms of time. Good thinking Wonder Dog? Me thinks so!

Upgrade Path

There have been several different upgrades to the UCK-G2-PLUS line-up in the last three-(3) years. I'm going to to stay with the line-up that still offers UniFi Protect at the ~$500 range and keep this entry geared towards my prosumer/SOHO audience.

You would think that the similarly priced UXG-Max would be the upgrade, and you would be wrong. This needs a Cloud Key to function as noted in the description. Think of it as a network bridge.

The most logical upgrade a few months would have been the Dream line-up starting with the, so called, Dream Router (UDR). The addition of a WiFi 6 access point at the same price point is the right path if you don't mind the pill-like form factor.

The next one to consider would be the Dream Machine Pro (UDM-Pro) which gives you a taste of 10G networking via SFP+

If you're jumping from the original UCK or UDR then it would be worthwhile upgrade to the Dream Machine Special Edition (UDM-SE) which adds some incremental IPS routing upgrade and POE features.

No longer in the sub-$500 range? It is sneaky for UniFi to slot-in an upgrade path to 2.5GbE with all the above features for an additional $100. But, that's what they did in the Dream Machine Pro Max (UDM-Pro-Max) and it answered one of the most requested features during the pandemic as the industry and the world looked to improving network speeds without having to commit to fiber or totally re-line their networks.

If you'd like me to talk a bit more on the enterprise-grade switches in my initial video, let me know and I might just make a follow-up on that.

Rack Mounting

If you purchased the rack mounting accessory, you may find yourself wondering why it doesn't slot correctly:

And, if after doing that the screen doesn't light-up? It's the tape seal:

If the LCD goes into Night Mode immediately and you can't see the display this has everything to do with your Console Settings and the time of day you were doing this installation. Check there.

Storage vs Protect

Last week the internal 2.5-inch one-(1) TB Toshiba 5400RPM spinning disk finally gave-up. These are OEM priced at about $67/piece and constitute about 30% of the total hardware price. Based on the specs these run at 5V, which is the expected available power for any disk replacement option. Take note of that.

Cloud Keys (and some in the line-up) run hot and don't have a fan by design. Instead, it uses the casing to dissipate the heat. This makes it so the devices are absolutely acceptable right next to your work space, just make sure it is well ventilated.

Are you dedicating a room and air-conditioning to a converted server room? I once did that in Northern California. We're not doing that again. This human comes first, this time. Come talk to me in the weekly Hardware Geeks Anonymous meetings.

Ubiquiti made the tool-less storage upgrade process painless. All you have to do is slide the switch and the drive tray will pop out.

When selecting the replacement drive make sure it is a CMR (not SMR disk). There are not CMR disks beyond 4TB, as of today's search. That's a clue if you go online-shopping for a spinning disk. Western Digital, who now owns SanDisk, got a lot of flak a few years ago and now have a page about the difference.

Back to why I mentioned heat. This is a spinning drive disk killer. AKCP has a great article about this. It is for that reason I decided to repurpose one of the old Samsung SSDs that I had plugged in to the Dell R720xd mounted into a virtual TrueNAS Scale ZFS RAID1.

Did I mention this blog just came from the dead about a month ago? Here's that story.

None of the official join, adopt, migrate instructions worked. The cameras were connected to their respective access points but not showing up under Protect. How frustrating!

It was probably around 3am by when I gave-up and decided to go nuclear and hard reset each camera in turn. Voila! That worked!

We're now back in business and ready for Monday morning!

As part of this local LLM series, I wanted to know the lower limits of being able to self-host and potentially share demos or development frameworks with my team. Given the acceptable baseline discussion in my previous post, we'll use that to audition the hardware we come across. For testing we don't need creature comforts of a GUI and text chat is acceptable. In that case Ollama is probably one of the most straightforward for us to get things going.

Dell Latitude 5400

Let's start with the spare laptops that I remembered were lying around in the semi-discard pile. On this system I imagined it would be initially sitting at my cube connected to the network and eventually relocated to the server room when fully deployed.

These systems were purchased before the pandemic of 2020 and that kind of makes sense from n Intel Core i5-8365U chip production perspective of Q2 2019.

From the low eval rate and the slowness at which the simple response stuttered across the screen, I promptly returned it to the pile.

Hindsight learning, you should consult with an experienced PC builder when purchasing hardware instead of going with default specs. My most recent and oldest system lasted a full decade before decommission due to hardware failure.

Lenovo X1 Carbon (5th Gen)

This is my go-to vacation travel laptop apart from the fact that its technically of no value coming out from 2017 if it sunk under the sea, yet, good enough to let me reach by VPN and allow me to stream my remote watching fix. The X-series is known for the 180-degree opening and water-resistance rating on keyboard splashes.

The test results are even worse than the initial spare laptop that is still in service. But, that's to be expected. In reality my Google Pixel 6 Pro or Samsung Galaxy Fold 4 would probably have better stats (and those are mobile phones!).

I'm, also, looking to see if it's even possible to get some local LLM mobile testing going as a separate set of experiments given as ARM chip usage started there, and not the other way around for Copilot+PC. We'll see.

Dell PowerEdge R720xd

This is what I really thought would be worthy of hosting and showcasing all the development frameworks, builds, and demos. There are a bunch of these in the Lab that had replaced era~2006 PE 2950 (Dell Generation 9). Mind you, these "newer" systems are now two-(2) years well past the decade mark and system production for Dell Generation 12 was ~2012 from this month of June 2024. I randomly picked one of them amongst my Proxmox VE high availability cluster set and ran a test:

I really had high hopes for this server given that it is technically had a 16-core Intel Xeon E5-2650 v2 running at 2.6GHz (for a total of 32-threads) loaded with 188-GB of on-board RAM. You can run several production-ready Elastic stacks at 16GB maxed heap (32GB total) and the fact that the Proxmox VE cluster that I configured is running the latest version of Ceph for data redundancy. But, at barely five-(5) tokens/second this isn't going to be a fun multi-user experience.

Hardware to capability

It is important to note that the number of parameters used to train language models and the contents of said parameters are each model developer's secret sauce, the current normal non-critical standard appears to be 7B (7 billion parameters). The trick is how to cram all of that into the smallest size possible (~4GB for most modest hardware), and that is where quantization comes in. HuggingFace has a gentle primer if you want to dig a bit more on 4-bit quantization.

As I've mentioned before, the rate of quantization directly affects the quality of responses. When we say quality, we also mean guardrails against hallucinations (or in common parlance, "making sh*t up").

While improvements in the quality of quantization and fine tuning progresses in parallel, one can only hope those improvements continue to be small enough that last or older-gen hardware can still make use of it.

Ergo, the observations

Per the optimal system requirements of Ollama the above set of hardware examples gets away with the RAM side of things even if they aren't necessarily DDR5.

However, things start to fail on the CPU side because AVX-512 (an extension of Single Instruction-Multiple Data on chip) was only introduced in 2013 and only really got to the server chips past ~2016.

These two-(2) things as hobbled variables are why a ~2012 maxed out server with more cores and RAM appears to trounce a 2017 laptop; but ultimately is as unusable as a 2020 laptop with bare minimum specs.

For all of these reasons and more, the recommendation is to:

• have more than 16GB of DDR5 RAM, and
• a chip manufactured in 2021 or with a lot of cores, plus
• highly recommended, a GPU in the compatibility list

Addendum

There might still be hope!

In the latter part of my IoT/SBC experiments there is an overlooked segment of USB/mPCIe/PCIe/M.2 manufacturers that are creating products that specifically provide the requisite NPU hardware as hardware add-ons. These are potentially of optimal end-user value than a full-fledged GPU (unless you're also planning to do some serious gaming or trying to do some form of digital creation workflow).

During the start of the pandemic in 2020 and we were scrambling to do work-from-home set-ups, I recorded a short video on how to make an OPNsense/pfSense router using any spare desktop lying around and as many USB dongles you could get your hands on to extend your networking. The intent was to share the knowledge as quick and as fast not just to my team but to anyone who might have needed it:

I never got to publish a public follow-up about the recommended rules as requested from the user comments. Although, I just might at some point find the internal Zoom sharing session I did for my team in the archives.

Anyways, fast forward to 2022 and I'd found myself needing to move my virtual router from my hyper-converged Proxmox VE set-up due to it running hot on enterprise-class SAS spinning drives. In fact, some of those same drives had started failing or their backplanes failing since Dell R720xd models were released back in 2012 – a full decade earlier.

The half-dozen Raspberry Pi 3Bs from 2016 that I'd been using as dashboard heads had come home with me to be repurposed for something else but, alas, not compatible for *sense but perhaps as very (very, very,) underpowered OpenWRT when combined with a USB dongle and an external switch. Throughput would be quite poor, however, given that the top-end would barely achieve 100Mbps under load in real-world scenarios.

This led me down the path of alternative single-board computers and perhaps an upgrade. However, if you remember 2022, you'll remember the supply-chain issues and the price-gouging prices in the open market. It was so bad, the Raspberry Foundation blog an update to everyone about the current status of things. By July of that year I'd discovered a few alternative boards like the NanoPi R5S from FriendlyElec and a few others. I can attest to the industrial hardiness of the NanoPi and case combo given that those things sat (and still do) in the garage under the hot Texas summers of contiguous 100-degree Fahrenheit weather.

Now, with the 2nd (or was it the 4th) downing of this blog site, I'm looking for a bit more resiliency apart from being hosted online. My sights turned to maybe using a Kubernetes cluster using the aforementioned rPi 3Bs. How exciting, learning new technology + working use-case! But, trust me, this all just ended in Smurf tears the last time I attempted to do that sometime last year and all I got to do was make an SD-card performance graph (which I seem to have forgotten to re-post).

TL;DR

If you're still with me, then let's dive into the previous, current, and potential future hardware I plan to unleash my devious designs on:

Full Comparison Table

SoC and CPU

GPU and Video

RAM and Storage

Networking and USB

Display and Audio

Other Features

Summary of Average Prices

• Raspberry Pi 3 Model B: $35
• Raspberry Pi 3 Model B+: $35
• Raspberry Pi 5 System (8GB model): $150 - $180
• NanoPi R5S (4GB model): $55 - $70
• NanoPi R6S: $120 - $150
• NanoPi R6C (8GB model): $100 - $140

I'm doing some experiments with the Markdown function of this Ghost platform. The above was generated using GPT-4o after a few prompting nudges and having it parse the specs links from:

• https://datasheets.raspberrypi.com/rpi3/raspberry-pi-3-b-plus-product-brief.pdf
• https://datasheets.raspberrypi.com/rpi5/raspberry-pi-5-product-brief.pdf
• https://datasheets.raspberrypi.com/hat/hat-plus-specification.pdf
• https://datasheets.raspberrypi.com/ai-kit/raspberry-pi-ai-kit-product-brief.pdf
• https://wiki.friendlyelec.com/wiki/index.php/NanoPi_R5S#Hardware_Spec
• https://wiki.friendlyelec.com/wiki/index.php/NanoPi_R6C#Hardware_Spec
• https://wiki.friendlyelec.com/wiki/index.php/NanoPi_R6S#Hardware_Spec

Here's that data again in a snapshot gallery:

I've decided for the now to keep both of my NanoPi R5S (with dual 2.5GbE ports and NVMe) each loaded with 512GB NVMe drives and 32GB SDXC cards.

While, they aren't the strongest of the bunch it would be a far cry even if I eventually managed to get all six-(6) of the rPi 3Bs working in tandem into a Kubernetes cluster:

Comparison:

Raspberry Pi 3 Model B Cluster (6 Nodes)

• SoC: Broadcom BCM2837
• CPU: 1.2 GHz 64/32-bit quad-core ARM Cortex-A53
• RAM: 1 GB LPDDR2 per node
• Total Cores: 24 cores (6 nodes x 4 cores)
• Total Clock Speed: 28.8 GHz (6 nodes x 4 cores x 1.2 GHz)
• Total RAM: 6 GB (6 nodes x 1 GB)
• GPU: Broadcom VideoCore IV (one per node)
• Networking: 10/100 Ethernet per node
• Storage: microSD card slot per node
• Operating Temperature: 0°C to 70°C (32°F to 158°F)
• Price per Node: $35
• Total Price: $210 (6 nodes x $35)

NanoPi R5S

• SoC: Rockchip RK3568B2
• CPU: Quad-core ARM Cortex-A55, up to 2.0 GHz
• RAM: 4GB LPDDR4X
• Total Cores: 4 cores
• Total Clock Speed: 8.0 GHz (4 cores x 2.0 GHz)
• Total RAM: 4 GB
• GPU: Mali-G52 1-Core-2EE
• Networking: 1 x Native Gigabit Ethernet, 2 x PCIe 2.5G Ethernet
• Storage: 8GB/32GB eMMC, additional 512GB NVMe
• Operating Temperature: 0°C to 70°C (32°F to 158°F)
• Price: $70 (4GB model) + $60 (512GB NVMe) = $130

Comparison Table

Summary

• Processing Power: Raspberry Pi 3B Cluster wins in terms of total cores and clock speed.
• RAM: Raspberry Pi 3B Cluster has more total RAM.
• GPU: NanoPi R5S has a more advanced GPU.
• Networking: NanoPi R5S offers superior networking capabilities.
• Storage: NanoPi R5S with 512GB NVMe offers faster and more reliable storage.
• Operating Temperature: Both are suitable for 100°F environment.
• Price: NanoPi R5S is more cost-effective.

Overall Recommendation

For hosting container applications in a 100°F environment:

• NanoPi R5S (with 512GB NVMe) is recommended due to its superior GPU, networking, and storage capabilities, along with a better price-performance ratio.
• Raspberry Pi 3B Cluster excels in raw CPU power and RAM, making it better for CPU-intensive and memory-intensive distributed workloads.

However, if I were in a future upgrading mode then the only real comparison to the current set of compared hardware would be between the rPi 5 (with M.2 HAT + AI Kit) and the NanoPi R6C:

The NanoPi R6S is shut-out from this comparison because it doesn't have an NVMe storage capability which severely hobbles it from any storage related processing which happens often enough in AI-related activity.

Comparison:

NanoPi R6C vs Raspberry Pi 5 Kit

SoC and CPU

GPU and Video

RAM and Storage

AI Capabilities

Connectivity and Expansion

Display and Audio

Power Supply

Real-time Clock (RTC)

Price

Summary of Winners

• Processor and Performance: NanoPi R6C
• GPU and Video: NanoPi R6C
• RAM and Storage: Tie
• AI Capabilities: Raspberry Pi 5 Kit
• Connectivity and Expansion: Raspberry Pi 5
• Display and Audio: Tie
• Power Supply: Tie
• Real-time Clock (RTC): Tie
• Price: NanoPi R6C

Conclusion

• Overall Winner: NanoPi R6C offers excellent value with competitive performance, advanced video capabilities, and a more affordable price.
• Raspberry Pi 5 Kit: Excels in AI capabilities and connectivity, making it a strong contender for specific use cases requiring AI processing and wireless communication.

The final choice depends on your specific needs, such as AI processing power, video capabilities, and budget.

What about ....

Post publication of this article I started looking at SBC add-ons, specifically in the M.2 form factor. To note, that that AI kit for the rPi 5 which does NPU is currently the cheapest and ready to market which is the Hailo 8L (13 TOPS).

But, what really interested me was the possibility of leveraging the other beefier models that could help meet the AI PC/Copilot+ PC designation of 40 TOPS. And for that the Hailo 8 Century has in spades at 208 TOPS in a 16-lane PCIe slot. Its also possible to perhaps make a cluster of Hailo 8R (13 TOPS) in the mPCIe form factor.

As for Coral, unfortunately at the moment the dev boards are using TPU which is great for the more general ML and PyTorch but not directly comparable to NPU. You may, however, still be interested in picking up one or more of these prototyping boards if your applications are more geared to visual training such as for object identification. I'm actually considering to use the USB version in a Frigate set-up to augment the person identification of my smart home automation and security projects.

This entry was intended to be Part-2 of my most recent report but realized on review that it is probably the natural follow-up to the other writings on the current state of LLMs (we've not even gone down the road of Small Language Models, yet!) in the first half of 2024. Also, isn't it apt that Star Wars: Episode II is titled "Attack of the Clones"?

Measurement numbers

So, how do we do this?

Let's start with the overall measurements. There performance metrics can be grouped into four-(4) and are described as:

• Total Duration: The total time taken for each model to complete the tasks.
• Load Duration: The time taken to load the model.
• Prompt Eval Count: The number of tokens evaluated in the prompt.
  • Prompt Eval Duration: The time taken to evaluate the prompt.
  • Prompt Eval Rate: The rate at which tokens are evaluated in the prompt (tokens per second).
• Eval Count: The total number of tokens evaluated.
  • Eval Duration: The time taken to evaluate the tokens.
  • Eval Rate: The rate at which tokens are evaluated (tokens per second).

Here are two-(2) really good inference measuring articles from Baseten and Databricks. While, there is guidance in terms of hardware, quantization (model re/compression), power, batch sizing, network speed – and the like; remember that we're sort of stuck on some of these given the aforementioned base hardware. Again, thats fine, since this has afforded us a semi-stable baseline to compare things from in this discussion. When and if we need to map it against cloud cost then we can simply make an assumption for apples-to-apples comparison.

Mapping the metrics

Moving on. We're going to concentrate on these specific metrics:

1. Time To First Token (TTFT) - When streaming, it is literally how soon users start seeing the first character of a model's output after entering their query. Low waiting times for a response are a must in real-time interactions, but less so obvious in offline workloads. This metric is derived from the time required to process the prompt and then generate the first output token.
2. Time Per Output Token (TPOT) - This is amount of time needed to generate an output token for each user that is querying your system. This metric corresponds with how each user perceives the "speed" of the model. This translates to how fast the output is able to be processed and converted to the speed and cadence at which a response displays on screen in milliseconds.
3. Words Per Minute (WPM) - In Part-1 of this performance test we used the formula for calculating Words per Minute (WPM) based on Tokens per Second (TPS).

Cost calculations

To estimate the potential cost of running the models based on the data, we need to consider several factors, including the compute time, instance type, and hourly cost of the compute resources. Since the performance metrics such as total duration, load duration, and evaluation duration are known, we can use these to approximate the compute time.

Assumptions: Google Cloud Platform
Instance Type: NVIDIA V100 GPU instance.
Hourly Cost: Approximate cost of an NVIDIA V100 GPU instance is $3/hour.
Total Duration: Includes the load duration and evaluation duration.

Example calculations: myllama2
Total Duration: 56.044795989 seconds
Compute Time (hours): ( \frac{56.044795989}{3600} = 0.01557 ) hours
Cost: ( 0.01557 \times 3.00 = $0.04671 )

When Cost($) is factored-in, however, the sorting goes like this:

• Primary Criteria: Cost (lower is better)
• Secondary Criteria: TTFT (lower is better)
• Tertiary Criteria: TPOT (lower is better)
• Quaternary Criteria: WPM (higher is better)

All things considered, Mistral 7B is the most economical with metric performance around the upper third of the pack. This is unsurprising as apart from cost, it's performance when combination with Llama 2B is what used to power Perplexity AI.

Llama 3 and its derivatives rank lower in the cost spectrum. However, that needs to be noted here is that this model shines in the conversational length side of things when temperate and top_p are tuned. I was really impressed by its chat capability the first time I tested this model on Ollama, granted it was at temperature = 0.8

Note that when planning for your particular infrastructure and investment costs to plug-in your current real values to get a more accurate picture!

Overall observations

1. Time to First Token (TTFT) in seconds: Lower is better. Code Llama and is the quickest to start generating output.
2. Tokens Per Output Token (TPOT) in seconds: Lower is better. Mistral leads in accuracy with 0.1461 tokens per second.
3. Words Per Minute (WPM): Higher is better. Phi3 is the fastest in word generation at 2851.2 WPM.
4. Cost per use in USD: Lower is generally better. Mistral is the most economical at $0.0151 per use.
5. Speed vs. Accuracy vs. Cost: There's a clear trade-off between these factors. For example, Llama3-8bi is the slowest to start (highest TTFT) and generate tokens (highest TPOT), but it's among the more expensive options.
6. Performance variations: Llama 3 variants generally have higher costs but show varying performance across other metrics.
7. Cost range: Prices range from $0.0151 (Mistral) to $0.3691 (Llama3-grad-8bi-1m) per use.
8. Specialized models: Code Llama excel in quick start times, which could be beneficial for certain applications.

Some key takeaways

It keeps being said that this space moves quite fast, but at some point one has to stick a line in the sand to make a start. There is no perfect set to rule them all.

1. Parameter tuning has an impact. Playing around with a model's temperature (top_k and top_p) can definitely help optimize for performance and may even save on costs when properly tuned.
2. Model characteristics trade-off. Using the right model for the right task will greatly benefit the stage at which it is being implemented. You may want to use a fast response model but be aware its responses could be less accurate.
3. Special model capabilities and Diversity. Some implementations try to toe the all-purpose function but it may be more optimal to use a model trained and tuned to a specific task, such as coding or numbers. This could spell a difference in latency output depending on the objective as well accuracy and response veracity.
4. Newer model functionality vs Re-work. Some models now allow natively extending capabilities using function calling agents, making it more performant and worth the extra effort to replace a current working model. Consider mixing and matching sections of your flow to use the most appropriate model.
5. Cost-efficiency considerations. Sacrificing a little cost to gain better performance could be worth it in the long run. Model training is a different class all itself and while you might get away with using the cheapest it might come back later in having to validate against hallucinations or other unintended artifacts.

In summary, hopefully this set of experiments leads you to a path of using more relevant evaluation questions as these are only a partial list of insights based on the data. Lastly, if you have a possibly larger set of observations that could be true in a specific use case, such as some longer term accuracy observations on a specific domain of knowledge, then definitely make use of it as that will better tune a model or a set of varied working models (recommended) for what you need to accomplish.

Remember, the more you know ... and, that more knowledge is power.

Microsoft Build 2024 just concluded last week and one of the highlights was the catch-up release of OpenAI GPT-4o alongside a new hardware line-up of Copilot+ PCs running Windows on ARM with Qualcomm Snapdragon X chips under the hood.

I'll address the huge x86 elephants that were in the room separately. For now, let's focus on the GPT side of things.

Test prepping

Since there aren't local models for GPT-4 and we want to find out which of the available Ollama models are close enough for usage, I proceeded to create new models using the recommended process:

All things to be equal

While all my previous model performance tests for Mistral, Llama 2, Code Llama, and Llama 3 were run using defaults, I decided to normalize the outputs for a balanced chatbot. That means we have to mess with temperature, top_k, and top_p. The default parameter values for Ollama are described in modelfile.md in its GitHub page.

Notice that temperature = 0.8 and top_p = 0.9? These are, accordingly, rather high. While those values would be great out of the box if you were to use Ollama as a chatbot, Llama 3 instruct and derivatives will simply take you for a nice and very long ride if you asked about the sky color. Per the ChatGPT Cheat Sheet these values should be at 0.5. And so, this is what I did for the model file (which can be named anything):

Testbed adjustments

One mistake I did, not reading further enough why it mattered, was not dumping and copying the other parts of the base modelfile. This was easy enough to do for each model that is included in this quick test:

The second mistake, at least in terms of building the modified model, is to refer to the model directly. The correct way, to avoid any confusion, is to refer to the exact path of the downloaded model per the model file dump details. So, all together now:

Making a baseline

And now, for a quick comparative test using Mistral. What is noticeable from the modified model is the lower: total duration, prompt eval duration, eval count and eval duration. Conversely, there is a higher: load duration, prompt eval rate, and eval rate. Also, that explanation seems about 50% shorter than the original. What we can infer from this result is that for Mistral a higher set of parameter values would be better suited to make it less of a text-book like response.

Let's do the same for Code Llama:

Everything in terms of resources is higher, not good? However, take a look at the text details of the verbose results which provides an extra scenario example, that's good as a chatty response. Also, I do like the inclusion of numbers from the base model. Do note that Code Llama is supposed to be built for, ehm, coding. In this case it, also, looks like higher parameters would be more beneficial for this particular model to speed-up the response but would perhaps slightly compromise a slightly more detailed response. We should probably do some code completion tests as a future separate exercise and see what's what.

Local LLMs, all together now!

Alright. Now, let's get the rest of evaluation results across all modified models:

Enter the cloud

Also, here are the results of the same question from Azure OpenAI Playground. To the left is GPT-4 and to the right is GPT-4o:

While the responses are quite similar, GPT-4o appears to extract an extra explanation (point #5) by clarifying the answers from (point #3 and #4) of the GPT-4 response.

Here's an extra point, I went all in and raised the temperature = 1.0 and it responded with a slightly terse version. To me, that is like diminishing returns when the results are an almost exact response as from 0.5. See, thus:

The incumbent responds

And, finally, here's the gold-standard comparison itself from none other than OpenAI GPT-4o for Everyone's offer (temperature and other parameters as secret sauce!):

The other guys

For completeness, and because I can for kicks, how about some guest (not their best model foot forward) data comparisons from Google Gemini, Anthropic Claude AI (Sonnet), Perplexity AI, and Meta AI?

That's the test data. So, let's discuss the results in next part of this blog entry:

What have we learned so far in terms of all the model responses based on this baseline test question: Why is the sky blue?

Updated: 2024-June-03 with Meta AI output

We'd had some marble sized hail in the afternoon, followed by a big storm last night which upturned the gazebo chair, and then the power went out which left me a bit stuck at home due to the weight of the garage doors.

That being said, this site should be pretty much back online right now. I've had the whole morning trying to remember how I was able to get it going the last time.

See, I no longer have the luxury of a triple ISP vendor connection at home. While that sort of sounds amazing it came with a whole lot of power draw due to servers running at the old home base in Cupertino.

I've also moved away from a static IP connection at home, what with moving on from being on eBay and moving servers back to the office as I transplanted the whole family to the Dallas area. And out with that, too, went my IPv6 hosting dreams.

That means dealing with whatever dynamic IP address my ISP tips my way. But, ever the tinker, I found that some router, switch, or NAS vendors actually have their own shared cloud hosting. I've been able to do some things that way.

That is, until I stumbled across Cloud flared Warp, err, Argo (now). More on this, later.

Excuse thumbs, am blogging on a phone and enjoying the fruits of this blogging platform that I got running from a Turnkey Linux Ghost LXC package.