The Gauntlet: Running a Bespoke 7-Task Evaluation Harness Across 20+ Models

With our local directory junction surgery complete and 350GB of duplicated GGUF cache bloat thoroughly expunged from the NVMe storage array, the infrastructure bounds were finally behaving themselves. But what good is a lean, sovereign local inference registry if the models you run inside it still act like corporate customer-service reps?

Time to put our core local model registry through a brutal evaluation loop. Let’s look at the messy reality of why academic benchmark theater fails under real-world threat intelligence constraints.

The Catalyst: Impatience with Academic Safety-Theater

Let me start by saying that I am completely done with standard corporate evaluation metrics. Hyperscalers love bragging about high MMLU scores, HumanEval coding compliance, and rigid safety alignment. But inside a legitimate ThreatLabs CTI pipeline, those polished cloud metrics translate to an unmitigated disaster. You point an n8n workflow at a cloud API to parse a raw dark web threat report containing active cartel operational security practices or malware scripts, and the model flinches, refuses to process the data, and gives you a patronizing corporate lecture on safety compliance.

We don't need a model trained to act like an HR representative; we need a cynical forensic specialist who doesn’t blink at raw malicious payload variables.

So, I built a bespoke 7-task automated validation harness inside our cti-net Docker space to test actual threat intelligence extraction under fire. No academic theory here—just raw script parsing, STIX 2.1 JSON schema alignment, and aggressive refusal-free tracking. Deployed the suite across 20+ models between February and July, and the casualties were heavy.

The Technical Frustration: CoT Suppression Bugs and SYCL Cache Traps

The execution script seemed straightforward on paper, but running these validation loops locally on our newly minted Intel Arc Pro B70 workstation turned into pure runtime friction.

First, the deep reasoning models completely lost their collective minds trying to parse complex TTP strings under strict constraint parameters. The prompt instructions required structured JSON output, but the moment a reasoning model encountered an abliterated or uncensored tune, it fell into a catastrophic Chain-of-Thought (CoT) suppression bug. Instead of thinking logically through the malware indicators, the suppressed weights caused the model's brain to loop indefinitely, outputting garbage syntax, infinite repetitions, and breaking output format bounds entirely.

Worse, when driving these models locally through llama-server.exe using the SYCL processing backend, the compute engine kept hitting a hard persistence blindspot. The compilation cache handles would randomly lock up or bork between model swaps. You'd load a 26B quantized orchestrator right after a 7B triage subagent, the persistent compilation state wouldn't refresh cleanly, and the Level Zero graphics drivers would throw a system fault, tanking your inference metrics down to zero tokens per second.

Friction debt at its absolute finest. I had to forcefully kill orphaned llama-server handles on the Windows host and clear out the raw device cache layers manually before the binary engines would behave.

The Fix: Ripping Out the Sp sprawl

The "Aha!" moment happened when I looked closely at the logic tradeoffs we were making. Trying to preserve highly marketed "cybersecurity specialist" fine-tunes like WhiteRabbitNeo or aggressive uncensored merges was a massive waste of disk space; they suffered from severe hallucinations, failed basic tool-calling JSON schemas, and routinely ignored output format rules.

We needed a massive registry purge to save storage and stabilize the local VRAM footprint. Cut the fat ruthlessly, keeping only the leanest, most logically sound Quantization-Aware Training (QAT) variants that preserve reasoning at lower bit rates.

The Blueprint

To prevent compilation state traps from locking up the B70's 32GB VRAM pool between model swaps, I injected an explicit host-level process termination wrapper directly into our backend launcher configurations:

# Forcefully clear out active llama-server child trees to release locked VRAM handles
taskkill /F /T /PID $(pgrep llama-server)

# Launch the primary uncensored orchestrator running optimized SYCL parameters
llama-server.exe -ngl 99 -c 65536 --port 8080 --host 0.0.0.0 --tools all --metrics \
  -fa on -ctk q4_0 -ctv q4_0 -t 8 -tb 8 -b 2048 -ub 512 \
  -hf mradermacher/gemma4-ara-2pass-APEX-Q5_K_M.gguf

The Final Standings Table

After testing over twenty configurations across 500GB of weight casualties, we consolidated the entire ThreatLabs registry down to an elite local core fleet. Here is the exact performance breakdown of the kept fleet versus the pruned sprawl:

Voila! The final stands are locked, the local inference node is behaves beautifully, and we have a local model pipeline that can rip through raw incident response logs without needing corporate permission strings.

The 11-part local intelligence stack engine is officially online.

Until then, see ya later. Happy tinkering!