ai LLM

 

I got some pretty interesting results from the latest Gemini test. I had Gemini 3.8 Flash / high perform the remediation work, then had Opus 5 / high independently audit what it did.

 

The big takeaway is that Gemini 3.8 seems **materially better at the actual technical work** than my earlier experiences with Gemini 3.6 Flash and 3.1 Pro. It correctly addressed all six findings from the previous audit, substantially expanded the focused tests, and importantly left the production MMIO implementation completely unchanged where it was supposed to. Opus later confirmed that the remediation itself was good, so this wasn't a case of Gemini producing something that merely looked plausible.

 

Where Gemini still stumbled was the process around the work. After doing the remediation, it essentially turned around in the same context and declared itself an "independent auditor," despite the task specifically requiring a fresh independent audit. It also never clearly completed/reported the full pytest suite. So while the actual changes were good, I still couldn't treat Gemini's own final verdict as authoritative.

 

Opus's independent verification was considerably more rigorous. Gemini had reported catching 9/9 targeted mutations; Opus expanded that to **37/37 mutations caught**, covering a much wider range of ways the implementation or tests could have been broken. It also completed the validation Gemini left unclear: **8,249 full-suite tests passed**, along with 52 Translation 0.20 tests, 206 Language 0.23 tests, 476 predecessor/static-storage tests, and clean Ruff, Black, and mypy checks. It also did a stronger repository-integrity check to make sure nothing unexpected had been modified or left behind.

 

So my current impression is that **Gemini 3.8 Flash / high may actually be becoming a pretty capable implementation/remediation agent**, provided somebody else verifies its work afterward. That's a noticeable improvement over my experiences with Gemini 3.6 Flash / high and 3.1 Pro / high.

 

**Opus 5 / high is still in another league as an auditor**, though. It was both more adversarial technically and much better at following the process contract. Gemini understood the rules, but then managed to violate one at the orchestration level by calling its own same-context review "independent."

 

The funny part is that Gemini's *code work itself may have been entirely fine*. What disqualified its result was mostly the scaffolding around it. It's increasingly looking like a competent mechanic whose clipboard occasionally signs its own inspection form. 😄