AI Digest, Oct 29: AI Peer Review, Private Benchmarks, Agent Cloud Costs

0:02PM Lead ProThis is PM Lead Pro. Today's digest covers AI peer review, private benchmarks, the cost of running agents, and China's chip workaround. Worth a listen if you decide where AI budget goes.

0:17Imagine writing a scientific paper, submitting it for peer review, and then getting rejected by an AI.

0:25Yeah, which is crazy to think about.

0:26Right. But then finding out the AI actually exposed how heavily biased the human reviewers were.

0:32I mean, that is not a sci-fi pitch. That is happening right now.

0:36It really is.

0:37So today, our mission in this deep dive is to catch you up on the frontier of AI research.

0:41We are looking at a trust crisis in AI benchmarking, a massive shift in cloud economics for autonomous agents, and how China's engineering around hardware bans.

0:52Yeah, the pace is just relentless right now. And I mean, the common thread here is really maturation.

0:57How do you mean?

0:58Well, the industry is moving past just proving these systems work. Now we're wrestling with how we reliably measure them, how businesses can actually afford them, and just the sheer physical hardware needed at scale.

1:10Exactly. So let's start with the measurement side. Marginal Revolution had this piece highlighting that Clawed Opus 4.8 generates peer reviews that will quantifiably beat well-compensated human experts.

1:23Right, which is a huge milestone.

1:25It is. But the wild part is the audit exposed massive human ideological bias. Like human errors favored the author's own ideology 97% of the time.

1:35Which just showcases AI as this incredible auditing tool.

1:39But paradoxically, we are facing a crisis in how we audit the AI itself.

1:45Wait, what do you mean?

1:46So Interconnect AI reported that the new artificial analysis V5 benchmark has actually shifted to using 60 private hidden data Oh because the open models were getting too good at bench maxing right

1:58Like just memorizing the public test questions.

2:00Precisely.

2:01So by moving to private tests, the performance gap between closed and open models widened drastically.

2:07Wow.

2:07Yeah, I mean, China's highly anticipated Kimi K3.1 Pro dropped to a score of just 46% when it couldn't see the test in advance.

2:15Okay, see, here's where it gets really interesting to me. If AI is grading humans, and secret benchmarks are grading the AI, who is verifying the secret tests?

2:24That is the exact uncomfortable question the ecosystem is facing.

2:27Right, because are we just losing transparency? And it feels like we're being asked to put blind trust in a few benchmark creators.

2:34Yeah, and if evaluating a single AI response is this complex, just imagine the infrastructural headaches for businesses trying to run thousands of autonomous agents.

2:43PM Lead ProFor a team lead: a vendor's benchmark score is not proof. When tests went private, Kimi K3.1 Pro fell to forty-six percent. Test models on your own tasks before you commit.

2:56Right, which brings us to a total rewrite of cloud economics.

3:01The agent report detailed AWS Infinity.

3:04Oh, this is a total game changer.

3:06Yeah, it's a serverless runtime that suspends an AI agent whenever it makes a tool call.

3:11So think of it like hiring a chef.

3:14You used to pay their hourly wage just to stand in the kitchen staring at the oven while the cake bakes.

3:18I love that analogy.

3:19AWS Infinity basically clocks the chef out the second the cake goes in and clocks them back in when the timer beeps.

3:26Exactly And by not blocking a worker process they managed to fit 75 idle agents into just 8 gigabytes of memory That is insane It is If you building an AI startup right now this flips your budget upside down You used to pay for process time multiplied by latency

3:45Now the economic bottleneck is state size multiplied by retention.

3:49But wait, if an agent is wanting a 500-turn conversation, doesn't this just shift our

3:55exorbitant costs from compute directly over to storage bloat, like paying for giant transcripts

4:01instead of idle processors?

4:03Yes, and that is the exact trade-off businesses will have to navigate now.

4:07Okay, so manage memory aggressively or go broke.

4:10PM Lead ProFor a company owner: with agents that pause during tool calls, the cost driver shifts from idle compute to state size and retention. Ask your team how much memory each agent keeps.

4:23Basically, yes. And keeping all those memory states active requires so much server space, meaning physical hardware demands are going to hit a ceiling.

4:31Which brings us to China. InfoQ reported on Tianqiu Zhixian releasing the Tiangai 300 GPU and the TY1200 Edge chip.

4:40Obviously, they can't import advanced Western chips, so how are they competing?

4:45Well, they can't rely on raw monolithic chip power because of the export bans, so they bypass it by linking specialized chips together in a heterogeneous approach.

4:54Meaning they split the AI's tasks.

4:56Right. GPUs handle the pre-fill stage, so processing the initial prompt, while XPUs handle the decode stage, generating the actual response.

5:04So they compensate for trade restrictions with clever system-level routing.

5:08They claim the Tiangai 300 gets 1.05 times the throughput of NVIDIA's H200 in these specific split setups.

5:16It's a brilliant architectural workaround.

5:18And the Edge chip hits 1 times the performance of NVIDIA Thor T5000 Unreal Okay let pivot to the model radar First up Google Gemini 4 Carbon is completely dominating cybersecurity

5:32It hit 88.2% on SecBench Pro with a 0% prompt injection success rate.

5:38Huge numbers.

5:39We're also tracking NVIDIA's Nematron 3.5 Transcribe.

5:42It's an efficient 4 billion parameter model for multilingual diarization running entirely

5:47locally on RTX Spark hardware.

5:50And for those unfamiliar, diarization just means the AI accurately identifies exactly who's speaking in a crowded room.

5:56But the final model, Qwenputer, this one broke my brain.

6:01Oh, this is a radical new class of model.

6:03Right. Divyach Agrawal compiled a 32-bit RISC-V CPU directly into the weights of a Qwen 3.5 model.

6:11Yeah, so instead of calling an external calculator, it uses its own internal architecture, specifically the gated delta net, as literal RAM.

6:19Wait, it just executes exact math and C code natively inside its own brain?

6:24Exactly, during its forward passes. Just a CPU living inside a neural network.

6:29Wow. Okay, so what does this all mean for you listening? The practical takeaway today is to

6:35audit your autonomous agent infrastructure. You should be paying for retained memory state,

6:40not idle compute time.

6:41Yep. And a final thought to mull over on your commute, if models like QNputer can run

6:46and Deterministic CPUs entirely inside their own weights, what happens to traditional software

6:51sandboxing and cybersecurity?

6:53That is a mildly terrifying new computing paradigm.

6:57Catch you on the next deep dive.

6:59PM Lead ProMore at pmlead.pro: the full transcript and sources are on the episode page. If a model can run a CPU inside its own weights, how would you rethink sandboxing for your agents?