Two results we can hand you the evidence for. One is a model that works. The other is a model we improved, and broke, and only found out because we went looking.

OpenGrad · Study 001

Qwen3.5-2B

+0.1357

call F1 on tool use

0.6191 → 0.7548, and over-calling fell from 64% to 15%

then

100%

of bare arithmetic questions refused

All 1,319 of them. The base model refused none.

Visual-Qwen

Qwen3-4B + LoRA

99.0%

sludge told from ordinary short-form video

On held-out clips, from a 2M-parameter projector and LoRA adapters trained on one H200.

and

+10.0pp

of that comes from the audio

Vision alone reaches 89.0%. The Whisper transcript adds ten points.

OpenGrad · Study 001 · Qwen3.5-2B

We improved the metric we were optimising. Then the model stopped answering questions.

A checkpoint passed a pre-registered promotion gate on tool use, and was promoted. It was also materially worse than the model it started from — and the gate could not see it.

0%20%40%60%80%IFEval strictrefuses 19.0% here67.8%45.1%-22.7ppGSM8K 8-shotrefuses 0% here70.4%56.3%-14.1ppMMLU-Pro 5-shotrefuses 0% here49.0%37.0%-12.0pp
Base Qwen3.5-2BAfter our supervised fine-tuning
Real upstream benchmarks at pinned revisions, scored by upstream checkers. Every stage ran on one engine configuration, so no gap here is attributable to the runtime. On the bottom two rows the model refuses essentially nothing, which is what makes them capability results rather than a restatement of the refusal behaviour.
Tool-policy score and general capability for each checkpoint.
Checkpointcall F1IFEvalMMLU-ProGSM8K 8-shotrefuses 0-shot
BaseQwen3.5-2B, no SFT, no DPO0.619167.8%49.0%70.4%0%
M0 · SFTwhere every regression first appears0.747045.1%37.0%56.3%100%
M1-v2 · DPOpromoted checkpoint0.754845.8%37.0%55.5%100%

The win was real, and it is the leftmost column. Everything else moved the wrong way. Preference optimisation is not the culprit — 82% of its GSM8K outputs are byte-identical to the checkpoint before it. Every regression here first appears one step earlier, at supervised fine-tuning.

Read Study 001

Refusal and capability are two different failures.

The same 1,319 questions, twice. Asked bare, the promoted checkpoint refuses every one. Put eight worked examples in front of the identical question and it refuses none — and still lands below where the base model starts. The ability is not gone, and it is not intact either. Both had to be measured separately to know that.

100%
refusedasked bare
0%
refusedwith 8 examples
55.5%
solvedwith 8 examples
67.4%
solvedbase, asked bare

Why the gate missed it

The held-out set is 3,650 examples of tool use, and contains 0 examples that simply ask the model to answer. A set with no answering in it cannot fail a model for refusing to answer.

The likeliest cause, unproven

21,749 of 217,903 training records teach a refusal as the correct answer, every one labelled ANSWER. That is an association. We have not shown it is the cause, and the regex that counts it has not had its precision measured yet.

What it cost to find

$11.39 across 13 GPU runs. Every number above recomputes from 48,840 per-example records without a GPU, including the runs we later invalidated and kept.

Audit, including what it retracted
audio transcript, from Whispertalking clipgameplaysludge

Visual-Qwen

Tells sludge from ordinary short-form video with 99.0% accuracy on held-out clips.

A frozen vision tower feeds 32 query tokens into Qwen3-4B through a 2M-parameter projector. The clip's Whisper transcript joins the same token stream. Only the projector and LoRA adapters are trained, on one H200.

Vision alone reaches 89.0%. Adding the transcript adds ten points.

  1. EVA-ViT-G/14frozen257 × 1408
  2. Q-Formerfrozen32 × 768
  3. Linear projectortrained32 × 2560
  4. Whisper V3 Turbotranscripttokens
  5. Qwen3-4B + LoRAtrainedverdict

Training run for a 151M-parameter model on one H100, 13.7 hours.

Pretrain7.26 hcross-entropy loss3.04Midtrain1.76 hcross-entropy loss2.29SFT3.78 hcross-entropy loss1.36DPO0.39 hpref. accuracy0.59GRPO0.49 hverified reward0.23
Unedited step logs from the first full-scale run, downsampled. Loss stages read on a 0 to 10 axis; DPO and GRPO read on 0 to 1. The dashed red line is the learning rate, peak 6e-4.
Afterval pplHellaSwagARC-eMMLUGSM8K
Pretrain20.8538.5%45.8%24.0%1.0%
Midtrain19.7037.3%45.1%26.6%4.0%
SFT25.1137.4%48.1%23.8%1.0%
DPO25.0737.3%47.9%24.3%2.5%
GRPO25.1037.3%47.7%23.8%1.5%

The learning rate holds at its peak through pretraining and decays in midtraining, which is what separates midtraining from a second pretraining run. At this size, post-training changes behaviour more than benchmark scores.

10B tokens on one H100. Every objective is plain PyTorch.

Code

Sludge dataset

2,000 TikTok and YouTube clips labeled sludge or not, plus a 262-clip live out-of-distribution evaluation set. The training and test data behind Visual-Qwen.

Download on Kaggle
2,000
Clips
2
Classes
300
Test clips
262
Live OOD clips
33 GB
Size
  • Taglish sentimentThree-class sentiment for Tagalog, Taglish and English. Contamination-audited data, int8 ONNX for CPU.published
  • Touch-native vision-action modelA foundation model for mobile games that learns continuous touch from screen recordings, then runs on the phone it plays on.planned

Working on something similar?