Claims that an unlearning method is robust typically rest on a relearning attack: the unlearned model is briefly fine-tuned, and one checks whether the erased facts resurface. We show that this measurement is fragile. In a controlled study on Llama-3-8B with the Sports Facts task, we identify six pitfalls: recovery scores that reflect how deeply a model forgot rather than how well it resists; attacks that train on facts the model still knows; attacks that degrade the model itself; attacks that corrupt even the original model; accuracy metrics that mask a  collapsed or merely suppressed model; and statistical baselines that turn noise into significance. We propose a remedy for each and use them to re-examine a published localization claim, namely that unlearning only the MLP layers identified by a probe (FLU) is more robust than unlearning equally many random layers. Under the corrected protocol the claim is fragile. At a matched forgetting level, FLU recovers fewer facts than random layers in one seed (10 vs. 16 of 16, p = 0.03) but ties in another (14 vs. 14); the pooled difference is not significant (p = 0.07), and the gap vanishes with 64 forget facts and under NPO. Sharpness-aware minimization (SAM) yields shallower forgetting and no reliable robustness gain. We conclude with a checklist for robust-unlearning evaluations.

machine unlearning,Large Language Models,Relearning Attacks,localization,sharpness-aware minimization,evaluation methodology