QUAM-SM is a post-hoc framework that uses adversarial search to identify 'adversarially fragile' pixels in medical image segmentation. It disentangles epistemic and aleatoric uncertainty and outperforms existing methods in reliability and boundary sensitivity on public datasets with expert annotations.