If the wall collapses, or is proven useless by the white walkers just walking around the thing, then what's the point of the men in black? Just give the area to the free folk (who would never follow the night's watch long term anyway) with an agreement that they are the first line of defense and can call in reinforcements from the South in the next apocalypse.
Moreover, Sam just stealing rare manuscripts and leaving the Citadel is a real turning point for his character arc. He chose pragmatism over duty (very Randal Tarly like, and different than Dickon who should have shut the F up when his dad burned, so he could fight another day). Also, there's no way he gives up dipping his stick in the Gilly. Sam's a man now. Sam the Slayer!!! Watch out Westeros!