How to Deal With Nuisance Neighbors