Human observers are able to keep track of several independently moving objects among other objects. Within theories of multiple object tracking (MOT), distractors are assumed to influence tracking performance only by their distance toward the next target. In order to test this assumption, we designed a variant of the MOT paradigm that involved spatially arranged target-distractor pairs and sudden displacements of distractors during a brief flash. Critically, these displacements maintained target-distractor spacing. Our results show that displacing distractors hurts tracking performance (Experiment 1). Importantly, target-distractor confusions occur within target-distractor pairs with displaced distractors (Experiment 2). This displacement effect increases with an increasing displacement angle (Experiment 3) but is equal at different distances between target and distractor (Experiment 4). This finding illustrates that distractors influence tracking performance beyond pure interobject spacing. We discuss how inhibitory processes as well as relations between targets and distractors might interfere with target tracking.