Why were southern slaveholders anxious about the loyalty of white southerners who did not own slaves?

When Lincoln was elected as President, it was clear to slaveowners that there would be efforts to abolish slavery. White southerners who did not own slaves had nothing to lose, therefore southern slaveowners worried their loyalty would sway towards the anti-slavery movement. This would be detrimental to slaveowners as they would not only have Northerners trying to take away their slaves, but their fellow neighbours as well. 

Hope this helped!