It could be an irritant. And it could effect lining up putts, etc.Shouldn't make a difference IMO.
Back when I used to be real good at golf my Dad asked me what I think about when i'm swinging the club. I said "I dont know, once I start my down swing I kind of black out". Another reason why i've never had a problem with people talking during my swing.