This is one of the most commonly asked cat questions there is and the answer is very simple when you think about it from the cat's point of view.
Cats don't like being stared at:
People who love cats will attempt to engage with them. Picture this, you visit a friend who's cat is quietly sitting there observing, and you want a cuddle, what do you do? You lean forward, speak in a high tone, look directly at the cat and call him over. To a cat, direct eye contact is a sign of dominance and can be viewed as threatening behaviour. He doesn't like it and will avoid you.
Meanwhile, the visitor who doesn't like cats quietly sits down, avoids all eye contact with the cat and hopes the cat will stay away. He is behaving in such a way that is viewed as non-threatening towards the cat. So who does Fluffy go to? The non-cat lover.
Eye contact with cats is interesting though, if you make direct eye contact with a cat, and blink very slowly, this is seen as friendly behaviour. You are telling the cat that you are not threatening, and if you watch carefully, your cat may blink slowly back at you. This is the ultimate compliment.
So next time you visit a friend with a cat, sit down, look at the cat and slowly blink at him, or you could try ignoring him. Either way, you will be more likely to get his attention.
Ailurophilia is the love of cats. Ailurophobia is a fear of cats.
I can't explain why light coloured cats seem to be attracted to people in dark clothes. That's for another time.