Is it wrong for people in England to prefer hearing English spoken in their own country?

It is sometimes suggested that there is something xenophobic or racist about English people wishing to work with, and live next door to, people who speak their own language and share their culture.