One does not wear shows in a mosque, or inside people's homes. This is a social custom. Like Gentile American politicians wearing yarmulkes when they are photographed in Israel.
If you go to the Taj Mahal, you will not be allowed to wear your shoes inside there, either.
When in Rome...
What you might say is "I was unaware of a common social custom of 40% of the human race", but I doubt that you will.
