When we build a world for our fictions, games, and shows, we construct cultures. Cultures explain why people do what they do, how they think, why they eat, how they war, how they make peace, and more. Culture is unavoidable when you make a setting – one could even argue that characters are often expressions of their cultures.
When we build these cultures we're often thinking about important things. We want to know why characters believe as they do. We want to explain why characters go where they go and do what they do. We want to explain why magic is the prerogative of the ruling class, why there's a different language spoken on this distant space station, and so on. We want to explain what matters – in short, cultures explain the Big Plot.
This is short-sighted.