In Armitt's book, she mentioned the idea of world building, especially in J.R.R. Tolkien's Lord of the Rings. Armitt however mentions how there were always be generic limitations. World building is something that comes straight from the imagination, whether it be from the author or reader.
Saying there is limitations of world building, in my opinion, is incorrect. It would be saying there is limitations on the imagination. Mo matter how much detail one can put into a story, there will always be some sort of gap or space for more development.
Armitt stated: " there is nothing beyond Middle Earth because Middle Earth is nowhere." Though Middle Earth may not exist in terms of human reality, it does not mean it doesn't have a substantial history and exist in one form or the other. At risk of sounding cliche, the world only exist in the heart and mind of the person enjoying the book at that time. The world building gives the Middle Earth a place, even if it was in the imagination. Giving history, giving a name to something gives it purpose and placement within the lives of people.