We could work some kind of catastrophe into the story, an event that caused the lost realms to slip between the cracks of our universe.
Edit : I wrote an origin story for the lost realms. Basically, TLR is a new universe that brought parts of other worlds into it. These parts include locations on Earth, other planets and even other dimensions (e.g. the end), which were teleported away from their normal positions to TLR, in an event I'm calling the sundering. TLR is basically a hybrid world of different places abducted from space and time and as such, contains people and creatures from different places and realities. Because of this, factions would be easy to create, as some entire factions could have been brought in by the sundering. We could have patches of land that resemble different places, even buildings that were only half teleported. We could have a sky realm, where the teleported locations exist as floating islands. Basically, if you end up in TLR, you've fallen through a crack in the universe and can never get back, so you must build a new life. Because of this story, we could include real locations, and locations from films and games. I wrote a short account of one of these teleportations, of a town called Athos, and the book is in a chest in the central tree.