How does crawling my site work?

It starts with the sitemap. When there is no sitemap, it starts at the homepage and follows the internal links from there.

robots.txt is respected: a page blocked there never gets read.

What it records is what builds the map: the URL, the server response, the title, the text, and above all, who links to whom. That last part is where the Citadel and the Wilds come from.