To understand SEO you need to be aware of the architecture of search engines. They all contain the following main components:
Spider - a browser-like program that downloads web pages.
Crawler – a program that automatically follows all of the links on each web page.
Indexer - a program that analyzes web pages downloaded by the spider and the crawler.
