The first automated webcrawler or robo, was the World wide web
wanderer created by MIT student Mathew Gray.
Martin Koster created the first web directory. ALIWeb in late 1993, but
it, like the wanderer, met with limited success.
1994 was a
big year in the history of web search. The first hierarchical
directory Galaxy, was launched in January and, in April
Satndford students David Filo and Jerry Yang created ‘Yet
Another Hierarchical Officious Oracle'., better known as
YAHOO.
During the same month, Brian Pinkerton at the University of Washington
released WebCrawler. This is the first true web search engine,
indexed the entire content of the web pages, where previous
crawlers had indexed little more than page titles, headings
and URLs. Lycos was launched a few months later.
