skip to Main Content

MSN now crawling JavaScript

Last April, Google began crawling JavaScript to get at more data and now MSN has quietly launched a new spider to crawl JavaScript. This is great for all the sites that use JavaScript in their navigation, but could pose a problem for some sites wanting to keep certain information out of the search engines. They’ll need to exclude the new robot (along with Googlebot) in their robots.txt file.

This Post Has 0 Comments

Leave a Reply

Your email address will not be published.

Back To Top