This Working Droid Episode 72 Robots.txt
Your browser doesn't support HTML5 audio
Episode Guide
- Keep websites and individual pages hidden from search engine crawlers
- Tells search engines how much or how little to crawl
- Not required to follow by any crawlers
- Wikipedia
E-Mail me or fill out our online contact form!
Support Geek I/O on Patreon!
HeadphonesNeil Amazon Author Profile
[Sponsor - ProXPN] Visit http://proxpn.com and use promo code 'GEEK20' to get 20% off your order today!