# /robots.txt file for http://webcrawler.com/ User-agent: * Disallow: /tmp Disallow: /logs Disallow: /ventes_flash Disallow: /pub Disallow: /stats Disallow: /atos Disallow: /admin Disallow: /_admin_special Disallow: /functions Disallow: /includes Disallow: /library Disallow: /newsletters Disallow: /phpAdsNew Disallow: /phpmyvisites Disallow: /images Disallow: /medias Disallow: /blogcms