User-Agent: *Allow: /Disallow: /cgi-bin

User-Agent: W3C-checklink Allow: /
Disallow: