Test Cases: https://books.google.com/robots.txt Disallow: /search Allow: /search/about https://www.cbc.ca/robots.txt https://www.cbc.ca/sports/all/ (allowed) https://www.cbc.ca/sports/includes/ (disallowed)