User-agent: * Crawl-Delay: 10 allow: / Disallow: /june06/ User-agent: SemrushBot Disallow: /terms-and-privacy