User-agent: * Disallow: /wp-admin/ Allow: /wp-admin/admin-ajax.php Disallow: /redirect.php Disallow: /users/register Sitemap: User-agent: MJ12bot Disallow: / User-agent: SemrushBot Disallow: / Disallow: /redirect.php Disallow: /users/register User-agent: AhrefsBot Disallow: /