{"id":3522,"date":"2019-11-16T15:18:19","date_gmt":"2019-11-16T07:18:19","guid":{"rendered":"https:\/\/www.ookangzheng.com\/?p=3522"},"modified":"2020-05-24T15:58:06","modified_gmt":"2020-05-24T07:58:06","slug":"prevent-baiduspider-from-crawling-your-site","status":"publish","type":"post","link":"https:\/\/www.ookangzheng.com\/prevent-baiduspider-from-crawling-your-site\/","title":{"rendered":"Prevent BaiduSpider from crawling your site"},"content":{"rendered":"\n<p>Method 1: User-agent block<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\/\/ Nginx conf\n\nif ($http_user_agent ~* ^Baiduspider) {\n  return 403;\n}<\/code><\/pre>\n\n\n\n<p>Method 2: robots.txt<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>User-agent: Baiduspider\nDisallow: \/\nUser-agent: Baiduspider-video\nDisallow: \/\nUser-agent: Baiduspider-image\nDisallow: \/<\/code><\/pre>\n\n\n\n<p>Method 3: Block IP<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code># Baidu crawlers\ndeny 123.125.71.0\/24;\ndeny 180.76.5.0\/24;\ndeny 180.76.15.0\/24;\ndeny 220.181.108.0\/24;<\/code><\/pre>\n\n\n\n<p>Method 4: .htaccess<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>order allow,deny \nallow from all\n# Block access to Baiduspider \ndeny from 180.76.5.0\/24 180.76.6.0\/24 123.125.71.0\/24 220.181.108.0\/24 <\/code><\/pre>\n\n\n\n<p>Photo by\u00a0<a href=\"https:\/\/unsplash.com\/@artnok?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText\">Nicolas Picard<\/a>\u00a0on\u00a0<a href=\"https:\/\/unsplash.com\/s\/photos\/spider?utm_source=unsplash&amp;utm_medium=referral&amp;utm_content=creditCopyText\">Unsplash<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Method 1: User-agent block Method 2: robots.txt Method 3: Block IP Method 4: .htaccess Photo by\u00a0Nicolas Picard\u00a0on\u00a0Unsplash<\/p>\n","protected":false},"author":1,"featured_media":3525,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[37,32],"tags":[],"class_list":["post-3522","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-linux","category-technology"],"_links":{"self":[{"href":"https:\/\/www.ookangzheng.com\/wp-json\/wp\/v2\/posts\/3522","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.ookangzheng.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.ookangzheng.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.ookangzheng.com\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.ookangzheng.com\/wp-json\/wp\/v2\/comments?post=3522"}],"version-history":[{"count":0,"href":"https:\/\/www.ookangzheng.com\/wp-json\/wp\/v2\/posts\/3522\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.ookangzheng.com\/wp-json\/wp\/v2\/media\/3525"}],"wp:attachment":[{"href":"https:\/\/www.ookangzheng.com\/wp-json\/wp\/v2\/media?parent=3522"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.ookangzheng.com\/wp-json\/wp\/v2\/categories?post=3522"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.ookangzheng.com\/wp-json\/wp\/v2\/tags?post=3522"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}