懸賞wordpress英文站robots.txt 規則的完整寫法

" k/ N7 r1 C' i( ~3 ]& y
( W& |) o7 b( d懸賞wordpress英文站robots.txt 規則的完整寫法
/ l% j3 F% R0 b: i1 x' P& j% O. U按以下要求寫
+ ?7 M$ m, _  s* v( I1，只能Alexa, aol, Ask, Google、Yahoo、Bing的爬蟲訪問，其他的禁止
2，禁止訪問特定目錄，文件類型的通用寫法，適用於wordpress英文版3.6~4.1版本
( ~) I; y4 a8 \' |6 S3,  Sitemaps文件





+ L$ E( Y3 Y2 A  e( i! y
6 X- C) `0 ]; ?
: C/ n+ B" ~2 T

! ^0 K* I. c& h& O: W. [* z: Z2 M9 CUser-agent: Alexa*
Allow: /
" |+ A6 `( [! _2 H. b5 `User-agent: Aol*
Allow: /
User-agent: Ask*
9 ?+ b9 P# O* r) E* M; nAllow: /
9 `6 v* a0 z2 x, `5 ?User-agent: Yahoo*
Allow: /
3 M2 [9 _3 G  _7 x8 Y/ X+ XUser-agent: Bing*
Allow: /
User-agent: Google*
3 \' y$ J  `) k5 R$ D' @Allow: /
4 N) V+ m. y# [8 G7 m" I7 Q5 ]
User-agent: *
Disallow: /
Disallow: /*.php$
+ \' R4 e) n# f3 n. d8 rDisallow: /search/
; m3 j! X) e8 n" M" M( }Disallow: /tag/*?*
Disallow: /*.html?*
- G/ E7 e+ F$ w; uDisallow: /page/*?*
$ z! s( o" K3 L$ T, FDisallow: /wp-admin
Disallow: /wp-*.php
) i$ e0 V$ t" g, IDisallow: /comments/
! n: I! s! }9 D5 nDisallow: /trackback
! r1 x, f4 m3 G  M6 a" [Disallow: /*.html/*?*
Disallow: /*/trackback
Disallow: /comments/feed
Disallow: /*?replytocom=*
5 \% S3 t1 A  f) `Disallow: /*/comment-page-*
8 f! Y8 h5 x$ L. f$ c( j
6 m' r% U9 L/ ^Sitemap: http://www.luosuo.net/old/sitemap.xml
# J2 S5 U- i, R. n4 z很粗率，完了上robots驗證驗證一下或者麻煩些。。。
& A4 @; d" Q- X- B$ ~
4 Z- p" I0 b: D/ cUser-agent: Alexa*
Disallow: /*.php$
8 O2 ^6 j# d! O9 i$ f! j" wDisallow: /search/
Disallow: /tag/*?*
Disallow: /*.html?*
8 ?4 f2 r4 y1 [" v6 o( bDisallow: /page/*?*
: R7 M+ _! w; `( q" i, `Disallow: /wp-admin
Disallow: /wp-*.php
Disallow: /comments/
3 d) v  c- p3 H4 vDisallow: /trackback
) x- F! b( f* u- \1 {" _6 ^/ P2 \Disallow: /*.html/*?*
Disallow: /*/trackback
& d/ [; P+ V/ b" UDisallow: /comments/feed
Disallow: /*?replytocom=*
5 }. l) A  M7 U; VDisallow: /*/comment-page-*
) }5 N  I- T8 h, V3 W/ P# K
User-agent: Aol*
Disallow: /*.php$
Disallow: /search/
0 a+ x* v2 n9 HDisallow: /tag/*?*
Disallow: /*.html?*
Disallow: /page/*?*
Disallow: /wp-admin
& @: t# A1 }; E0 j: t& aDisallow: /wp-*.php
Disallow: /comments/
Disallow: /trackback
Disallow: /*.html/*?*
Disallow: /*/trackback
Disallow: /comments/feed
) j! m* j; s' u9 U/ u7 D1 s% ODisallow: /*?replytocom=*
Disallow: /*/comment-page-*

; Y& Y! Q7 n; k7 v# [! z, j7 fUser-agent: Ask*
Disallow: /*.php$
- q4 Q% t; V+ a0 U1 }9 \. fDisallow: /search/
Disallow: /tag/*?*
! ^0 T5 K8 g. VDisallow: /*.html?*
/ T+ D2 j) o6 kDisallow: /page/*?*
Disallow: /wp-admin
/ F3 q0 c7 w; H- [" ^Disallow: /wp-*.php
Disallow: /comments/
- J( v0 X( B' |& r! g# dDisallow: /trackback
7 g) w  M' U, }/ v* C9 F) yDisallow: /*.html/*?*
Disallow: /*/trackback
Disallow: /comments/feed
# E" m% r- ?) H' e8 d5 EDisallow: /*?replytocom=*
Disallow: /*/comment-page-*
  W0 W3 e, c8 m2 F4 i: e% I
User-agent: Yahoo*
Disallow: /*.php$
Disallow: /search/
$ Z7 r8 m, p9 w0 S( BDisallow: /tag/*?*
Disallow: /*.html?*
Disallow: /page/*?*
  }# J" A: |1 l! x) P- zDisallow: /wp-admin
# E3 Y4 J+ E& m0 I8 G5 k/ l% G6 f$ CDisallow: /wp-*.php
0 H! e: c6 I1 {! r; ^Disallow: /comments/
Disallow: /trackback
2 O* ?& a1 M5 E& q; D/ kDisallow: /*.html/*?*
' g, Y2 ?$ {1 c% S$ w; H8 ^Disallow: /*/trackback
Disallow: /comments/feed
: _# X3 n/ c* Y* O( ^0 i* m# _Disallow: /*?replytocom=*
5 K) L6 H; W: J4 c5 U2 TDisallow: /*/comment-page-*

User-agent: Bing*
Disallow: /*.php$
Disallow: /search/
Disallow: /tag/*?*
Disallow: /*.html?*
Disallow: /page/*?*
1 A8 l8 E/ [; Q+ d' `, ]Disallow: /wp-admin
+ d9 h3 u6 |3 p( m9 DDisallow: /wp-*.php
Disallow: /comments/
# a$ D) o  M; X6 p+ L$ BDisallow: /trackback
Disallow: /*.html/*?*
Disallow: /*/trackback
0 w, V% D) b; h$ z5 rDisallow: /comments/feed
Disallow: /*?replytocom=*
Disallow: /*/comment-page-*
2 x- Y) J5 _5 R9 D- r# o
3 H" \( I  b; e! kUser-agent: Google*
Disallow: /*.php$
Disallow: /search/
( P% I* }  m' L6 _4 P4 S" VDisallow: /tag/*?*
+ O7 @( ]- L0 k; t/ \3 dDisallow: /*.html?*
$ ~1 ?/ G+ s4 \- y% \$ \Disallow: /page/*?*
Disallow: /wp-admin
Disallow: /wp-*.php
Disallow: /comments/
Disallow: /trackback
Disallow: /*.html/*?*
$ x9 ?; g0 F- o. G0 gDisallow: /*/trackback
Disallow: /comments/feed
Disallow: /*?replytocom=*
6 l3 i- ~% S, H, s1 |% J9 v) EDisallow: /*/comment-page-*

7 T( R5 {% Z1 ^! R3 p5 ^( d+ r" a# ?User-agent: *
, j# w9 @% R% I) }( R! k1 j; UDisallow: /
Disallow: *
9 {9 d' ]2 v4 d9 r2 D7 T
Sitemap: http://www.luosuo.net/old/sitemap.xml
0 h! G9 j. T& G% T  _  v( Y


; g9 L9 M& r5 n
# w8 C! z+ w: h( x; I

& W) Z8 ?- ~2 ~: ~1 V

1.學習下robots.txt語法，其實很簡單
/ ^* \2 M! ^# O- }. J2.google下各個ua

% v+ t2 C  h8 T& E$ f+ i/ S2 t

只寫了google的寫法，你需要讓哪個爬蟲爬，照抄一份UA進去就行了。
User-agent: Googlebot
- C4 t1 Z+ n+ m" g2 ~Allow: /
/ H  n% f! C+ K* TDisallow: /mulu/
Disallow: /*.css$

. J$ J' D$ f$ G8 ~5 B& [
User-agent: Googlebot-Mobile
2 N9 f9 l- C" E/ K% ?Allow: /
2 h5 r' z. Q/ Q  ?5 NDisallow: /mulu/
Disallow: /*.css$
2 W  O2 E8 @0 W; ~8 k' }: ?
  M2 u2 Y: A9 ~8 B9 mUser-agent: *
- F4 z/ p4 H& v1 g. f% \+ kDisallow: /

Sitemap:http://www.xxx.com/sitemap.xml
其實百度百科裡面寫的很詳細啊。。。
http://baike.baidu.com/view/9274458.htm#3_1
, Y0 R$ q% ^' f! n1 ~

work4seo 發表於 2014-12-29 17:39
5 s9 f; z0 g( q8 H# D! a只寫了google的寫法，你需要讓哪個爬蟲爬，照抄一份UA進去就行了。
User-agent: Googlebot
2 O2 Y1 h5 r8 S% ^) V( MAllow: /

Alexa, aol, Ask, Yahoo、Bing  ??
3 I4 n  m3 ~0 v4 T8 u

要將
, b! u0 h/ L* G! L$ x; r$ }User-agent: *
Disallow: /

放在最前面吧？後面的設置會覆蓋前面的設置？

User-agent: *
3 ]4 E! A- K! \Disallow: /

User-agent: Google*,Yahoo*,msn*,.....
Allow: /
+ s8 x) f) o/ V) W  @- l$ b' S# O( ~Disallow: /mulu/
Disallow: /*.css$
" n& v( T' D) T: `/ }/ w6 j
Sitemap: http://www.luosuo.net/old/sitemap.xml

8 O/ m8 E2 _! VUser-agent: *
Disallow: /
! h7 {8 K0 J6 i# w6 S
, y; Q5 Z6 u' F( i/ d# fUser-agent: Google*,Yahoo*,msn*,.....
Allow: /
Disallow: /mulu/
& x/ l5 s3 ~( f' s# ]Disallow: /*.css$
& u  X9 R8 I9 }- K! C* I
7 E  V+ N3 B, o6 XSitemap: http://www.luosuo.net/old/sitemap.xml
# k. y, {- g$ @, K
% w  I3 h3 B$ }2 q. k

樓上幾個都不錯，炒一下吧
0 U: A6 B& a8 s  ]) b9 w

這麼多都要寫啊。。。