懸賞wordpress英文站robots.txt 規則的完整寫法

5 h& h# d" Y2 h9 e7 o
$ ~  s( ~6 h3 x8 J懸賞wordpress英文站robots.txt 規則的完整寫法
4 y" _+ ]/ s. ?, v* X# A6 d4 H/ h按以下要求寫
8 t. y3 o6 i5 Y1 I9 T# V1，只能Alexa, aol, Ask, Google、Yahoo、Bing的爬蟲訪問，其他的禁止
2，禁止訪問特定目錄，文件類型的通用寫法，適用於wordpress英文版3.6~4.1版本
3,  Sitemaps文件
# y" L' e* a4 Q
% |/ A* s& F, `( `. k2 y% v: K


! _% E! S- V0 I1 B

( [7 R* V' }% A& g9 J" u

; `5 p0 [. ?8 n/ T; GUser-agent: Alexa*
1 G/ |- N# i4 O- p9 @/ fAllow: /
, X! R/ |7 \5 {( JUser-agent: Aol*
Allow: /
' ]! H9 ~" l) j- \: }User-agent: Ask*
Allow: /
User-agent: Yahoo*
Allow: /
User-agent: Bing*
$ [/ h# R0 v' i) j" eAllow: /
! d% Z" E+ S" N. t0 s0 qUser-agent: Google*
Allow: /
- c' O) H' y: J/ S, k4 \/ A) |
5 J# W  U& j$ p; [9 oUser-agent: *
Disallow: /
9 l8 l2 e# {- l) _Disallow: /*.php$
Disallow: /search/
8 e0 m" k4 o& N% C/ oDisallow: /tag/*?*
' O+ V. l  p& ]( YDisallow: /*.html?*
4 C2 z4 [, K; q, F% H' U8 V; ?5 EDisallow: /page/*?*
Disallow: /wp-admin
Disallow: /wp-*.php
Disallow: /comments/
& \# q; a& R; ~2 w1 FDisallow: /trackback
Disallow: /*.html/*?*
' J8 J; j4 `) c& m5 h2 vDisallow: /*/trackback
Disallow: /comments/feed
Disallow: /*?replytocom=*
Disallow: /*/comment-page-*

Sitemap: http://www.luosuo.net/old/sitemap.xml
, Y% I7 Z" O* h8 i1 i很粗率，完了上robots驗證驗證一下或者麻煩些。。。

* [  z; A0 G! j. Q; Y0 X- UUser-agent: Alexa*
, \6 D6 `) G5 g; q* EDisallow: /*.php$
0 }' l* v( a6 h/ SDisallow: /search/
: \: Y3 Z% W6 r+ YDisallow: /tag/*?*
2 i6 s4 U" y8 v8 }/ jDisallow: /*.html?*
  e% C/ W+ G0 E, [" ODisallow: /page/*?*
$ n, c! i- X8 Z( U" kDisallow: /wp-admin
Disallow: /wp-*.php
Disallow: /comments/
Disallow: /trackback
" D6 V! [% B" U1 c; yDisallow: /*.html/*?*
0 i! T! C1 ?# M7 W' oDisallow: /*/trackback
Disallow: /comments/feed
/ t# I) d: d! a- P% N' A9 PDisallow: /*?replytocom=*
# T7 D  H0 {8 `8 Q  l3 LDisallow: /*/comment-page-*

User-agent: Aol*
& F$ R0 B& _7 ]Disallow: /*.php$
Disallow: /search/
Disallow: /tag/*?*
Disallow: /*.html?*
  y# U6 Q1 c" I; L8 |Disallow: /page/*?*
" c1 Q0 d9 a# ~# x) N4 g8 zDisallow: /wp-admin
9 Y1 b5 Y3 n$ p, fDisallow: /wp-*.php
4 R! S* X- d& z, p( T' r9 fDisallow: /comments/
$ L6 h3 L0 ~0 b4 K$ ]3 N0 A" ADisallow: /trackback
Disallow: /*.html/*?*
Disallow: /*/trackback
Disallow: /comments/feed
Disallow: /*?replytocom=*
Disallow: /*/comment-page-*
. D4 [4 c+ |+ T( e+ L# ~, k
2 _8 k9 |  n- Y# e$ yUser-agent: Ask*
# P9 [8 e. I. j3 x. ~Disallow: /*.php$
Disallow: /search/
Disallow: /tag/*?*
Disallow: /*.html?*
2 u: G0 o. X+ I& A( ]Disallow: /page/*?*
7 I; i5 J6 z  @9 [1 Q7 O6 @Disallow: /wp-admin
1 ]6 }( W4 G: Y# DDisallow: /wp-*.php
2 }$ L1 @( l% s' A* T8 ^% V; l# \& jDisallow: /comments/
Disallow: /trackback
Disallow: /*.html/*?*
Disallow: /*/trackback
Disallow: /comments/feed
0 a, L; q; F0 m; m: TDisallow: /*?replytocom=*
Disallow: /*/comment-page-*
! F6 y. I+ x0 u3 P# ~; }
User-agent: Yahoo*
+ Z4 j+ u" P! Z! y% c5 {Disallow: /*.php$
# y' t% ?/ _, W. S7 iDisallow: /search/
5 D; g! w7 [/ T! g4 Z! W3 HDisallow: /tag/*?*
: X. E6 Y8 D8 C: n% a) ADisallow: /*.html?*
7 {0 o3 Z- e# Y$ y7 sDisallow: /page/*?*
Disallow: /wp-admin
5 D% A7 a% ?0 z% \7 ^$ xDisallow: /wp-*.php
2 I9 w7 \$ U% C; D2 N9 Y6 VDisallow: /comments/
0 K  f3 ]9 x+ o' k$ I; I% W  |1 w. HDisallow: /trackback
Disallow: /*.html/*?*
Disallow: /*/trackback
. F. d5 [" V& z6 k7 }* ]Disallow: /comments/feed
Disallow: /*?replytocom=*
Disallow: /*/comment-page-*

User-agent: Bing*
Disallow: /*.php$
  ]$ _# L/ F- M- E% iDisallow: /search/
. b" N2 l: x! ?+ v9 s: v! yDisallow: /tag/*?*
& x* p& E  ^+ R% k- F/ r& M0 @Disallow: /*.html?*
Disallow: /page/*?*
2 X. F0 c4 R0 F5 RDisallow: /wp-admin
Disallow: /wp-*.php
1 B1 J6 L& C5 C! Q3 o5 e/ X8 ZDisallow: /comments/
- F' k: n; q" {( Z- ^/ NDisallow: /trackback
Disallow: /*.html/*?*
3 A( [+ o" ~, c2 M+ C4 L; L" HDisallow: /*/trackback
Disallow: /comments/feed
Disallow: /*?replytocom=*
8 |. M' |3 y! L* J, X! iDisallow: /*/comment-page-*
# L* H3 L% [% p6 I2 }8 A3 H, Q
3 L1 ^  D# m8 m/ V) @User-agent: Google*
% w: o2 Y5 k7 G+ F( r; oDisallow: /*.php$
Disallow: /search/
Disallow: /tag/*?*
Disallow: /*.html?*
- m+ Q& V! |: U$ H. WDisallow: /page/*?*
  d! Y2 ]: D! |8 m/ x; I6 I2 NDisallow: /wp-admin
+ R( l: z6 M" ~$ Z% LDisallow: /wp-*.php
Disallow: /comments/
1 N0 X, l. E& f, e& f9 z; Y+ p9 cDisallow: /trackback
Disallow: /*.html/*?*
9 g. b' n3 D- P, y/ r! f( w2 c1 H; rDisallow: /*/trackback
7 e% v. g  R. _% {Disallow: /comments/feed
% t. C( R$ g7 pDisallow: /*?replytocom=*
Disallow: /*/comment-page-*
0 l! o# X  E6 o+ v5 U  K
User-agent: *
Disallow: /
" j# F9 F/ M; tDisallow: *

Sitemap: http://www.luosuo.net/old/sitemap.xml
( R5 p& ~- S% U4 x4 M
! R) ]$ {8 z! U0 ]" B: `
# O, |& w7 B( T4 s' K
5 K# X' b7 m7 h9 M2 N

, y4 Q! g7 A) O9 r6 n' L
7 }& `" L  V( z8 `9 g4 L

1.學習下robots.txt語法，其實很簡單
2.google下各個ua
$ h; B% @' ~' l! O5 i+ U
% ?& l4 ^/ L3 X  F$ A

! l; h: o1 Q" R8 L6 H4 U( W
只寫了google的寫法，你需要讓哪個爬蟲爬，照抄一份UA進去就行了。
User-agent: Googlebot
( f3 F7 c) H$ R7 z' @Allow: /
Disallow: /mulu/
* s% v5 g0 \. w; J' ?2 {0 A' e# ~! LDisallow: /*.css$

  z4 T3 y; z1 u. B
User-agent: Googlebot-Mobile
% n6 T7 c" q* r6 L$ N  @Allow: /
Disallow: /mulu/
Disallow: /*.css$

# _2 l# A6 F. Q# J8 z( H) u: Z  rUser-agent: *
4 ^* H- ^# C0 U( [Disallow: /
1 D4 ^! v  [6 `' n
4 E! @) Q0 p1 b; |. K) E" KSitemap:http://www.xxx.com/sitemap.xml
( n; V( I" e9 C! N" d" }! R& `其實百度百科裡面寫的很詳細啊。。。
6 B( a/ ]# C4 y) u2 Z% ?http://baike.baidu.com/view/9274458.htm#3_1
8 V4 n# V# w# |4 w

work4seo 發表於 2014-12-29 17:39
只寫了google的寫法，你需要讓哪個爬蟲爬，照抄一份UA進去就行了。
, O8 g3 G8 ?9 WUser-agent: Googlebot
Allow: /

Alexa, aol, Ask, Yahoo、Bing  ??

要將
User-agent: *
Disallow: /
! J2 C! E# T- y) p: B3 F
放在最前面吧？後面的設置會覆蓋前面的設置？

User-agent: *
Disallow: /
( r% Z2 y3 H% I
1 i8 e% B' a* Z+ R5 aUser-agent: Google*,Yahoo*,msn*,.....
Allow: /
Disallow: /mulu/
Disallow: /*.css$
( N7 x0 K3 R3 C0 \6 x- y& |6 V9 O
, P/ D9 t% C1 N/ h6 K4 _; ]# n% oSitemap: http://www.luosuo.net/old/sitemap.xml

* h( p* @, f2 [: t5 h

: c/ ?1 s* h- {. R4 o& oUser-agent: *
Disallow: /

User-agent: Google*,Yahoo*,msn*,.....
Allow: /
Disallow: /mulu/
Disallow: /*.css$
. Q: T  N) @" K# S6 R
Sitemap: http://www.luosuo.net/old/sitemap.xml

樓上幾個都不錯，炒一下吧

這麼多都要寫啊。。。
/ a4 R) }9 g" N% p