|
Server : Apache/2.2.17 (Unix) mod_ssl/2.2.17 OpenSSL/0.9.8e-fips-rhel5 DAV/2 PHP/5.2.17 System : Linux localhost 2.6.18-419.el5 #1 SMP Fri Feb 24 22:47:42 UTC 2017 x86_64 User : nobody ( 99) PHP Version : 5.2.17 Disable Function : NONE Directory : /proc/21573/task/21573/root/usr/lib/python2.4/site-packages/urlgrabber/ |
Upload File : |
mò
5Ec @ s¹ d Z d k Z d k Z d k Z d k Z d a d k Z d k Z e i d j o
d a
n d a
d f d „ ƒ YZ d f d „ ƒ YZ d
e e i
f d „ ƒ YZ
d e e i f d
„ ƒ YZ d e i f d „ ƒ YZ d e i f d „ ƒ YZ d e i f d „ ƒ YZ d „ Z d „ Z d „ Z d d „ Z d „ Z d d „ Z e d j oa d k Z d k Z y$ e e i d ƒ Z e i d Z Wn d e i d GHqµXe e e ƒ n d S( s
An HTTP handler for urllib2 that supports HTTP 1.1 and keepalive.
>>> import urllib2
>>> from keepalive import HTTPHandler
>>> keepalive_handler = HTTPHandler()
>>> opener = urllib2.build_opener(keepalive_handler)
>>> urllib2.install_opener(opener)
>>>
>>> fo = urllib2.urlopen('http://www.python.org')
If a connection to a given host is requested, and all of the existing
connections are still in use, another connection will be opened. If
the handler tries to use an existing connection but it fails in some
way, it will be closed and removed from the pool.
To remove the handler, simply re-run build_opener with no arguments, and
install that opener.
You can explicitly close connections by using the close_connection()
method of the returned file-like object (described below) or you can
use the handler methods:
close_connection(host)
close_all()
open_connections()
NOTE: using the close_connection and close_all methods of the handler
should be done with care when using multiple threads.
* there is nothing that prevents another thread from creating new
connections immediately after connections are closed
* no checks are done to prevent in-use connections from being closed
>>> keepalive_handler.close_all()
EXTRA ATTRIBUTES AND METHODS
Upon a status of 200, the object returned has a few additional
attributes and methods, which should not be used if you want to
remain consistent with the normal urllib2-returned objects:
close_connection() - close the connection to the host
readlines() - you know, readlines()
status - the return status (ie 404)
reason - english translation of status (ie 'File not found')
If you want the best of both worlds, use this inside an
AttributeError-catching try:
>>> try: status = fo.status
>>> except AttributeError: status = None
Unfortunately, these are ONLY there if status == 200, so it's not
easy to distinguish between non-200 responses. The reason is that
urllib2 tries to do clever things with error codes 301, 302, 401,
and 407, and it wraps the object upon return.
For python versions earlier than 2.4, you can avoid this fancy error
handling by setting the module-level global HANDLE_ERRORS to zero.
You see, prior to 2.4, it's the HTTP Handler's job to determine what
to handle specially, and what to just pass up. HANDLE_ERRORS == 0
means "pass everything up". In python 2.4, however, this job no
longer belongs to the HTTP Handler and is now done by a NEW handler,
HTTPErrorProcessor. Here's the bottom line:
python version < 2.4
HANDLE_ERRORS == 1 (default) pass up 200, treat the rest as
errors
HANDLE_ERRORS == 0 pass everything up, error processing is
left to the calling code
python version >= 2.4
HANDLE_ERRORS == 1 pass up 200, treat the rest as errors
HANDLE_ERRORS == 0 (default) pass everything up, let the
other handlers (specifically,
HTTPErrorProcessor) decide what to do
In practice, setting the variable either way makes little difference
in python 2.4, so for the most consistent behavior across versions,
you probably just want to use the defaults, which will give you
exceptions on errors.
Ni i i i t ConnectionManagerc B sG t Z d Z d „ Z d „ Z d „ Z d „ Z d „ Z e d „ Z RS( sV
The connection manager must be able to:
* keep track of all existing
c C s. t i ƒ | _ h | _ h | _ h | _ d S( N( t threadt
allocate_lockt selft _lockt _hostmapt _connmapt _readymap( R ( ( t8 /usr/lib/python2.4/site-packages/urlgrabber/keepalive.pyt __init__z s c C sx | i i ƒ zV | i i | ƒ p g | i | <n | i | i | ƒ | | i | <| | i
| <Wd | i i ƒ Xd S( N( R R t acquireR t has_keyt hostt appendt
connectionR t readyR t release( R R R R ( ( R t add€ s
c C s‘ | i i ƒ zo y | i | } Wn t j
o nF X| i | =| i | =| i | i | ƒ | i | p | i | =n Wd | i i
ƒ Xd S( N( R R R
R R R t KeyErrorR R t removeR ( R R R ( ( R R Š s
c C s+ y | | i | <Wn t j
o n Xd S( N( R R R R R ( R R R ( ( R t set_ready™ s c C s d } | i i ƒ zY | i i | ƒ oB x? | i | D], } | i | o d | i | <| } Pq7 q7 Wn Wd | i i
ƒ X| S( Ni ( t Nonet connR R R
R R R t cR R ( R R R R ( ( R t get_ready_conn s
c C s5 | o t | i i | g ƒ ƒ Sn t | i ƒ Sd S( N( R t listR R t gett dict( R R ( ( R t get_all« s (
t __name__t
__module__t __doc__R R R R R R R ( ( ( R R u s
t KeepAliveHandlerc B se t Z d „ Z d „ Z d „ Z d „ Z d „ Z d d „ Z d „ Z d „ Z d
„ Z
d „ Z RS( Nc C s t ƒ | _ d S( N( R R t _cm( R ( ( R R ² s c C s@ g } | i i ƒ i ƒ D] \ } } | | t | ƒ f q ~ S( st return a list of connected hosts and the number of connections
to each. [('foo.com:80', 2), ('bar.org', 1)]N( t _[1]R R! R t itemsR t lit len( R R" R R$ ( ( R t open_connections¶ s c C s; x4 | i i | ƒ D] } | i i | ƒ | i ƒ q Wd S( s¨ close connection(s) to <host>
host is the host:port spec, as in 'www.cnn.com:8080' as passed in.
no error occurs if there is no connection to that host.N( R R! R R t hR t close( R R R' ( ( R t close_connection» s
c C sU xN | i i ƒ i ƒ D]7 \ } } x( | D] } | i i | ƒ | i ƒ q) Wq Wd S( s close all open connectionsN( R R! R R# R t connsR' R R( ( R R* R' R ( ( R t close_allà s c C s | i i | d ƒ d S( sd tells us that this request is now closed and the the
connection is ready for another requesti N( R R! R R ( R t requestR R ( ( R t _request_closedÊ s i c C s) | o | i ƒ n | i i | ƒ d S( N( R( R R R! R ( R R R R( ( ( R t _remove_connectionÏ s c C s | i ƒ } | p t i d ƒ ‚ n y× | i i | ƒ } x¾ | oQ | i | | | ƒ } | o Pn | i ƒ | i i | ƒ | i i | ƒ } q> W| i
| ƒ } t o t i d | t | ƒ ƒ n | i i | | d ƒ | i | | ƒ | i ƒ } Wn0 t i t i f j
o } t i | ƒ ‚ n X| i o | i i | ƒ n t o t i d | i | i ƒ n | | _ | | _ | i ƒ | _ | | _ | i | _! | i" | _# | i | _" | i d j p t$ o | Sn) | i% i d | | | i | i" | i# ƒ Sd S( Ns
no host givens" creating new connection to %s (%d)i s STATUS: %s, %siÈ t http(& t reqt get_hostR t urllib2t URLErrorR R! R R' t _reuse_connectiont rR( R t _get_connectiont DEBUGt infot idR t _start_transactiont getresponset sockett errort httplibt
HTTPExceptiont errt
will_closet statust reasont _handlert _hostt get_full_urlt _urlt _connectiont codet msgt headerst
HANDLE_ERRORSt parent( R R0 R@ R' R R5 ( ( R t do_openÔ sJ
c C s y | i | | ƒ | i ƒ } Wnp t i t i f j
o
d } nM t o! t i d d | t
| ƒ ƒ n | i i | ƒ | i ƒ ‚ n X| d j p | i d j o. t o t i d | t
| ƒ ƒ n d } n% t o t i d | t
| ƒ ƒ n | S( sG start the transaction with a re-used connection
return a response object (r) upon success or None on failure.
This DOES not close or remove bad connections in cases where
it returns. However, if an unexpected exception occurs, it
will close and remove the connection before re-raising.
s unexpected exception - closing s connection to %s (%d)i s& failed to re-use connection to %s (%d)s re-using connection to %s (%d)N( R R: R' R0 R; R5 R< R= R> R? R R7 R R9 R! R R( t versionR8 ( R R' R0 R R5 ( ( R R4 s*
c C sX y¥ | i ƒ o~ | i ƒ } | i d | i ƒ ƒ | i i d ƒ p | i d d ƒ n | i i d ƒ p | i d d t
| ƒ ƒ q¤ n | i d | i ƒ ƒ Wn0 t i t
i f j
o } t i | ƒ ‚ n Xx! | i i D] } | i | Œ qå Wx- | i i ƒ D] \ } } | i | | ƒ q W| i ƒ | i ƒ o | i | ƒ n d S( Nt POSTs Content-types! application/x-www-form-urlencodeds Content-lengths %dt GET( R0 t has_datat get_datat dataR' t
putrequestt get_selectorRK R t putheaderR% R< R= R> R? R@ R2 R3 R RM t
addheaderst argsR# t kt vt
endheaderst send( R R' R0 RY R@ R[ RT RZ ( ( R R: / s*
"
c C s t S( N( t NotImplementedError( R R ( ( R R6 F s ( R R R R& R) R+ R- R. RN R4 R: R6 ( ( ( R R ± s 0 + t HTTPHandlerc B s# t Z d „ Z d „ Z d „ Z RS( Nc C s t i | ƒ d S( N( R R R ( R ( ( R R J s c C s
| i | ƒ S( N( R RN R0 ( R R0 ( ( R t http_openM s c C s
t | ƒ S( N( t HTTPConnectionR ( R R ( ( R R6 P s ( R R R R` R6 ( ( ( R R_ I s t HTTPSHandlerc B s& t Z d d „ Z d „ Z d „ Z RS( Nc C s1 t i | ƒ | p t i ƒ } n | | _ d S( N( R R R t ssl_factoryt
sslfactoryt get_factoryt _ssl_factory( R Rc ( ( R R T s
c C s
| i | ƒ S( N( R RN R0 ( R R0 ( ( R t
https_openZ s c C s | i i | ƒ S( N( R Rf t get_https_connectionR ( R R ( ( R R6 ] s ( R R R R Rg R6 ( ( ( R Rb S s t HTTPResponsec B sn t Z d d d d „ Z e i i Z d „ Z d „ Z d „ Z
d „ Z d d „ Z d d „ Z d d
„ Z
RS( Ni c C s‰ | o t i i | | | | ƒ n t i i | | | ƒ | i | _ d | _ d | _
d | _ d | _ d | _
d | _ d | _ d S( Nt i ( t methodR> Ri R R t sockt
debuglevelt filenoR RI t _rbuft _rbufsizeRD RE RG RH ( R Rl Rm t strictRk ( ( R R t s c C sR | i oD | i i ƒ d | _ | i o | i i | | i | i ƒ qN n d S( N( R t fpR( R RD R- RE RH ( R ( ( R R( „ s
c C s- | i i | i | i d d ƒ| i ƒ d S( NR( i ( R RD R. RE RH R( ( R ( ( R R) Œ s c C s | i S( N( R RK ( R ( ( R R8 s c C s | i S( N( R RG ( R ( ( R t geturl“ s c C s‰ | i o\ | d j oO t | i ƒ } | | j o | | 8} qf | i | } | i | | _ | Sn | i | i | ƒ } d | _ | S( NRj ( R Ro t amtR R% t Lt st _raw_read( R Rt Ru Rv ( ( R t read– s
iÿÿÿÿc C s3 d } | i i d ƒ } x› | d j o d | j o t | i ƒ j n oe | i | i ƒ } | p Pn | i d ƒ } | d j o | t | i ƒ } n | i | | _ q W| d j o t | i ƒ } n | d } d | j o t | i ƒ j n o
| } n | i | | i | } | _ | S( NRj s
i i (
RT R Ro t findt it limitR% Rw Rp t new( R R{ Rz R| RT ( ( R t readline¦ s$ 5
'
c C sf d } g } xS | i ƒ } | p Pn | i | ƒ | t | ƒ 7} | o | | j o Pq q W| S( Ni i ( t totalR R R} t lineR
R% t sizehint( R R€ R R R~ ( ( R t readlinesµ s
( R R R R R> Ri Rx Rw R( R) R8 Rs R} R ( ( ( R Ri ` s Ra c B s t Z e Z RS( N( R R Ri t response_class( ( ( R Ra  s t HTTPSConnectionc B s t Z e Z RS( N( R R Ri R‚ ( ( ( R Rƒ Æ s c C s+ t } t ƒ }
t i |
ƒ } t i | ƒ h d d <d d <} x¼ d D]´ } d | | | f GH| a yb t i
| ƒ } | i
ƒ } | i ƒ y | i | i } } Wn t j
o d
\ } } n XWn! t j
o } d | GH‚ qJ Xd | | f GHqJ W| a |
i ƒ } d G| GH|
i ƒ d S( Ni t offi t ons. fancy error handling %s (HANDLE_ERRORS = %i)s EXCEPTION: %ss status = %s, reason = %ss open connections:( i i ( NN( RL t origR_ t keepalive_handlerR2 t build_openert openert install_openert posRz t urlopent urlt foRx t fooR( RB RC t AttributeErrorR t IOErrort eR& t hostsR+ ( R RB R R’ Rz RC R‰ R‹ R“ RŽ R‡ R† ( ( R t
error_handlerÍ s4
c C sJ d k } d } t i ƒ } t i | ƒ t i | ƒ } | i ƒ } | i ƒ | i | ƒ } | d | i ƒ f GHt i t ƒ ƒ } t i | ƒ t i | ƒ } | i ƒ } | i ƒ | i | ƒ } | d | i ƒ f GHt i | ƒ } d } x' | i ƒ } | o | | } qñ Pqñ W| i ƒ | i | ƒ } | d | i ƒ f GHd S( Ns %25s: %ss
normal urllibs keepalive readRj i s keepalive readline( t md5t formatR2 Rˆ R‰ RŠ RŒ R RŽ Rx R R( R| t mt hexdigestR_ R} t f( R R R– R™ R— R‰ R• RŽ ( ( R t
continuityç s6
c C s« d | | f GHt i i d ƒ t i ƒ } t i | ƒ t | | ƒ } d | GHt i i d ƒ t i t ƒ ƒ } t i | ƒ t | | ƒ } d | GHd | | f GHd S( Ns making %i connections to:
%ss( first using the normal urllib handlerss TIME: %.3f ss( now using the keepalive handler s improvement factor: %.2f(
t NR t syst stdoutt writeR2 Rˆ R‰ RŠ t fetcht t1R_ t t2( R› R R¡ R R‰ ( ( R t comp s
c C sì d k } g } | i ƒ } xq t | ƒ D]c } | o | d j o | i | ƒ n t i | ƒ }
|
i ƒ } |
i ƒ | i t | ƒ ƒ q( W| i ƒ | } d } x@ | d D]4 } | d } | | d j p d | | f GHq° q° W| S( Ni i s+ WARNING: inconsistent length on read %i: %i( t timet lenst starttimet rangeR› Rz t delayt sleepR2 RŒ R RŽ Rx R R( R
R% t difft j( R› R R§ R¤ Rª Rz R¥ R£ R© R RŽ ( ( R RŸ s&
c C s t } d f d „ ƒ Y} | ƒ a d GHt i | ƒ } | i ƒ } | i ƒ d } d | GHxJ | d j o< t i i
d | ƒ t i i ƒ t i d ƒ | d 8} q^ Wt i i
d ƒ d
GHt i | ƒ } | i ƒ } | i ƒ | | j o d GHn d GH| a d S(
Nt
FakeLoggerc B s t Z d „ Z e Z Z Z RS( Nc G s
| | GHd S( N( RJ RY ( R RJ RY ( ( R t debug2 s ( R R R¬ R8 t warningR= ( ( ( R R« 1 s s- fetching the file to establish a connectioni s; waiting %i seconds for the server to close the connectioni s
%2ii s
s! fetching the file a second times data are identicals ERROR: DATA DIFFER( R7 t dbbackupR« R2 RŒ R RŽ Rx t data1R( Rz Rœ R Rž t flushR£ R¨ t stderrt data2( R Rz R¯ R« R² R® RŽ ( ( R t test_timeout. s2
i
c C sq d GHy t | ƒ Wn$ t j
o } d GHt i ƒ n XHd GHt | ƒ Hd GHt | | ƒ Hd GHt | ƒ d S( Ns, checking error hander (do this on a non-200)s. exiting - exception will prevent further testss> performing continuity test (making sure stuff isn't corrupted)s performing speed comparisons# performing dropped-connection check(
R” R R‘ R’ Rœ t exitRš R¢ R› R³ ( R R› R’ ( ( R t testP s
t __main__s %s <integer> <url>( i i ( R R2 R> R< R R R7 Rd Rœ t version_infoRL R R R_ Rb Ri Ra Rƒ R” Rš R¢ RŸ R³ Rµ R R£ t intt argvR› R ( R³ R R” R_ R¢ Rµ Ri Rš Rb R2 Rœ Ra R› R R> R< R R Rd R£ Rƒ RŸ ( ( R t ?d s@
<˜
b ! "