{"id":556,"date":"2007-07-27T08:36:58","date_gmt":"2007-07-27T15:36:58","guid":{"rendered":"http:\/\/www.gubatron.com\/blog\/2007\/07\/27\/python-how-to-debug-http-while-using-urllib2\/"},"modified":"2007-07-27T08:36:58","modified_gmt":"2007-07-27T15:36:58","slug":"python-how-to-debug-http-while-using-urllib2","status":"publish","type":"post","link":"https:\/\/www.gubatron.com\/blog\/python-how-to-debug-http-while-using-urllib2\/","title":{"rendered":"Python: How to debug HTTP while using urllib2"},"content":{"rendered":"<pre>\n...\nimport urllib\nimport urllib2\n\n\n#this is just to prepare a dynamic uri (this is actual code from a system I'm building, sorry)\nfileDownloadServiceURL = '%s:\/\/%s:%s\/%s' % (transport,server,port,pathToController)\npostData = {'URI':fileUri} #add more post stuff here\npostData = urllib.urlencode(postData) #make sure you encode your post data\n\n#add some custom headers if you need them\nheaders={\"Host\": server+':'+port,\"Cookie\":\"JSESSIONID=\"+sessionId,\"User-Agent\":\"Name of your User-Agent Here\"}\n\n#prepare your request, with headers and post data\nreq = urllib2.Request(fileDownloadServiceURL,postData,headers)\n\n\n#and this is the magic. Create a HTTPHandler object and put its debug level to 1\nhttpHandler = urllib2.HTTPHandler()\nhttpHandler.set_http_debuglevel(1)\n        \n#Instead of using urllib2.urlopen, create an opener, and pass the HTTPHandler\n#and any other handlers... to it.\nopener = urllib2.build_opener(httpHandler)\n\n#User your opener to open the Request.\nurlHandle = opener.open(req)\n \n#you'll end up with a file-like object... which I called urlHandle\n...\n<\/pre>\n<p>The ouput will have useful debugging info about the HTTP connection.<\/p>\n<pre>\nconnect: (localhost, 8080)\nsend: u'POST \/arcturus-web\/fileVariableDownload.service HTTP\/1.1rnAccept-Encoding: identityrnContent-Length: 57rnConnection: closernUser-Agent: Temboo Twyla\/Arcturus HTTP DownloaderrnHost: localhost:8080rnCookie: JSESSIONID=1b8xl8nozb2irnContent-Type: application\/x-www-form-urlencodedrnrn'\nsend: 'URI=temboo%3A%2F%2Fwww.one.com%2Ffiles.f%2Fmapping1.m.var'\nreply: 'HTTP\/1.1 200 OKrn'\nheader: Content-Language: en-US\nheader: Content-Type: text\/html; charset=ISO-8859-1\nheader: Connection: close\nheader: Server: Jetty(6.0.2)\n<\/pre>\n","protected":false},"excerpt":{"rendered":"<p>&#8230; import urllib import urllib2 #this is just to prepare a dynamic uri (this is actual code from a system I&#8217;m building, sorry) fileDownloadServiceURL = &#8216;%s:\/\/%s:%s\/%s&#8217; % (transport,server,port,pathToController) postData = {&#8216;URI&#8217;:fileUri} #add more post stuff here postData = urllib.urlencode(postData) #make sure you encode your post data #add some custom headers if you need them headers={&#8220;Host&#8221;: [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[15,30,43,65],"tags":[],"class_list":["post-556","post","type-post","status-publish","format-standard","hentry","category-code","category-geeklife","category-linux","category-python"],"jetpack_publicize_connections":[],"jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p5Unzf-8Y","jetpack-related-posts":[{"id":254,"url":"https:\/\/www.gubatron.com\/blog\/create-base32-sha1-hashes-out-of-files-in-php-5\/","url_meta":{"origin":556,"position":0},"title":"Create Base32 SHA1 hashes out of files in PHP 5","author":"gubatron","date":"February 9, 2006","format":false,"excerpt":"LimeWire and other Gnutella clients uniquely identify files by creating a SHA1 hash of the files. I thought that it might be useful to the gnutella network to have regular webservers sharing legal files. Usually regular Joe webmasters don't have permissions to run java programs or have tcp ports open\u2026","rel":"","context":"In &quot;Code&quot;","block_context":{"text":"Code","link":"https:\/\/www.gubatron.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2593,"url":"https:\/\/www.gubatron.com\/blog\/lighttpd-allow-access-control-allow-origin-headers-on-the-server-status-page\/","url_meta":{"origin":556,"position":1},"title":"lighttpd, allow &#8220;Access-Control-Allow-Origin:*&#8221; headers on the server status page","author":"gubatron","date":"November 29, 2011","format":false,"excerpt":"Maybe there's someone out there who needs to read the output of lighttpd's status for monitoring purpose like me tonight, and also, like me, you want to do this using JavaScript, but your browser gives you this nasty error: XMLHttpRequest cannot load http:\/\/otherSubdomain.server.com\/lighttpd-status-url-you-have-configured. Origin http:\/\/requestingSubdomain.server.com is not allowed by Access-Control-Allow-Origin.\u2026","rel":"","context":"In &quot;AJAX&quot;","block_context":{"text":"AJAX","link":"https:\/\/www.gubatron.com\/blog\/category\/ajax\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2767,"url":"https:\/\/www.gubatron.com\/blog\/ubuntu-packages-for-a-kick-ass-web-server\/","url_meta":{"origin":556,"position":2},"title":"ubuntu packages for a kick ass web server","author":"gubatron","date":"September 7, 2012","format":false,"excerpt":"Copy and paste the following list on a file, say \"packages.txt\". To install all just do: sudo apt-get install $(cat packages.txt) accountsservice acpid adduser ant ant-optional apache2-utils apparmor apport apport-symptoms apt apt-transport-https apt-utils apt-xapian-index aptitude at base-files base-passwd bash bash-completion bc bind9-host bsdmainutils bsdutils busybox-initramfs busybox-static byobu bzip2 ca-certificates ca-certificates-java\u2026","rel":"","context":"In &quot;Code&quot;","block_context":{"text":"Code","link":"https:\/\/www.gubatron.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":156,"url":"https:\/\/www.gubatron.com\/blog\/portscanner-only-60-lines-of-code\/","url_meta":{"origin":556,"position":3},"title":"PortScanner &#8211; Only 60 lines of code","author":"gubatron","date":"May 19, 2005","format":false,"excerpt":"Hi, here's a simple port scanner I wrote today in Java. I needed to see something on my server, perhaps you can find it useful. I would say its pretty fast, you can tweak the number of threads and timeouts to adjust to your server\/connection. I did't do much testing\u2026","rel":"","context":"In &quot;Gubatron&quot;","block_context":{"text":"Gubatron","link":"https:\/\/www.gubatron.com\/blog\/category\/gubatron\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":516,"url":"https:\/\/www.gubatron.com\/blog\/ejemplo-de-automatizacion-entre-2-maquinas-remotas-con-bash-scripting-y-python\/","url_meta":{"origin":556,"position":4},"title":"Ejemplo de automatizacion entre 2 maquinas remotas con bash scripting y Python","author":"gubatron","date":"May 10, 2007","format":false,"excerpt":"Para los amigos que se inician en el mundo *nix, ya sea con su nueva Mac, o con una PC corriendo Linux, les recomiendo que aprendan a manejar bien los siguientes lenguajes, y el mundo sera suyo: - bash scripting (aliases, variables, exports, iteraciones, condicionales) - python (para programar logica\u2026","rel":"","context":"In &quot;Code&quot;","block_context":{"text":"Code","link":"https:\/\/www.gubatron.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":1457,"url":"https:\/\/www.gubatron.com\/blog\/map-commands-to-servers-via-ssh\/","url_meta":{"origin":556,"position":5},"title":"Quick N Dirty way to Map Commands to remote servers via ssh","author":"gubatron","date":"October 10, 2009","format":false,"excerpt":"You may be running several independent but similar servers at the same time and wasting time by executing commands in all of them one by one. Wouldn't it be nice to send a command to all of them at once? or to monitor all of them at once. The following\u2026","rel":"","context":"In &quot;Code&quot;","block_context":{"text":"Code","link":"https:\/\/www.gubatron.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/posts\/556","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/comments?post=556"}],"version-history":[{"count":0,"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/posts\/556\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/media?parent=556"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/categories?post=556"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/tags?post=556"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}