{"id":556,"date":"2007-07-27T08:36:58","date_gmt":"2007-07-27T15:36:58","guid":{"rendered":"http:\/\/www.gubatron.com\/blog\/2007\/07\/27\/python-how-to-debug-http-while-using-urllib2\/"},"modified":"2007-07-27T08:36:58","modified_gmt":"2007-07-27T15:36:58","slug":"python-how-to-debug-http-while-using-urllib2","status":"publish","type":"post","link":"https:\/\/www.gubatron.com\/blog\/python-how-to-debug-http-while-using-urllib2\/","title":{"rendered":"Python: How to debug HTTP while using urllib2"},"content":{"rendered":"<pre>\n...\nimport urllib\nimport urllib2\n\n\n#this is just to prepare a dynamic uri (this is actual code from a system I'm building, sorry)\nfileDownloadServiceURL = '%s:\/\/%s:%s\/%s' % (transport,server,port,pathToController)\npostData = {'URI':fileUri} #add more post stuff here\npostData = urllib.urlencode(postData) #make sure you encode your post data\n\n#add some custom headers if you need them\nheaders={\"Host\": server+':'+port,\"Cookie\":\"JSESSIONID=\"+sessionId,\"User-Agent\":\"Name of your User-Agent Here\"}\n\n#prepare your request, with headers and post data\nreq = urllib2.Request(fileDownloadServiceURL,postData,headers)\n\n\n#and this is the magic. Create a HTTPHandler object and put its debug level to 1\nhttpHandler = urllib2.HTTPHandler()\nhttpHandler.set_http_debuglevel(1)\n        \n#Instead of using urllib2.urlopen, create an opener, and pass the HTTPHandler\n#and any other handlers... to it.\nopener = urllib2.build_opener(httpHandler)\n\n#User your opener to open the Request.\nurlHandle = opener.open(req)\n \n#you'll end up with a file-like object... which I called urlHandle\n...\n<\/pre>\n<p>The ouput will have useful debugging info about the HTTP connection.<\/p>\n<pre>\nconnect: (localhost, 8080)\nsend: u'POST \/arcturus-web\/fileVariableDownload.service HTTP\/1.1rnAccept-Encoding: identityrnContent-Length: 57rnConnection: closernUser-Agent: Temboo Twyla\/Arcturus HTTP DownloaderrnHost: localhost:8080rnCookie: JSESSIONID=1b8xl8nozb2irnContent-Type: application\/x-www-form-urlencodedrnrn'\nsend: 'URI=temboo%3A%2F%2Fwww.one.com%2Ffiles.f%2Fmapping1.m.var'\nreply: 'HTTP\/1.1 200 OKrn'\nheader: Content-Language: en-US\nheader: Content-Type: text\/html; charset=ISO-8859-1\nheader: Connection: close\nheader: Server: Jetty(6.0.2)\n<\/pre>\n","protected":false},"excerpt":{"rendered":"<p>&#8230; import urllib import urllib2 #this is just to prepare a dynamic uri (this is actual code from a system I&#8217;m building, sorry) fileDownloadServiceURL = &#8216;%s:\/\/%s:%s\/%s&#8217; % (transport,server,port,pathToController) postData = {&#8216;URI&#8217;:fileUri} #add more post stuff here postData = urllib.urlencode(postData) #make sure you encode your post data #add some custom headers if you need them headers={&#8220;Host&#8221;: [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":false,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[15,30,43,65],"tags":[],"class_list":["post-556","post","type-post","status-publish","format-standard","hentry","category-code","category-geeklife","category-linux","category-python"],"jetpack_publicize_connections":[],"jetpack_featured_media_url":"","jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p5Unzf-8Y","jetpack-related-posts":[{"id":254,"url":"https:\/\/www.gubatron.com\/blog\/create-base32-sha1-hashes-out-of-files-in-php-5\/","url_meta":{"origin":556,"position":0},"title":"Create Base32 SHA1 hashes out of files in PHP 5","author":"gubatron","date":"February 9, 2006","format":false,"excerpt":"LimeWire and other Gnutella clients uniquely identify files by creating a SHA1 hash of the files. I thought that it might be useful to the gnutella network to have regular webservers sharing legal files. Usually regular Joe webmasters don't have permissions to run java programs or have tcp ports open\u2026","rel":"","context":"In &quot;Code&quot;","block_context":{"text":"Code","link":"https:\/\/www.gubatron.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2593,"url":"https:\/\/www.gubatron.com\/blog\/lighttpd-allow-access-control-allow-origin-headers-on-the-server-status-page\/","url_meta":{"origin":556,"position":1},"title":"lighttpd, allow &#8220;Access-Control-Allow-Origin:*&#8221; headers on the server status page","author":"gubatron","date":"November 29, 2011","format":false,"excerpt":"Maybe there's someone out there who needs to read the output of lighttpd's status for monitoring purpose like me tonight, and also, like me, you want to do this using JavaScript, but your browser gives you this nasty error: XMLHttpRequest cannot load http:\/\/otherSubdomain.server.com\/lighttpd-status-url-you-have-configured. Origin http:\/\/requestingSubdomain.server.com is not allowed by Access-Control-Allow-Origin.\u2026","rel":"","context":"In &quot;AJAX&quot;","block_context":{"text":"AJAX","link":"https:\/\/www.gubatron.com\/blog\/category\/ajax\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":2767,"url":"https:\/\/www.gubatron.com\/blog\/ubuntu-packages-for-a-kick-ass-web-server\/","url_meta":{"origin":556,"position":2},"title":"ubuntu packages for a kick ass web server","author":"gubatron","date":"September 7, 2012","format":false,"excerpt":"Copy and paste the following list on a file, say \"packages.txt\". To install all just do: sudo apt-get install $(cat packages.txt) accountsservice acpid adduser ant ant-optional apache2-utils apparmor apport apport-symptoms apt apt-transport-https apt-utils apt-xapian-index aptitude at base-files base-passwd bash bash-completion bc bind9-host bsdmainutils bsdutils busybox-initramfs busybox-static byobu bzip2 ca-certificates ca-certificates-java\u2026","rel":"","context":"In &quot;Code&quot;","block_context":{"text":"Code","link":"https:\/\/www.gubatron.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":3679,"url":"https:\/\/www.gubatron.com\/blog\/monero-development-installing-dependencies-on-mac\/","url_meta":{"origin":556,"position":3},"title":"[Monero-Development] Installing dependencies on Mac","author":"gubatron","date":"March 12, 2017","format":false,"excerpt":"March 12th 2017. In order to build Monero on MacOSX with cmake, I had to install the following dependencies via homebrew brew install libunwind-headers --force brew link libunwind-headers --force brew install miniupnpc brew install ldns brew link ldns --force brew install expat brew link expat --force brew install doxygen","rel":"","context":"In &quot;Code&quot;","block_context":{"text":"Code","link":"https:\/\/www.gubatron.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":156,"url":"https:\/\/www.gubatron.com\/blog\/portscanner-only-60-lines-of-code\/","url_meta":{"origin":556,"position":4},"title":"PortScanner &#8211; Only 60 lines of code","author":"gubatron","date":"May 19, 2005","format":false,"excerpt":"Hi, here's a simple port scanner I wrote today in Java. I needed to see something on my server, perhaps you can find it useful. I would say its pretty fast, you can tweak the number of threads and timeouts to adjust to your server\/connection. I did't do much testing\u2026","rel":"","context":"In &quot;Gubatron&quot;","block_context":{"text":"Gubatron","link":"https:\/\/www.gubatron.com\/blog\/category\/gubatron\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":516,"url":"https:\/\/www.gubatron.com\/blog\/ejemplo-de-automatizacion-entre-2-maquinas-remotas-con-bash-scripting-y-python\/","url_meta":{"origin":556,"position":5},"title":"Ejemplo de automatizacion entre 2 maquinas remotas con bash scripting y Python","author":"gubatron","date":"May 10, 2007","format":false,"excerpt":"Para los amigos que se inician en el mundo *nix, ya sea con su nueva Mac, o con una PC corriendo Linux, les recomiendo que aprendan a manejar bien los siguientes lenguajes, y el mundo sera suyo: - bash scripting (aliases, variables, exports, iteraciones, condicionales) - python (para programar logica\u2026","rel":"","context":"In &quot;Code&quot;","block_context":{"text":"Code","link":"https:\/\/www.gubatron.com\/blog\/category\/code\/"},"img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]}],"_links":{"self":[{"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/posts\/556","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/comments?post=556"}],"version-history":[{"count":0,"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/posts\/556\/revisions"}],"wp:attachment":[{"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/media?parent=556"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/categories?post=556"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.gubatron.com\/blog\/wp-json\/wp\/v2\/tags?post=556"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}