Everytime I need to do so, I always end up coding functions to convert a (x,y) coordinate to the real index n in the array.

Let me illustrate, with an example. You have a string that defines the elements of a game board, and you want to work using (x,y) coordinates.

s="xxxxx@xx@xxx@@xx"

If you were to look at it as a matrix (width=4), it’d be something like this

s="xxxx
   x@xx
   @xxx
   @@xx"

However, I can’t do
s[x,y], since it’s a linear array, it’s a string.

You need to convert from (x,y) to a number that represents an index in the array.

This is very simple:

width=4
def getNforXY(x,y):
  return x + width*y

What if you want to do it backwards. What if you need to know what’s the X and Y for a given index N in the string?

def getXYforN(n):
  y = int(n/width)
  x = n - width/y
  return (x,y)

Cheers

                          

How I keep my wordpress updated by hand
I tend to keep my wp-content folder outside of my wordpress installation for 2 reasons:

1. I don’t like to loose my themes, plugins and customizations
2. I like to keep all my customization changes under subversion

So, if I had my wordpress installation say at:
/home/user/public_html/blog

I’d keep my wp-content folder for that here:

/home/user/public_html/wp-content-for-blog

So when I upgrade my blog, I always remove the original wp-content folder that comes along wordpress, and I symlink my hard worked on wp-content folder that lives outside to the freshly unzipped wordpress folder.

user@machine:~/public_html/blog$ ls -l
...
lrwxrwxr-x 1 user www    54 2008-11-26 09:29 wp-content -> /home/user/public_html/wp-content-for-blog
...

So what I endup doing all the time, is downloading the latest.zip to ~/public_html/, it will unzip under ~/public_html/wordpress, and then I’ll copy the current ~/public_html/blog/wp-config.php to ~/public_html/wordpress, then I’ll remove the default ~/public_html/wordpress/wp-content and symlink the outer wp-content with all my customizations, themes and plugins to it. Once done, I’ll make a backup of the old wordpress folder, and then I’ll rename wordpress folder to the name of the blog folder, and it’s all done.

It’s simple, but when you have to do it for 5 blogs, every week, it’s not fun anymore.

The Update Script

So here’s a script to do it in one step. If you’re not using my symlinked technique, this will do it for you, you only need to specify the full path to the folder where you want to keep your current wp-content folder outside the new installation before you apply the update, and the name of the folder where your current blog lives. The script below will have its configuration variables towards the beginning set so that they are in line with the example I’ve been talking about.

#!/usr/bin/python
#########################################################################################
#
# upgrade_wordpress.py - Script to automatically upgrade your wordpress installation.
#
# Requirements:
#   - Python 2.4 or older
#   - WordPress should already be installed
#   - CURL (sudo apt-get install curl)
#
# Author: Angel (Gubatron) Leon
# LICENSE: See the GPL2 license.
# 2008
#########################################################################################
import os

#########################################################################################
#Config (relative to the folder where this script will be run from)
#########################################################################################

#The current folder where the blog lives
BLOG_FOLDER='blog'

#
# The first time you run the script, it will try to make a copy of your
# current wp-content folder outside. Copy here the location of where
# the wp-content folder with your themes and plugins should exist.
#
# After it unzips, it will remove the default wp-content folder from
# the new installation, and it will symlink the external wp-content
# That way you don't ever have to worry about loosing your customizations
# and plugins.
#
WP_CONTENT_OUTSIDE_COPY_FOLDER="/home/user/public_html/wp-content-for-blog"

#This is where a backup of your current blog will be
BLOG_FOLDER_BACKUP_FOLDER=BLOG_FOLDER+'.old'

#Where to download the wordpress latest.zip from
WORDPRESS_LATEST_ZIP_URL='http://wordpress.org/latest.zip'

#### DO NOT MODIFY AFTER THESE LINES ####

def downloadWordPress(url=WORDPRESS_LATEST_ZIP_URL):
    if os.path.exists('latest.zip'):
        print "Removing old latest.zip"
        os.remove('latest.zip')

    #Try to download with CURL
    print "Attempting to download latest.zip from wordpress.org"
    os.system('curl %s -o latest.zip' % url)

    if not os.path.exists('latest.zip'):
        os.system('wget ' + url)

    return os.path.exists('latest.zip')

def dirExists(dirName):
    return os.path.exists(dirName) and os.path.isdir(dirName)

def backupBlog(currentBlogFolder=BLOG_FOLDER,
               wpContentOriginalFolder=WP_CONTENT_OUTSIDE_COPY_FOLDER,
               backupFolder=BLOG_FOLDER_BACKUP_FOLDER):

    #Remove any previous backups
    if os.path.exists(backupFolder) and os.path.isdir(backupFolder):
        print "Removing previous backup folder"
        os.system('rm -fr ' + backupFolder)

    #Copy the current blog folder into a backup folder just in case.
    #We won't do any database backups for now.
    print "Creating new backup folder"
    os.system('cp -r %s %s' % (currentBlogFolder,backupFolder))

    #Check for the copy of wp-content outside the blog, if it doesn't exist
    #we'll make it for the first time.
    if not dirExists(wpContentOriginalFolder):
        print "Creating outside copy of wp-content"
        os.system('cp -r %s %s' % (os.path.join(currentBlogFolder,'wp-content'),
                                   wpContentOriginalFolder))

    #Copy the latest wp-config.php outside to the current folder
    print "Copying your latest wp-config.php outside"
    os.system('cp %s .' % (os.path.join(currentBlogFolder,'wp-config.php')))

    backupFolderExists = dirExists(backupFolder)
    wpContentFolderExists = dirExists(wpContentOriginalFolder)
    configFileExists = os.path.exists('wp-config.php')

    return backupFolderExists and wpContentOriginalFolder and configFileExists

def upgradeBlog(currentBlogFolder=BLOG_FOLDER,
                backupFolder=BLOG_FOLDER_BACKUP_FOLDER,
                url=WORDPRESS_LATEST_ZIP_URL,
                wpContentOriginalFolder=WP_CONTENT_OUTSIDE_COPY_FOLDER):

    if not downloadWordPress(url):
        print "Could not download latest.zip, aborting."
        return False

    if not backupBlog(currentBlogFolder,wpContentOriginalFolder,backupFolder):
        print "Could not backup blog or wp-config.ph, aborting."
        return False

    if currentBlogFolder == 'wordpress':
        print "The current blog folder cannot be 'wordpress, aborting."
        return False

    #1. If a wordpress/ folder exists, wipe it.
    if dirExists('wordpress'):
        print "Removing old wordpress folder"
        os.system('rm -fr wordpress')

    if dirExists('%s.delete' % currentBlogFolder):
        print "Removing old %s.delete folder" % currentBlogFolder
        os.system('rm -fr %s.delete folder' % currentBlogFolder)

    #2. Unzip new copy
    os.system('unzip latest.zip')

    if not dirExists('wordpress'):
        print "Could not unzip the wordpress installation, aborting."
        return False

    #1. Copy wp-config.php into the new installation
    os.system('cp wp-config.php wordpress/')

    #2. Remove the default wp-content folder
    os.system('rm -fr wordpress/wp-content')

    #3. Symlink the original wp-content that lives outside
    os.system('ln -s %s wordpress/wp-content' % (wpContentOriginalFolder))

    #4. Verify symlink was created
    if not (os.path.exists('wordpress/wp-content') and os.path.islink('wordpress/wp-content')):
        print "Could not create symlink to wp-content, aborting."
        return False

    #5. Move original folder to folder.delete, and make this wordpress folder the current folder.
    os.system('mv %s %s.delete' % (currentBlogFolder,currentBlogFolder))

    if not dirExists(currentBlogFolder + ".delete"):
        print "Could not rename current folder for later deletion, aborting."
        return False

    #6. Rename the new installation as the current blog
    os.system('mv %s %s' % ('wordpress',currentBlogFolder))

    if dirExists('wordpress'):
        print "ALERT: The wordpress folder still exists."
        return False

    if not dirExists(currentBlogFolder):
        print "ALERT: The blog doesn't exist, recover from the backup folder %s please" % (backupFolder)
        return False

    #7 Cleanup
    os.system('rm -fr %s.delete' % (currentBlogFolder))

    return True

if __name__ == '__main__':
    upgradeBlog()

Requirements

Installation

Usage:

python upgrade_wordpress.py

The script is very fault proof, it will always try to abort in case something is not going the way it’s expected. At the end of the day it’ll also leave a backup copy of your current blog in case something goes bad, you can always recover.

                          

With the help of the last post on how to ban an IP, and the following python script, you’ll be able to have a cronjob that runs once or twice a day and automagically bans all the offending ips from ever trying to brute force their way in ever again.

touch and make executable a file called “detect_ssh_hostiles”

touch detect_ssh_hostiles
chmod +x detect_ssh_hostiles

Then copy the following code inside:

# Usage:
# python detect_ssh_hostiles [auth.log file path]
#
# Requirement: There should be "ban_ip" and "unban_ip" command availability on the path
#
# Note: you gotta have read permissions on the auth.log file and sudo
#       permissions for the script to ban the ips.

#If an IP meets this number of failed login attemmpts it will be banned
BAN_THRESHOLD = 7
SUSPECTS = {}

#Put here IP addresses you trust, could be making genuine login errors
SAFE_IPS = ['81.73.111.49','101.73.111.160','72.31.171.235','72.36.23.234','82.36.180.210','202.132.82.16']

import os
import sys
import re

BANNED = {}
def loadBanned():
  '''
  This function will load all the banned IPS into the BANNED Dict.
  It will also count how many times (by mistake) the same IP has
  been banned, and it will unban it, so that it will appear only once.
  '''
  global BANNED
  command = 'sudo iptables --list --numeric'
  try:
    p = os.popen(command,'rb')
  except Exception,e:
    print e
    sys.exit(1)

  line = '-'

  while line != '':
    line = p.readline().strip()

    if line.startswith("DROP"):
      parts = line.split()
      ip = parts[3]

      #add hit or register banned ip
      if BANNED.has_key(ip):
        BANNED[ip]+=1
      else:
        BANNED[ip]=1

  #Make sure banned IPs are banned only once
  for ip in BANNED:
    if BANNED[ip] > 1:
      print "IP %s has been banned %d times" % (ip, BANNED[ip])
      n=BANNED[ip]-1
      while n > 0:
        os.system("unban_ip %s" % ip)
        print ("unban_ip %s" % ip)
        n=n-1

  p.close()

# ---- here we go ----
loadBanned()

#read auth log
logfile = '/var/log/auth.log'

if len(sys.argv)==2:
  logfile = sys.argv[1]

command = 'grep "Failed password for " %s' % logfile
#print command

try:
  p = os.popen(command,'rb')
except Exception,e:
  print e
  sys.exit(1)

line = "123"

while line != '':
  line = p.readline()

  #Sample line:
  # May 25 03:29:49 main sshd[6933]: Failed password for root from 202.118.236.132 port 54863 ssh2
  pattern = "(.*)(from\s)(\d+\.\d+\.\d+\.\d+)(.*)"
  matchObject = re.match(pattern, line)

  suspect = None
  if matchObject is not None:
    suspect = matchObject.groups()[2]

    #skip safe IPs
    if suspect in SAFE_IPS:
      continue

    if SUSPECTS.has_key(suspect):
      #add a hit
      SUSPECTS[suspect] += 1
    else:
      #add first hit
      SUSPECTS[suspect] = 1

p.close() #close the pipe

print "=="*30

import time
t = time.localtime()
#(2008, 6, 6, 9, 35, 21, 4, 158, 1)

timestr = "%d-%d-%d@%d:%d:%d" % (t[0],t[1],t[2],t[3],t[4],t[5])
print timestr
print "--"*30
if len(SUSPECTS) > 0:
  for suspect in SUSPECTS:
    if SUSPECTS[suspect] >= BAN_THRESHOLD and not BANNED.has_key(suspect):
      print "Banning %s with %d attempts" % (suspect,SUSPECTS[suspect])
      BANNED[suspect]=1
      os.system("ban_ip %s" % suspect)
    elif BANNED.has_key(suspect):
      print "Ip %s has already been banned" % (suspect)
    else:
      print "Suspect candidate? %s with %d attempts" % (suspect,SUSPECTS[suspect])
else:
  print "Found no suspects to ban"

print "=="*30

Then add this as a cronjob of your root user, and it will automatically ban all those IPs that have tried to break in. See the script for configuration. You can always make some IPs immune to banning by adding them on the SAFE_IPS list.

                          

Over 1,000 people gathered at the Crowne Plaza Chicago O’Hare Hotel for PyCon 2008. The influx of proposals was so great this year that organizers could only accept half the proposals received, simply due to lack of space. Demand for tutorials, in particular, rose so high that organizers added an extra timeslot full of new sessions. Twenty-five of the 28 tutorials were sold out and there were 22 open source development sprint projects available for attendees to participate in, up from 13 in 2007. More than 270 people took part in the development sprints, which is more than attended the entire first PyCon in 2003.

The best part about PyCon is the people you get to meet
“As jam-packed as the week’s events were, a number of attendees and speakers remarked to me that they felt the conference this year was the best yet,” said PyCon 2008 chair David Goodger. “It’s great to see months of work come together and have people really enjoy it.”

PyCon 2008 is organized and run entirely by volunteers. A true grassroots movement, it is a community conference put on by the community for the community. Many attendees enjoy the numerous sessions, but more importantly, they believe the real value of the conference is the community all being there together.

“The best part about PyCon is the people you get to meet,” said Chris McAvoy, founder and president of the Chicago Python Users Group, which hosted PyCon 2008. “That, coupled with more than 1,000 attendees, compared to the 600 last year, says to me that Python is really building a market here in Chicago and elsewhere. It’s an exciting time to be a Python developer.”

PyCon 2009 will also be held in Chicago. For more information about PyCon, please visit http://us.pycon.org.

About PyCon

Presented by the Python Software Foundation, the world’s largest Python conference brings together a diverse group of developers, enthusiasts, and organizations to explore new challenges, launch new businesses and forge new connections within the Python community. PyCon provides attendees with the opportunity to delve into the dynamic programming language employed by well-known companies such as Google, Cisco, and the New York Times. PyCon helps people learn new tools and techniques, showcase projects, and meet other Python fans.

Source: PRWeb.com

                          

Basically, sometimes you don’t want to store IPs in Strings cause they take too much space, instead you want to be a good programmer and store them as 4 bytes (IPv4 that is).

So here’s a couple of functions in python to illustrate the conversion process between string to 4 byte integer, or viceversa:

def ip2num(ipString):
    if ipString is None:
        raise Exception("Invalid IP")

    try:
       octets = [octet.strip() for octet in ipString.split('.')]
    except Exception,e:
        raise e

    num = (int(octets[0])<<24) + (int(octets[1])<<16) + (int(octets[2])<<8) + int(octets[3])
    return num

def num2ip(numericIp):
    if numericIp is None or type(numericIp) != int:
        raise Exception("Invalid numeric IP. Must be an integer")
    return str(numericIp >> 24) + '.' + str((numericIp >> 16) & 255) + '.' + str((numericIp >> 8) & 255) + '.' + str(numericIp & 255)

                          


>>> PyQt4.Qt.qVersion()
'4.3.3'


                          

Lessons Learned:
* Use QMutex to protect data, not code. Try not to lock hughe amounts of code within a function with mutex.lock(), mutex.unlock(), if for any reason you forget to release the lock you’ll be in trouble. Use the mutex directly only when you know what it is that you want to protect concurrent access from.
* When you have a complex function and you don’t want to worry about what to protect and when to release the lock (on exceptions thrown, before returns,etc), you can create an instance of QMutexLocker and it should release the mutex lock upon destruction… this takes us to the next lesson
* When using a QMutexLocker, DO NOT make the QMutexLocker an attribute of your class, otherwise, the reference will live after the method finishes and the lock won’t be released.

Here’s some code.

from PyQt4.Qt import QObject, QMutex, QApplication, QThread, QMutexLocker
import sys

class MutexTestSubject(QObject):
	'''
	Class that uses a QMutex to synchronize
        access to its add(),substract() methods.

        This works perfectly fine.
	'''
	def __init__(self):
		QObject.__init__(self)
		self.MAX_LIMIT = 100
		self.MIN_LIMIT = 0
		self.counter = 50
		self.mutex = QMutex()

	def add(self):
		self.mutex.lock()
		if self.counter < self.MAX_LIMIT:
			self.counter = self.counter + 1
		self.mutex.unlock()

	def substract(self):
		self.mutex.lock()
		if self.counter > self.MIN_LIMIT:
			self.counter = self.counter - 1
		self.mutex.unlock()

	def printStatus(self,thread):
		print "Counter:",self.counter," - Thread:",id(thread)

		if self.counter > self.MAX_LIMIT+1 or self.counter < self.MIN_LIMIT:
			print "Stopping Threads, Max Surpassed, Not Thread Safe. Last Thread:",id(thread)
			sys.exit()

class MutexLockerTestSubject(QObject):
	'''
	Class that attemps to synchronize thread
 	access to its add(),substract() methods with
	the QMutexLocker object.
	'''
	def __init__(self):
		QObject.__init__(self)
		self.MAX_LIMIT = 100
		self.MIN_LIMIT = 0
		self.counter = 50
		self.mutex = QMutex()

	def add(self):
		#VIP: DO NOT MAKE mutexLocker an attribute of your class.
		#other wise it won't be destroyed and the lock will never be released.
		mutexLocker = QMutexLocker(self.mutex)
		if self.counter < self.MAX_LIMIT:
			self.counter = self.counter + 1

	def substract(self):
		mutexLocker = QMutexLocker(self.mutex)
		if self.counter > self.MIN_LIMIT:
			self.counter = self.counter - 1

	def printStatus(self,thread):
		print "Counter:",self.counter," - Thread:",id(thread)

		if self.counter > self.MAX_LIMIT+1 or self.counter < self.MIN_LIMIT:
			print "Stopping Threads, Max Surpassed, Not Thread Safe. Last Thread:",id(thread)
			sys.exit()

class AdderWorker(QThread):
	def __init__(self, mutexTestObject):
		self.mutexTestObject = mutexTestObject
		QThread.__init__(self)

	def run(self):
		while(True):
			self.mutexTestObject.add()
			self.mutexTestObject.printStatus(self)

class SubstractorWorker(QThread):
	def __init__(self, mutexTestObject):
		self.mutexTestObject = mutexTestObject
		QThread.__init__(self,mutexTestObject)

	def run(self):
		while(True):
			self.mutexTestObject.substract()
			self.mutexTestObject.printStatus(self)

if __name__ == "__main__":
	USE_MUTEX_LOCKER = True #switch this to use regular mutexes vd QMutexLocker
	app = QApplication(sys.argv)
	mutexTestObject = MutexTestSubject() if not USE_MUTEX_LOCKER else MutexLockerTestSubject()

	adderThread  = AdderWorker(mutexTestObject)
	substracterThread = SubstractorWorker(mutexTestObject)

	adderThread.start()
	substracterThread.start()

	sys.exit(app.exec_())

                          

Usually, if you have a binary field, and you want to, turn on bits, toggle bits, and check bits, you do the following.

Suppose “config” is an 8 bit number, and you want to modify this bits individually, say config=4 (0b100) and you want to turn on the rightmost bit to have 0b101 (which is a 5) you could do the following

>>> #Uses the | operator
... def turnBitOn(config, binaryFlag):
...   return config | binaryFlag
...
>>> config = 4
>>> config = turnBitOn(config,1)
>>> print config
5

If you want to check if a Bit is turned on, just use the “&” operator, if the result is the same as the bit you’re comparing, then its turned on.

>>> def checkBit(config,binaryFlag):
...   return binaryFlag == config & binaryFlag
...
>>> print checkBit(config,1)
True
>>> print checkBit(config,2) #checks 0b010 in 0b101
False

What if you just want to toggle a bit, no matter what’s in there?
Just use binary XOR, the “^” operator.

>>> def toggleBit(config,binaryFlag):
...   return config ^ binaryFlag
...
>>> print config
5
>>> config = toggleBit(config,1)
>>> print config
4
>>> config = toggleBit(config,1)
>>> print config
5

And now, the last basic operation would be to turn off a bit. For this, you should do a NAND operation. The Binary Not in python as in most programming languages is the “~” operator. This is how you can use it to turn off bits.

You have to do it in conjunction with the & operator, sort of doing a NAND

>>> 5 & (~1)
4
>>> 5 &~ 1
4

So we could define our turnOffBit function as follows:

>>> def turnOffBit(config,binaryFlag):
...   return config & (~binaryFlag)
...
>>> print config
5
>>> config = turnOffBit(config,1)
>>> print config
4
>>> config = turnOffBit(config,1)
>>> print config
4

Hope this makes a good reference for those trying to make the most out of their bytes.

                          

The ouput will have useful debugging info about the HTTP connection.

connect: (localhost, 8080)
send: u'POST /arcturus-web/fileVariableDownload.service HTTP/1.1\r\nAccept-Encoding: identity\r\nContent-Length: 57\r\nConnection: close\r\nUser-Agent: Temboo Twyla/Arcturus HTTP Downloader\r\nHost: localhost:8080\r\nCookie: JSESSIONID=1b8xl8nozb2i\r\nContent-Type: application/x-www-form-urlencoded\r\n\r\n'
send: 'URI=temboo%3A%2F%2Fwww.one.com%2Ffiles.f%2Fmapping1.m.var'
reply: 'HTTP/1.1 200 OK\r\n'
header: Content-Language: en-US
header: Content-Type: text/html; charset=ISO-8859-1
header: Connection: close
header: Server: Jetty(6.0.2)

                          

Bottom line:

If a signal has been defined by Qt, to send Qt objects, just copy and paste it, do not try to override it by exchanging the Qt objects for PyQt_PyObject.

This is because PyQt_PyObject is used to represent regular Python objects, therefore, original Qt4 classes will never emit signals under those weird signatures, Qt4 seems to be very strict on how it matches signals.

So when you see something like this on the documentation:

Qt Signals
void currentItemChanged (QTreeWidgetItem *,QTreeWidgetItem *)

Define your signal for example like this, and it will work:

SIGNAL_ITEM_CHANGED = SIGNAL('currentItemChanged (QTreeWidgetItem *,QTreeWidgetItem *)')

Do not do this, cause it won’t work:

SIGNAL_ITEM_CHANGED = SIGNAL('currentItemChanged (PyQt_PyObject, PyQt_PyObject)')

(Keep in mind PyQt_PyObject is used to represent Python objects only, and its useful only when YOU define and emit your own signals)

then connect it from a QTreeWidget item (or derived object) to a listener object’s method like this:

QObject.connect(myQTree,SIGNAL_ITEM_CHANGED,myListener.onItemChanged)

Your object’s listener method should look like this:

def onItemChanged(self, current, previous):
  #do what you gotta do, current and previous will be QTreeWidgetItems

                          

I wrote this script on the same folder where the pictures with the lagged modification times existed:

import os
import time
from stat import *

#returns a list of all the files on the current directory
files = os.listdir('.')

for f in files:
  #my folder has some jpegs and raw images
  if f.lower().endswith('jpg') or f.lower().endswith('crw'):
    st = os.stat(f)
    atime = st[ST_ATIME] #access time
    mtime = st[ST_MTIME] #modification time

    new_mtime = mtime + (4*3600) #new modification time

    #modify the file timestamp
    os.utime(f,(atime,new_mtime))

Very nice, I like

                          

Para los que no recuerdan quicksort es simplemente hacer 3 listas con la lista principal [PRINCIPAL]. Estas listas se generan bajo un criterio bien sencillo. Elige un numero cualquiera del arreglo, la primera lista tendra todos los numeros menores a ese numero [MENORES]. La segunda lista tendra todos los numeros que sean de igual valor a ese numero [IGUALES], y la tercera los mayores [MAYORES].

Una vez que tienes estas 3 listas, el resultado de Quicksort es el siquiente (Donde ++ es concatenar una lista):

Quicksort([MENORES]) ++ [IGUALES] ++ Quicksort([MAYORES])

Como veras, el algoritmo es recursivo, y llegara el momento que no encuentres numeros menores, iguales o mayores, asi que cuando Quicksort recibe una lista de un solo elemento, devuelve ese elemento.

En python esto sale en 4 lineas con list comprehension, literalmente:

def qs(lista):
   if len(lista) <= 1: return lista
   p = lista[0] #el pivote
   return qs([x for x in lista if x < p]) + [x for x in lista if x == p] + qs([x for x in lista if x > p])

Pudieras aplicar este algoritmo a cualquier tipo de objeto en Python si para ese objeto defines las funciones de comparacion.
Demasiado arrecho, viva python.

PS: Despues de escribir esto, encontre una implementacion aun mas corta, en la que no utilizan list comprehension, utilizan
funciones lambda, en vez de hacer [x for x in lista if x < pivote] hacen filter(lambda x,y=pivote: x < y,lista), eso devuelve los elementos que cumplen con la funcion lambda en la lista, es decir filter(funcion, lista)

                          

- bash scripting (aliases, variables, exports, iteraciones, condicionales)
- python (para programar logica mas compleja y portatil en cualquier sistema operativo)
- Uso de comandos como grep, egrep, awk (editor de streams) entre otros
- Expresiones regulares

En wedoit4you.com hicimos un simple script que va logeando visitas desde los blogs registrados.
Los blogeros ponen un pedacito de javascript, que al ser invocado, escribe una entrada en un log en el servidor.

Luego tenemos un script que analiza ese log, elimina cualquier intento de hacer muchos clicks, etc. etc.
Ese script se encarga luego de hacer matching de las URLs entrantes, con las URLs de los posts que wedoit4you.com
ya leyo. Este script lamentablemente tarda mucho en analizar mas de 150mb de data, mas lo que haya en el log,
y Dreamhost lo mata si dura mas de un minuto, o si hay mas de N procesos corriendo.

Que hacemos entonces?

Ponemos ese script en una maquina local, donde tenemos todo el cpu, y hacemos que el servidor a ciertas horas
del dia, haga un mysqldump de las tablas que me interesan (BLOGS, BLOG_POSTS, POST_HITS) y meta eso en un archivo
disponible via HTTP.


#!/bin/bash
DUMP_DIR=/home/cuenta_en_server/sitio.com/temp/
SQL_FILE_NAME=clicktrackr_dump.sql
SQL_FILE=${DUMP_DIR}/${SQL_FILE_NAME}
TGZ_FILE=${SQL_FILE_FNAME}.tar.gz
mysqldump bd_en_servidor BLOGS BLOG_POSTS POST_HITS > ${SQL_FILE}
cd ${DUMP_DIR}
pwd
echo Making tar
tar cfz ${TGZ_FILE} ${SQL_FILE}
echo Tar with SQL dump ready to be downloaded.
echo Finished.


Este script corre digamos a las 4am en el servidor.

Luego desde la casa
Luego en la maquina local corre un cronjob a las 4:30am, En mi caso una apple iMac Intel, y he aqui el poder de tener una Mac basada en Unix, y no la cagada de windows de mierda.

Hice sencillo bash script que se baja ese dump de la base de datos, y se baja el log del clickTracker para hacer los calculos en mi cpu,
(con el cual hago lo que me da la gana), los calculos son realizados con un script en python (click_tracker.py, incluido al final)
y una vez que termina de calcular, hace ftp de vuelta hacia el servidor y sube un archivo SQL
con instrucciones SQL para que se actualizen los hits de los posts. Este archivo de lolas que no lo subo a un directorio disponible en apache
pq alguien podria meterse con el y alterarnos los hits… Este Script local luce asi:


#!/bin/bash
rm /Users/gubatron/clicktrackr/*.tar.gz
rm /Users/gubatron/clicktrackr/*.sql
rm /Users/gubatron/clicktrackr/*.dat
rm /Users/gubatron/clicktrackr/*.log
echo "Downloading dump from server..."
wget http://www.wedoit4you.com/dir_del_dump/clicktrackr_dump.sql.tar.gz -O /Users/gubatron/clicktrackr/clicktrackr_dump.sql.tar.gz
cd /Users/gubatron/clicktrackr/
echo "Uncompressing Dump..."
tar xfz clicktrackr_dump.sql.tar.gz
echo "Loading data in MySQL"
mysql --user=usuario --password=password --database=bd_local < clicktrackr_dump.sql
echo "Downloading latest tracker.log"
wget http://www.wedoit4you.com/xxxxxxxxx/logs/tracker.log -O /Users/gubatron/clicktrackr/tracker.log
echo "Crunching Data with python script"
python click_tracker.py
echo "Compressing data crunched"
tar cvfz clicktrackr_update_tables.sql.tar.gz clicktrackr_update_tables.sql
echo "Uploading data"
#then upload tar.gz clicktrackr_update_tables.sql
ftp -u ftp://usuario:password@wedoit4you.com/directorioNoAccesiblePorApache/ clicktrackr_update_tables.sql.tar.gz
echo "Finished"


La salida de este script cuando se ejecuta es similar ea esto

imac:~ gubatron$ clicktrackr_processing
Downloading dump from server...
--09:45:14-- http://www.wedoit4you.com/xxxxxxx/clicktrackr_dump.sql.tar.gz
=> `/Users/gubatron/clicktrackr/clicktrackr_dump.sql.tar.gz'
Resolving www.wedoit4you.com... 208.113.146.143
Connecting to www.wedoit4you.com|208.113.146.143|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 41,227,814 [application/x-tar]

Crunching Data with python script
/Users/gubatron/clicktrackr
No timestamp from last time found.
Loading data from ClickTrackr log...
Saving ClickTrackr data to File...
Saving completed.
Loading Blogs and Last Posts from DB
Saving Blogs to File...
Saving completed.
Loading Posts from DB...
Saving Posts from DB on file
Saving completed.
Crunching data...
0 converted from blog to last post
Didnt find total 11 urls.
Didn't find distinct 11 urls.
Saving crunched data...
Data saved.
Writing SQL...
Finished Writing SQL
Wrote last timestamp.
Compressing data crunched
clicktrackr_update_tables.sql -> clicktrackr_update_tables.sql.tar.gz
Uploading data
Connected to wedoit4you.com.
220 ProFTPD 1.3.0rc2 Server (DreamHost FTP) [208.113.146.143]
331 Password required for wedoit4y.
230 User wedoit4y logged in.
Remote system type is UNIX.
Using binary mode to transfer files.
200 Type set to I
250 CWD command successful
local: clicktrackr_update_tables.sql.tar.gz remote: clicktrackr_update_tables.sql.tar.gz
229 Entering Extended Passive Mode (|||57539|)
150 Opening BINARY mode data connection for clicktrackr_update_tables.sql.tar.gz
100% |*************************************| 550 KB 155.72 KB/s 00:03
226 Transfer complete.
563661 bytes sent in 00:03 (143.48 KB/s)
Finished


Una vez que la data fue procesada y FTPeada al servidor, hay otro cronjob que corre una hora mas
tarde, y este asume que el nuevo archivo con la data procesada estara ahi, podriamos agregarle mas
checks, utilizando "stat" y anotando el ultimo timestamp del sql utlizado la vez anterior cosa que no
volvamos a anotar los hits del dia anterior...

Asi luce el script para actualizar finalmente en el servidor:


#!/bin/bash
DIR_PRIVADO=/home/usuario/dirPrivado
PATH_DEL_TRACKER_LOG=/home/usuario/algunaCarpeta/tracker.log
cd ${DIR_PRIVADO}
rm *.sql
tar xvfz clicktrackr_update_tables.sql.tar.gz
mysql bd_en_servidor < ${DIR_PRIVADO}/clicktrackr_update_tables.sql
rm *.tar.gz
rm *.sql
rm
touch ${PATH_DEL_TRACKER_LOG}
chmod 777 ${PATH_DEL_TRACKER_LOG}


Si tienes curiosidad de ver como cruncheo la data localmente, aqui esta el codigo en python.
(Es aun un trabajo en progreso)

#!/home/wedoit4y/bin/python/bin/python2.5
# This is the script that processes the ClickTrackr Log
import os
import sys
import pickle
import time

#NAMES OF FILES WHERE WE'LL STORE THE DIFFERENT STAGES OF RETRIEVED
#AND PROCESSED DATA.

#File that holds a dictionary with URLs and HITs we got from the original log file
FILE_MAX_AGE=3600*1
FILE_TIMESTAMP="clicktrackr_last_timestamp.dat"
FILE_001="clicktrackr_001_url_hits.dat"
FILE_002="clicktrackr_002_blogs_lastposts.dat"
FILE_003="clicktrackr_003_posts_hits.dat"
FILE_004="clicktrackr_004_processed_hits.dat" #...and urls not found
FILE_SQL="clicktrackr_update_tables.sql"

try:
    import snowrss_config
    from snowrss_config import getDbCursor
    #from snowrss import *
except Exception,e:
    print "Could not import snowrss_config [%s]" % e
    sys.exit()

def dbExec(sql):
    """Give it some SQL and it will return the returning cursor"""
    try:
        cursor = getDbCursor()
        cursor.execute(sql)
        cursor.connection.close()
    except Exception, e:
        #MySQL has gone away
        print 'dbExec(%s): ' % unicode(sql)
        print e
        return None
    return cursor

def isFileFresh(fileName):
    """
    Returns True if the file is still good to be used.
    Othewise returns false
    """
    try:
        file_stat = os.stat(fileName)
        file_age = time.time() - file_stat.st_mtime
        if file_age > FILE_MAX_AGE:
            return False
        return True
    except:
        return False

def getData(line):
    """Returns a dict with, IP, Timestamp, URL and User Agent if found

    Parameters
        line - A Line with a ClickTracker log entry

    Output
        {'ip':...,'timestamp':...,'url':....,'ua':...}
        ip-> IP Addres
        time -> Time of the event
        url -> Referer Url
        ua -> User Agent of the rerferer user
    """
    l = line.split()
    result = {}
    result['ip']=l[0]
    result['time']=l[1]
    result['url']=l[2]

    result['ua']='N/A'
    if len(l)>3:
        rest = l[3:]
        ua_name = ''
        for b in rest:
            ua_name = ua_name + ' ' + b
        result['ua'] = ua_name

    return result

#Maximum time to count a click from the same IP on the same URL
TIME_BETWEEN_CLICKS = 12*3600

#On the last run (if finished, we write down the time of the last timestamp on file)
#If we did finish a run, we'll get this number from the timestamp file, and we'll ignore
#all previous log entries to that timestamp.
LAST_TIMESTAMP = None
POSSIBLE_LAST_TIMESTAMP = None

try:
    f = fopen(FILE_TIMESTAMP,"rb")
    LAST_TIMESTAMP = pickle.load(f)
    LAST_TIMESTAMP = long(LAST_TIMESTAMP)
    f.close()
except:
    print "No timestamp from last time found."

urls = {}
urls_not_found = {}
LOG_CLICK_TRACKER='tracker.log'

#check if there is a version of the log file backed that's still good enough to be used.
USABLE_LOG_FILE = LOG_CLICK_TRACKER

#use a copy of the log if we got some pickled data
if isFileFresh(LOG_CLICK_TRACKER + '.last') and isFileFresh(FILE_001):
    USABLE_LOG_FILE = LOG_CLICK_TRACKER + ".last"

IGNORED_ENTRIES = 0
if not isFileFresh(FILE_001):

    #open the tracker log (current or old)
    print "Loading data from ClickTrackr log..."
    f = open(USABLE_LOG_FILE,'r')

    f.seek(0,2)
    eof = f.tell()
    f.seek(0)

    while f.tell() < eof:
        entry = getData(f.readline())

        url = entry['url']
        ip = entry['ip']
        timestamp = entry['time']

        if LAST_TIMESTAMP is not None and long(timestamp) < LAST_TIMESTAMP:
            print "i",
            IGNORED_ENTRIES += 1
            continue

        POSSIBLE_LAST_TIMESTAMP = long(timestamp)

        if not url.startswith('http') or \
           url.startswith('http://babelfish.altavista.com') or \
           url.startswith('http://6'):
            #IGNORED_ENTRIES += 1
            continue

        #Ask if this URL is already there
        if urls.has_key(url):
            #Ask if this IP is already there
            if urls[url].has_key(ip):
                #Get the last time stamp inside this IP
                times = urls[url][ip]
                last_time = times[len(times)-1]
                delta_time = long(timestamp) - long(last_time)
                #If its been more than acceptable time
                if delta_time >= TIME_BETWEEN_CLICKS:
                    urls[url][ip].append(timestamp)

                    hits = 0
                    for ipbuffer in urls[url]:
                        if ipbuffer == 'hits': #just count the keys that are not 'hits'
                            continue
                        hits += len(urls[url][ipbuffer])

                    urls[url]['hits'] = hits
            else:
                urls[url][ip] = [timestamp]
                urls[url]['hits'] = 1
        else:
            urls[url]={}
            urls[url][ip] = [timestamp]
            urls[url]['hits']=1
    f.close()

    urls['POSSIBLE_LAST_TIMESTAMP'] = POSSIBLE_LAST_TIMESTAMP

    #we serialize this data for later
    if IGNORED_ENTRIES > 0:
        print "Ignored %d entries." % IGNORED_ENTRIES

    print "Saving ClickTrackr data to File..."
    f = file(FILE_001,"wb")
    pickle.dump(urls,f)
    f.close()
    print "Saving completed."

    #we make a backup of the current ClickTrackr log (.last), in case we need to run again
    #we can diff with this to know from where to relog in the future
    os.system("cp %s %s" % (LOG_CLICK_TRACKER,LOG_CLICK_TRACKER + ".last"))
else:
    #we unserialize the data
    print "Loading ClickTrackr data from existing file..."
    f = file(FILE_001,"rb")
    urls = pickle.load(f)
    f.close()
    POSSIBLE_LAST_TIMESTAMP = urls.pop('POSSIBLE_LAST_TIMESTAMP') #we popup so we have only urls and we dont modify further down
    print "Loading completed."

#LOAD ALL BLOG POST URLS, IDS AND CURRENT NUMBER OF HITS.
blog_urls = {} #blogs hashed by their urls, Buckets have {'post_id':,'post_link':}
blog_ids = {} #blogs hashed by their ids, Buckets have {'post_id':,'post_link':}
if not isFileFresh(FILE_002):
    print "Loading Blogs and Last Posts from DB"
    sql = "SELECT Blog_pk_id, Blog_url FROM BLOGS WHERE Blog_active=1;"
    cursor = dbExec(sql)
    results = cursor.fetchall()

    for r in results:
        #Get the ID of the last post on each blog"
        sql = u"SELECT BP_pk_id,BP_link FROM BLOG_POSTS WHERE BP_fk_blog_id = %d ORDER BY BP_pk_id DESC LIMIT 1" % (r['Blog_pk_id']);
        cursor = dbExec(sql)
        last_post = cursor.fetchone()

        if last_post:
            blog_urls[r['Blog_url']] = {'post_id':last_post['BP_pk_id'],'post_link':last_post['BP_link']}
            blog_ids[r['Blog_pk_id']] = {'post_id':last_post['BP_pk_id'],'post_link':last_post['BP_link']}

    #serialize blog_urls and blog_ids
    print "Saving Blogs to File..."
    f = file(FILE_002,"wb")
    pickle.dump(blog_urls,f)
    pickle.dump(blog_ids,f)
    f.close()
    print "Saving completed."
else:
    #load blog_urls from serialized data
    print "Loading Blogs and Last Posts from File..."
    f = file(FILE_002,"rb")
    blog_urls = pickle.load(f)
    blog_ids = pickle.load(f)
    f.close()
    print "Loading completed."

#LOAD ALL BLOG_POSTS URL AND ITS HITS
post_hits = {} #posts hashed by url, Buckets have (post_id, post_hits, blog_id)
if not isFileFresh(FILE_003):
    print "Loading Posts from DB..."
    sql = "SELECT SQL_CACHE BP_link, BP_pk_id, BP_fk_blog_id, PH_hits "
    sql += "FROM BLOG_POSTS LEFT JOIN POST_HITS ON BP_pk_id = PH_fk_post_id;"
    cursor = dbExec(sql)
    results = cursor.fetchall()

    for r in results:
        #hits might be null, if the post has never been reached on our page
        hit_count = int(r['PH_hits']) if r['PH_hits'] is not None else 0
        post_hits[r['BP_link']] = {'post_id':int(r['BP_pk_id']),
                                   'post_hits':hit_count,
                                   'blog_id':r['BP_fk_blog_id']}

    #now get the blog_posts
    print "Saving Posts from DB on file"
    f = file(FILE_003,"wb")
    pickle.dump(post_hits,f)
    f.close()
    print "Saving completed."
else:
    print "Loading Posts from File..."
    f = file(FILE_003,"rb")
    post_hits = pickle.load(f)
    f.close()
    print "Loading completed."

# The stars of the game are:
# - urls {:{'ip':,'hits':}} //The urls and how many hits we got from the click trackr log
# - blog_urls {:{'post_id':,'post_link':}} //urls of blogs, holding each a tuple with last post info
# - post_hits {:{'post_id':
,'post_hits':
,'blog_id':]} //urls and hit info of all posts
# - urls_not_found {'':}
if not isFileFresh(FILE_004):
    total_not_found = 0
    distinct_not_found = 0
    total_converted = 0

    converting_blog_url_to_post_url = False

    print "Crunching data..."
    for url in urls:
	#if the current url is the home of a blog
        #we try to see if the blog has any hits.
	if blog_urls.has_key(url):
            url = blog_urls[url]['post_link']
            converting_blog_url_to_post_url = True

        #if you find a direct match add the hits right away
        if post_hits.has_key(url):
	    new_hits = 0
	    if urls.has_key(url) and urls[url].has_key('hits'):
                new_hits = urls[url]['hits']
                if converting_blog_url_to_post_url:
                    total_converted+=1
                    print "!",

            old_hits = 0
            if post_hits[url].has_key('post_hits'):
                old_hits = post_hits[url]['post_hits']

            total_hits = new_hits + old_hits

            #finally update post_hits arrays.
            post_hits[url]['post_hits'] = total_hits
            str_a = "(%(post_id)d):%(post_hits)d:" % post_hits[url]
            str_b = "%d+%d)" % (new_hits,old_hits)
            str_c = str_a + str_b
            print str_c,
        else:
            if urls_not_found.has_key(url):
                urls_not_found[url] += 1
            else:
                urls_not_found[url]=1
                distinct_not_found +=1
            total_not_found += 1
            print "-",

    print
    print "%d converted from blog to last post" % total_converted
    print "Didnt find total %d urls." % total_not_found
    print "Didn't find distinct %d urls." % distinct_not_found

    #serialize processed data in file 4
    print "Saving crunched data..."
    f = file(FILE_004,"wb")
    pickle.dump(post_hits,f)
    pickle.dump(urls_not_found,f)
    f.close()
    print "Data saved."
else:
    print "Loading Previously Crunched Data..."
    f = file(FILE_004,"rb")
    post_hits = pickle.load(f)
    urls_not_found = pickle.load(f)
    f.close
    print "Loading completed."

    #If we can't find it on the blog posts, we could try to
    #slim the URL of this url too http://servername.com/folder
    #and look up on the blog url

    #if nothing, we slim down to http://servername.com

    #if in any of these 2 cases we find a match then
    #we add a hit on the last post of this blog

    #the output of this file should be for now a file with SQL insert statements
print len(post_hits)

#Generate SQL output from post_hits array
# - post_hits {:{'post_id':
,'post_hits':
,'blog_id':]}
print "Writing SQL..."
f = file(FILE_SQL,"wb")
for url in post_hits:
    post_id = post_hits[url]['post_id']
    hits = post_hits[url]['post_hits']
    f.writelines("UPDATE POST_HITS SET PH_hits = %d WHERE PH_fk_post_id = %d;\n" % (hits,post_id))
f.close()
print "Finished Writing SQL"

#If we make it all the way till here, we write down the new LAST_TIMESTAMP
if POSSIBLE_LAST_TIMESTAMP is not None:
    f = file(FILE_TIMESTAMP,"wb")
    pickle.dump(POSSIBLE_LAST_TIMESTAMP,f)
    f.close()
    print "Wrote last timestamp."
else:
    print "Did not write LAST timestamp."

for u in urls_not_found:
    print u

                          

No fue hasta que programe en Python que me habia pillado que podia definir funciones dentro de funciones en PHP. Hoy tuve que arreglar un defecto en el home de wedoit4you.com del cual algunos bloggers se estaban aprovechando para permanecer en el home. Los articulos aparecen ordenados por fecha de publicacion, y algunos estaban publicando con fechas en el futuro, inclusive abusando y poniendo fechas a fin de mes.

So, en mi clase “BlogPost”, puse un metodo “getTimestamp()”, si la fecha interna en el objeto esta en el futuro, y no es uno de los blogs a los cuales les paso la gracia…
Penalizo la fecha del post, y le resto 24 horas a la ultima hora en que se actualizo el Blog.

Pero esto no me resuelve por completo el problema puesto que los posts se leen direct tv de una tabla, y el query ordena por timestamp, el resultado de mi funcion “getPosts” es un arreglo de Objetos, y lo ideal era tener esos objetos ordenados dependiendo de la funcion “getTimestamp()” en cada objeto “BlogPost”

La foto que ven arriba muestra como utilizo la funcion “usort” para ordenar (por referencia) el arreglo resultante. La funcion recibe una referencia al arreglo de objetos, y el nombre de una funcion a la cual hacer callback al momento de comparar. La funcion de callback debe recibir 2 objetos como parametro, y luego devolver un numero que represente si el primer parametro es menor o mayor que el segundo parametro. ( < que cero si es menor, cero si son iguales, > que cero si es mayor).

En mi caso llame esta funcion “cmp”, pero no la defini afuera de la clase nisiquiera, la defini ahi mismo dentro de la funcion. Si ven la logica de mi funcion el resultado es alrevez puesto que quiero ordenar de manera decrescente.

Espero que alguien haya leido esto si necesita ordenar arreglos.

En cuanto al scoping de las funciones internas, no se que sucederia si hubiese otra funcion llamada igual en el path, supongo que el interprete busca primero en el stack, y encontraria la funcion definida dentro de la funcion actual. Imagino que al terminar de ejecutarse esta funcion, podria no estar diponible para mas nadie. Se que en python puedes hacer Clases dentro de clases, funciones dentro de funciones. En el caso de Clases dentro de Clases, puedes hacer. Objeto.SubObjeto.metodo(), en el caso de funciones dentro de funciones, creo que no tiene sentido hacer algo como Objeto.function.subFuncion(), ya que las funciones necesitan parametros, pero seria cuestion de probar a ver, quizas funciona, dado que en python todo es un objeto. Alguien que me eche el cuento para PHP.

(el screenshot fue tomado de emacs en el terminal de OSX Tiger)

                          

I’ve read in parts of the web (and on the Martinelli’s Python Cookbok) that it’s better to do the Borg pattern over singletons, they say something alongs the lines of:

“who cares about identity, care about shared state”

Coming from the Java world, I just can’t understand that, why waste memory and cpu to address objects that share a state when you can have a single object in memory.

If you’re looking on how to implement singleton in a simple manner with Python, do the following my friend:

class MyClass:
  __INSTANCE__ = None

  def __init__(self):
    #do whatever you need on your constructor
    pass

  @staticmethod
  def getInstance():
    if MyClass.__INSTANCE__ is None:
      MyClass.__INSTANCE__ = MyClass()
    return MyClass.__INSTANCE__

#Then use it.
theOne = MyClass.getInstance()

Done deal, now start trolling on why this code sucks so we can fix it.