Often times I find the need to use a list or linear array as if it was a table.

Everytime I need to do so, I always end up coding functions to convert a (x,y) coordinate to the real index n in the array.

Let me illustrate, with an example. You have a string that defines the elements of a game board, and you want to work using (x,y) coordinates.

s="xxxxx@xx@xxx@@xx"

If you were to look at it as a matrix (width=4), it’d be something like this

s="xxxx
   x@xx
   @xxx
   @@xx"

However, I can’t do
s[x,y], since it’s a linear array, it’s a string.

You need to convert from (x,y) to a number that represents an index in the array.

This is very simple:

width=4
def getNforXY(x,y):
  return x + width*y

What if you want to do it backwards. What if you need to know what’s the X and Y for a given index N in the string?

def getXYforN(n):
  y = int(n/width)
  x = n - width/y
  return (x,y)

Cheers

During the past week, I think I had to update all my wordpress instances twice, and it’s become really annoying doing this manually. I’ve written a python script which I’ll share with you.

How I keep my wordpress updated by hand
I tend to keep my wp-content folder outside of my wordpress installation for 2 reasons:

1. I don’t like to loose my themes, plugins and customizations
2. I like to keep all my customization changes under subversion

So, if I had my wordpress installation say at:
/home/user/public_html/blog

I’d keep my wp-content folder for that here:

/home/user/public_html/wp-content-for-blog

So when I upgrade my blog, I always remove the original wp-content folder that comes along wordpress, and I symlink my hard worked on wp-content folder that lives outside to the freshly unzipped wordpress folder.

user@machine:~/public_html/blog$ ls -l
...
lrwxrwxr-x 1 user www    54 2008-11-26 09:29 wp-content -> /home/user/public_html/wp-content-for-blog
...

So what I endup doing all the time, is downloading the latest.zip to ~/public_html/, it will unzip under ~/public_html/wordpress, and then I’ll copy the current ~/public_html/blog/wp-config.php to ~/public_html/wordpress, then I’ll remove the default ~/public_html/wordpress/wp-content and symlink the outer wp-content with all my customizations, themes and plugins to it. Once done, I’ll make a backup of the old wordpress folder, and then I’ll rename wordpress folder to the name of the blog folder, and it’s all done.

It’s simple, but when you have to do it for 5 blogs, every week, it’s not fun anymore.

The Update Script

So here’s a script to do it in one step. If you’re not using my symlinked technique, this will do it for you, you only need to specify the full path to the folder where you want to keep your current wp-content folder outside the new installation before you apply the update, and the name of the folder where your current blog lives. The script below will have its configuration variables towards the beginning set so that they are in line with the example I’ve been talking about.

#!/usr/bin/python
#########################################################################################
#
# upgrade_wordpress.py - Script to automatically upgrade your wordpress installation.
#
# Requirements:
#   - Python 2.4 or older
#   - Wordpress should already be installed
#   - CURL (sudo apt-get install curl)
#
# Author: Angel (Gubatron) Leon
# LICENSE: See the GPL2 license.
# 2008
#########################################################################################
import os

#########################################################################################
#Config (relative to the folder where this script will be run from)
#########################################################################################

#The current folder where the blog lives
BLOG_FOLDER='blog'

#
# The first time you run the script, it will try to make a copy of your
# current wp-content folder outside. Copy here the location of where
# the wp-content folder with your themes and plugins should exist.
#
# After it unzips, it will remove the default wp-content folder from
# the new installation, and it will symlink the external wp-content
# That way you don't ever have to worry about loosing your customizations
# and plugins.
#
WP_CONTENT_OUTSIDE_COPY_FOLDER="/home/user/public_html/wp-content-for-blog"

#This is where a backup of your current blog will be
BLOG_FOLDER_BACKUP_FOLDER=BLOG_FOLDER+'.old'

#Where to download the wordpress latest.zip from
WORDPRESS_LATEST_ZIP_URL='http://wordpress.org/latest.zip'

#### DO NOT MODIFY AFTER THESE LINES ####

def downloadWordpress(url=WORDPRESS_LATEST_ZIP_URL):
    if os.path.exists('latest.zip'):
        print "Removing old latest.zip"
        os.remove('latest.zip')

    #Try to download with CURL
    print "Attempting to download latest.zip from wordpress.org"
    os.system('curl %s -o latest.zip' % url)

    if not os.path.exists('latest.zip'):
        os.system('wget ' + url)

    return os.path.exists('latest.zip')

def dirExists(dirName):
    return os.path.exists(dirName) and os.path.isdir(dirName)

def backupBlog(currentBlogFolder=BLOG_FOLDER,
               wpContentOriginalFolder=WP_CONTENT_OUTSIDE_COPY_FOLDER,
               backupFolder=BLOG_FOLDER_BACKUP_FOLDER):

    #Remove any previous backups
    if os.path.exists(backupFolder) and os.path.isdir(backupFolder):
        print "Removing previous backup folder"
        os.system('rm -fr ' + backupFolder)

    #Copy the current blog folder into a backup folder just in case.
    #We won't do any database backups for now.
    print "Creating new backup folder"
    os.system('cp -r %s %s' % (currentBlogFolder,backupFolder))

    #Check for the copy of wp-content outside the blog, if it doesn't exist
    #we'll make it for the first time.
    if not dirExists(wpContentOriginalFolder):
        print "Creating outside copy of wp-content"
        os.system('cp -r %s %s' % (os.path.join(currentBlogFolder,'wp-content'),
                                   wpContentOriginalFolder))

    #Copy the latest wp-config.php outside to the current folder
    print "Copying your latest wp-config.php outside"
    os.system('cp %s .' % (os.path.join(currentBlogFolder,'wp-config.php')))

    backupFolderExists = dirExists(backupFolder)
    wpContentFolderExists = dirExists(wpContentOriginalFolder)
    configFileExists = os.path.exists('wp-config.php')

    return backupFolderExists and wpContentOriginalFolder and configFileExists

def upgradeBlog(currentBlogFolder=BLOG_FOLDER,
                backupFolder=BLOG_FOLDER_BACKUP_FOLDER,
                url=WORDPRESS_LATEST_ZIP_URL,
                wpContentOriginalFolder=WP_CONTENT_OUTSIDE_COPY_FOLDER):

    if not downloadWordpress(url):
        print "Could not download latest.zip, aborting."
        return False

    if not backupBlog(currentBlogFolder,wpContentOriginalFolder,backupFolder):
        print "Could not backup blog or wp-config.ph, aborting."
        return False

    if currentBlogFolder == 'wordpress':
        print "The current blog folder cannot be 'wordpress, aborting."
        return False

    #1. If a wordpress/ folder exists, wipe it.
    if dirExists('wordpress'):
        print "Removing old wordpress folder"
        os.system('rm -fr wordpress')

    if dirExists('%s.delete' % currentBlogFolder):
        print "Removing old %s.delete folder" % currentBlogFolder
        os.system('rm -fr %s.delete folder' % currentBlogFolder)

    #2. Unzip new copy
    os.system('unzip latest.zip')

    if not dirExists('wordpress'):
        print "Could not unzip the wordpress installation, aborting."
        return False

    #1. Copy wp-config.php into the new installation
    os.system('cp wp-config.php wordpress/')

    #2. Remove the default wp-content folder
    os.system('rm -fr wordpress/wp-content')

    #3. Symlink the original wp-content that lives outside
    os.system('ln -s %s wordpress/wp-content' % (wpContentOriginalFolder))

    #4. Verify symlink was created
    if not (os.path.exists('wordpress/wp-content') and os.path.islink('wordpress/wp-content')):
        print "Could not create symlink to wp-content, aborting."
        return False

    #5. Move original folder to folder.delete, and make this wordpress folder the current folder.
    os.system('mv %s %s.delete' % (currentBlogFolder,currentBlogFolder))

    if not dirExists(currentBlogFolder + ".delete"):
        print "Could not rename current folder for later deletion, aborting."
        return False

    #6. Rename the new installation as the current blog
    os.system('mv %s %s' % ('wordpress',currentBlogFolder))

    if dirExists('wordpress'):
        print "ALERT: The wordpress folder still exists."
        return False

    if not dirExists(currentBlogFolder):
        print "ALERT: The blog doesn't exist, recover from the backup folder %s please" % (backupFolder)
        return False

    #7 Cleanup
    os.system('rm -fr %s.delete' % (currentBlogFolder))

    return True

if __name__ == '__main__':
    upgradeBlog()

Requirements

Installation

Usage:

python upgrade_wordpress.py

The script is very fault proof, it will always try to abort in case something is not going the way it’s expected. At the end of the day it’ll also leave a backup copy of your current blog in case something goes bad, you can always recover.

We’ve noticed that most of our servers have been under heavy attack from random IP addresses to break via SSH.

With the help of the last post on how to ban an IP, and the following python script, you’ll be able to have a cronjob that runs once or twice a day and automagically bans all the offending ips from ever trying to brute force their way in ever again.

touch and make executable a file called “detect_ssh_hostiles”

touch detect_ssh_hostiles
chmod +x detect_ssh_hostiles

Then copy the following code inside:

# Usage:
# python detect_ssh_hostiles [auth.log file path]
#
# Requirement: There should be "ban_ip" and "unban_ip" command availability on the path
#
# Note: you gotta have read permissions on the auth.log file and sudo
#       permissions for the script to ban the ips.

#If an IP meets this number of failed login attemmpts it will be banned
BAN_THRESHOLD = 7
SUSPECTS = {}

#Put here IP addresses you trust, could be making genuine login errors
SAFE_IPS = ['81.73.111.49','101.73.111.160','72.31.171.235','72.36.23.234','82.36.180.210','202.132.82.16']

import os
import sys
import re

BANNED = {}
def loadBanned():
  '''
  This function will load all the banned IPS into the BANNED Dict.
  It will also count how many times (by mistake) the same IP has
  been banned, and it will unban it, so that it will appear only once.
  '''
  global BANNED
  command = 'sudo iptables --list --numeric'
  try:
    p = os.popen(command,'rb')
  except Exception,e:
    print e
    sys.exit(1)

  line = '-'

  while line != '':
    line = p.readline().strip()

    if line.startswith("DROP"):
      parts = line.split()
      ip = parts[3]

      #add hit or register banned ip
      if BANNED.has_key(ip):
        BANNED[ip]+=1
      else:
        BANNED[ip]=1

  #Make sure banned IPs are banned only once
  for ip in BANNED:
    if BANNED[ip] > 1:
      print "IP %s has been banned %d times" % (ip, BANNED[ip])
      n=BANNED[ip]-1
      while n > 0:
        os.system("unban_ip %s" % ip)
        print ("unban_ip %s" % ip)
        n=n-1

  p.close()

# ---- here we go ----
loadBanned()

#read auth log
logfile = '/var/log/auth.log'

if len(sys.argv)==2:
  logfile = sys.argv[1]

command = 'grep "Failed password for " %s' % logfile
#print command

try:
  p = os.popen(command,'rb')
except Exception,e:
  print e
  sys.exit(1)

line = "123"

while line != '':
  line = p.readline()

  #Sample line:
  # May 25 03:29:49 main sshd[6933]: Failed password for root from 202.118.236.132 port 54863 ssh2
  pattern = "(.*)(from\s)(\d+\.\d+\.\d+\.\d+)(.*)"
  matchObject = re.match(pattern, line)

  suspect = None
  if matchObject is not None:
    suspect = matchObject.groups()[2]

    #skip safe IPs
    if suspect in SAFE_IPS:
      continue

    if SUSPECTS.has_key(suspect):
      #add a hit
      SUSPECTS[suspect] += 1
    else:
      #add first hit
      SUSPECTS[suspect] = 1

p.close() #close the pipe

print "=="*30

import time
t = time.localtime()
#(2008, 6, 6, 9, 35, 21, 4, 158, 1)

timestr = "%d-%d-%d@%d:%d:%d" % (t[0],t[1],t[2],t[3],t[4],t[5])
print timestr
print "--"*30
if len(SUSPECTS) > 0:
  for suspect in SUSPECTS:
    if SUSPECTS[suspect] >= BAN_THRESHOLD and not BANNED.has_key(suspect):
      print "Banning %s with %d attempts" % (suspect,SUSPECTS[suspect])
      BANNED[suspect]=1
      os.system("ban_ip %s" % suspect)
    elif BANNED.has_key(suspect):
      print "Ip %s has already been banned" % (suspect)
    else:
      print "Suspect candidate? %s with %d attempts" % (suspect,SUSPECTS[suspect])
else:
  print "Found no suspects to ban"

print "=="*30

Then add this as a cronjob of your root user, and it will automatically ban all those IPs that have tried to break in. See the script for configuration. You can always make some IPs immune to banning by adding them on the SAFE_IPS list.

Chicago, IL (PRWEB) March 27, 2008 — PyCon 2008, the annual community conference for Python developers, business leaders, and fans showcased the tremendous increase in growth among the community last week in Chicago. Attendance jumped 70 percent from 2007.

Over 1,000 people gathered at the Crowne Plaza Chicago O’Hare Hotel for PyCon 2008. The influx of proposals was so great this year that organizers could only accept half the proposals received, simply due to lack of space. Demand for tutorials, in particular, rose so high that organizers added an extra timeslot full of new sessions. Twenty-five of the 28 tutorials were sold out and there were 22 open source development sprint projects available for attendees to participate in, up from 13 in 2007. More than 270 people took part in the development sprints, which is more than attended the entire first PyCon in 2003.

The best part about PyCon is the people you get to meet
“As jam-packed as the week’s events were, a number of attendees and speakers remarked to me that they felt the conference this year was the best yet,” said PyCon 2008 chair David Goodger. “It’s great to see months of work come together and have people really enjoy it.”

PyCon 2008 is organized and run entirely by volunteers. A true grassroots movement, it is a community conference put on by the community for the community. Many attendees enjoy the numerous sessions, but more importantly, they believe the real value of the conference is the community all being there together.

“The best part about PyCon is the people you get to meet,” said Chris McAvoy, founder and president of the Chicago Python Users Group, which hosted PyCon 2008. “That, coupled with more than 1,000 attendees, compared to the 600 last year, says to me that Python is really building a market here in Chicago and elsewhere. It’s an exciting time to be a Python developer.”

PyCon 2009 will also be held in Chicago. For more information about PyCon, please visit http://us.pycon.org.

About PyCon

Presented by the Python Software Foundation, the world’s largest Python conference brings together a diverse group of developers, enthusiasts, and organizations to explore new challenges, launch new businesses and forge new connections within the Python community. PyCon provides attendees with the opportunity to delve into the dynamic programming language employed by well-known companies such as Google, Cisco, and the New York Times. PyCon helps people learn new tools and techniques, showcase projects, and meet other Python fans.

Source: PRWeb.com

This is probably everywhere, maybe python also comes with it, but I wanted to have my own implementation, and I’ll leave it here for future reference.

Basically, sometimes you don’t want to store IPs in Strings cause they take too much space, instead you want to be a good programmer and store them as 4 bytes (IPv4 that is).

So here’s a couple of functions in python to illustrate the conversion process between string to 4 byte integer, or viceversa:

def ip2num(ipString):
    if ipString is None:
        raise Exception("Invalid IP")

    try:
       octets = [octet.strip() for octet in ipString.split('.')]
    except Exception,e:
        raise e

    num = (int(octets[0])<<24) + (int(octets[1])<<16) + (int(octets[2])<<8) + int(octets[3])
    return num

def num2ip(numericIp):
    if numericIp is None or type(numericIp) != int:
        raise Exception("Invalid numeric IP. Must be an integer")
    return str(numericIp >> 24) + '.' + str((numericIp >> 16) & 255) + '.' + str((numericIp >> 8) & 255) + '.' + str(numericIp & 255)

This is one of those things I tend to forget


>>> PyQt4.Qt.qVersion()
'4.3.3'