Quick N Dirty way to Map Commands to remote servers via ssh

You may be running several independent but similar servers at the same time and wasting time by executing commands in all of them one by one.

Wouldn’t it be nice to send a command to all of them at once? or to monitor all of them at once.

The following script can be used as a building block to more complex automation tasks for a small size set of servers. (If you’re managing over 50 servers, I’d probably consider looking a different way to arrange servers (map/reduce cluster), but if you’re doing something below that number this might suffice)

[code lang=”python”]
#!/usr/bin/python

#########################################################
# Author: Angel Leon (gubatron@gmail.com) – October 2009
#
# Invokes a command locally and invokes the same command
# in all machines under the specified username, servers
#
# Requirement: Have a public ssh_key for that user on all
# the other machines so you don’t have to authenticate
# on all the other machines.
#########################################################
import sys
import os

# set the username that has access to all the machines here
user=’safeuser’

# add all your server names here
servers=[‘server1.mydomain.com’,’server2.mydomain.com’,’server3.mydomain.com’]

if __name__ == "__main__":
if len(sys.argv) < 2:
print "Usage: ssh_map_command <cmd>"
sys.exit(0)

cmd= ‘ ‘.join(sys.argv[1:])

#Execute locally first
print cmd
os.system(cmd)

#Execute for all the servers in the list
for server in servers:
remote_cmd="ssh %s@%s %s" % (user,server,cmd)
print remote_cmd
os.system(remote_cmd)
print
[/code]

Save as ssh_map_command and chmod +x it.

Sample uses
Check the average load of all machines at once (then use output to mitigate high load issues)
[code lang=”shell”]$ ssh_map_command uptime[/code]

Send HUP signal to all your web servers (put it in an alias or other script… and that’s how you start building more complex scripts)
[code lang=”shell”]$ ssh_map_command ps aux | grep [l]ighttpd | kill -HUP `awk {‘print $2’}`[/code]

Check if processes are alive, check memory usage on processes across different machines, grep remote all logs at once, svn up on all machines, rsync from one to many, hey, you can even tail -f and grep all the logs at once, you can go nuts with this thing. Depends on what you need to do.

Requirements

Security Advisory
Make sure only the desired user has read/write/execute access to it and keep your private ssh keys safe (preferably only read and execute for the owner, and no permissions whatsoever to anybody else chmod 500 ssh_mod_map), if possible change them as often as possible, for it may become a big security whole if an attacker can manage to write code on this script, specially if you have cronjobs invoking it. Your attacker would only need to change code here to mess up all of your machines.

Disclaimer and Call for Knowledge
Please, if someone knows of a standard way to map commands to multiple servers, please let me know in the comment section, in my case I needed a solution and I wrote a quick and dirty python script and tried to secure it as best as I could, by no means I’m saying that this is the best solution to mapping commands, in fact I believe it might be the least efficient way, however it works good enough for my personal needs.

Delete All Direct Messages of your Twitter Account at once (or at least try!)

Since Twitter doesn’t provide with a “Delete All Direct Messages” functionality, Here’s a Python script that attempts to delete all the direct messages stored on your Twitter account.

Limitations
The only problem with it is that given the limitations of the Twitter REST API, I was forced to send a request per message to be deleted, and it seems that Twitter will only allow 100 requests per hour (per client).

So in theory you will be able to delete 100 an hour, although I have seen it delete over 400 messages Twitter gets all grumpy on me.

Usage
Just save the script as twitter_delete_direct_messages.py, open a terminal and run it:

[bash]python twitter_delete_direct_messages.py[/bash]

After that just follow the instructions on screen and enjoy as messages get wiped out.

If you find this useful, you can thank me by following me on Twitter.

Get The Script

Using a linear array as a bidimensional matrix

Often times I find the need to use a list or linear array as if it was a table.

Everytime I need to do so, I always end up coding functions to convert a (x,y) coordinate to the real index n in the array.

Let me illustrate, with an example. You have a string that defines the elements of a game board, and you want to work using (x,y) coordinates.

[python]
s="xxxxx@xx@xxx@@xx"
[/python]

If you were to look at it as a matrix (width=4), it’d be something like this
[python]
s="xxxx
x@xx
@xxx
@@xx"
[/python]

However, I can’t do
s[x,y], since it’s a linear array, it’s a string.

You need to convert from (x,y) to a number that represents an index in the array.

This is very simple:
[python]
width=4
def getNforXY(x,y):
return x + width*y
[/python]

What if you want to do it backwards. What if you need to know what’s the X and Y for a given index N in the string?

[python]
def getXYforN(n):
y = int(n/width)
x = n – width/y
return (x,y)
[/python]

Cheers

Python Script to Update WordPress in One Step

During the past week, I think I had to update all my wordpress instances twice, and it’s become really annoying doing this manually. I’ve written a python script which I’ll share with you.

How I keep my wordpress updated by hand
I tend to keep my wp-content folder outside of my wordpress installation for 2 reasons:

1. I don’t like to loose my themes, plugins and customizations
2. I like to keep all my customization changes under subversion

So, if I had my wordpress installation say at:
/home/user/public_html/blog

I’d keep my wp-content folder for that here:

/home/user/public_html/wp-content-for-blog

So when I upgrade my blog, I always remove the original wp-content folder that comes along wordpress, and I symlink my hard worked on wp-content folder that lives outside to the freshly unzipped wordpress folder.

[bash]
user@machine:~/public_html/blog$ ls -l

lrwxrwxr-x 1 user www 54 2008-11-26 09:29 wp-content -> /home/user/public_html/wp-content-for-blog

[/bash]

So what I endup doing all the time, is downloading the latest.zip to ~/public_html/, it will unzip under ~/public_html/wordpress, and then I’ll copy the current ~/public_html/blog/wp-config.php to ~/public_html/wordpress, then I’ll remove the default ~/public_html/wordpress/wp-content and symlink the outer wp-content with all my customizations, themes and plugins to it. Once done, I’ll make a backup of the old wordpress folder, and then I’ll rename wordpress folder to the name of the blog folder, and it’s all done.

It’s simple, but when you have to do it for 5 blogs, every week, it’s not fun anymore.

The Update Script

So here’s a script to do it in one step. If you’re not using my symlinked technique, this will do it for you, you only need to specify the full path to the folder where you want to keep your current wp-content folder outside the new installation before you apply the update, and the name of the folder where your current blog lives. The script below will have its configuration variables towards the beginning set so that they are in line with the example I’ve been talking about.

[python]
#!/usr/bin/python
#########################################################################################
#
# upgrade_wordpress.py – Script to automatically upgrade your wordpress installation.
#
# Requirements:
# – Python 2.4 or older
# – WordPress should already be installed
# – CURL (sudo apt-get install curl)
#
# Author: Angel (Gubatron) Leon
# LICENSE: See the GPL2 license.
# 2008
#########################################################################################
import os

#########################################################################################
#Config (relative to the folder where this script will be run from)
#########################################################################################

#The current folder where the blog lives
BLOG_FOLDER=’blog’

#
# The first time you run the script, it will try to make a copy of your
# current wp-content folder outside. Copy here the location of where
# the wp-content folder with your themes and plugins should exist.
#
# After it unzips, it will remove the default wp-content folder from
# the new installation, and it will symlink the external wp-content
# That way you don’t ever have to worry about loosing your customizations
# and plugins.
#
WP_CONTENT_OUTSIDE_COPY_FOLDER="/home/user/public_html/wp-content-for-blog"

#This is where a backup of your current blog will be
BLOG_FOLDER_BACKUP_FOLDER=BLOG_FOLDER+’.old’

#Where to download the wordpress latest.zip from
WORDPRESS_LATEST_ZIP_URL=’http://wordpress.org/latest.zip’

#### DO NOT MODIFY AFTER THESE LINES ####

def downloadWordpress(url=WORDPRESS_LATEST_ZIP_URL):
if os.path.exists(‘latest.zip’):
print "Removing old latest.zip"
os.remove(‘latest.zip’)

#Try to download with CURL
print "Attempting to download latest.zip from wordpress.org"
os.system(‘curl %s -o latest.zip’ % url)

if not os.path.exists(‘latest.zip’):
os.system(‘wget ‘ + url)

return os.path.exists(‘latest.zip’)

def dirExists(dirName):
return os.path.exists(dirName) and os.path.isdir(dirName)

def backupBlog(currentBlogFolder=BLOG_FOLDER,
wpContentOriginalFolder=WP_CONTENT_OUTSIDE_COPY_FOLDER,
backupFolder=BLOG_FOLDER_BACKUP_FOLDER):

#Remove any previous backups
if os.path.exists(backupFolder) and os.path.isdir(backupFolder):
print "Removing previous backup folder"
os.system(‘rm -fr ‘ + backupFolder)

#Copy the current blog folder into a backup folder just in case.
#We won’t do any database backups for now.
print "Creating new backup folder"
os.system(‘cp -r %s %s’ % (currentBlogFolder,backupFolder))

#Check for the copy of wp-content outside the blog, if it doesn’t exist
#we’ll make it for the first time.
if not dirExists(wpContentOriginalFolder):
print "Creating outside copy of wp-content"
os.system(‘cp -r %s %s’ % (os.path.join(currentBlogFolder,’wp-content’),
wpContentOriginalFolder))

#Copy the latest wp-config.php outside to the current folder
print "Copying your latest wp-config.php outside"
os.system(‘cp %s .’ % (os.path.join(currentBlogFolder,’wp-config.php’)))

backupFolderExists = dirExists(backupFolder)
wpContentFolderExists = dirExists(wpContentOriginalFolder)
configFileExists = os.path.exists(‘wp-config.php’)

return backupFolderExists and wpContentOriginalFolder and configFileExists

def upgradeBlog(currentBlogFolder=BLOG_FOLDER,
backupFolder=BLOG_FOLDER_BACKUP_FOLDER,
url=WORDPRESS_LATEST_ZIP_URL,
wpContentOriginalFolder=WP_CONTENT_OUTSIDE_COPY_FOLDER):

if not downloadWordpress(url):
print "Could not download latest.zip, aborting."
return False

if not backupBlog(currentBlogFolder,wpContentOriginalFolder,backupFolder):
print "Could not backup blog or wp-config.ph, aborting."
return False

if currentBlogFolder == ‘wordpress’:
print "The current blog folder cannot be ‘wordpress, aborting."
return False

#1. If a wordpress/ folder exists, wipe it.
if dirExists(‘wordpress’):
print "Removing old wordpress folder"
os.system(‘rm -fr wordpress’)

if dirExists(‘%s.delete’ % currentBlogFolder):
print "Removing old %s.delete folder" % currentBlogFolder
os.system(‘rm -fr %s.delete folder’ % currentBlogFolder)

#2. Unzip new copy
os.system(‘unzip latest.zip’)

if not dirExists(‘wordpress’):
print "Could not unzip the wordpress installation, aborting."
return False

#1. Copy wp-config.php into the new installation
os.system(‘cp wp-config.php wordpress/’)

#2. Remove the default wp-content folder
os.system(‘rm -fr wordpress/wp-content’)

#3. Symlink the original wp-content that lives outside
os.system(‘ln -s %s wordpress/wp-content’ % (wpContentOriginalFolder))

#4. Verify symlink was created
if not (os.path.exists(‘wordpress/wp-content’) and os.path.islink(‘wordpress/wp-content’)):
print "Could not create symlink to wp-content, aborting."
return False

#5. Move original folder to folder.delete, and make this wordpress folder the current folder.
os.system(‘mv %s %s.delete’ % (currentBlogFolder,currentBlogFolder))

if not dirExists(currentBlogFolder + ".delete"):
print "Could not rename current folder for later deletion, aborting."
return False

#6. Rename the new installation as the current blog
os.system(‘mv %s %s’ % (‘wordpress’,currentBlogFolder))

if dirExists(‘wordpress’):
print "ALERT: The wordpress folder still exists."
return False

if not dirExists(currentBlogFolder):
print "ALERT: The blog doesn’t exist, recover from the backup folder %s please" % (backupFolder)
return False

#7 Cleanup
os.system(‘rm -fr %s.delete’ % (currentBlogFolder))

return True

if __name__ == ‘__main__’:
upgradeBlog()
[/python]

Requirements

  • shell access to the machine where you have your wordpress installed
  • a python interpreter installed
  • curl (sudo apt-get install curl) to download the zip. If you don’t have it it’ll attempt to use wget
  • Installation

  • Right outside your wordpress installation folder, create a new file called upgrade_wordpress.py
  • Copy and paste the script inside that file
  • Edit the configuration variables to point to the name of your wordpress installation folder, and give it a full path to where you want to keep your wp-content folder (including the name of the folder, so if you want to name it the same way, you could do for example /home/user/wp-content and it’ll be saved right under your home)
  • Usage:
    [bash]python upgrade_wordpress.py[/bash]

    The script is very fault proof, it will always try to abort in case something is not going the way it’s expected. At the end of the day it’ll also leave a backup copy of your current blog in case something goes bad, you can always recover.

    Script to automatically detect and ban malicious IPs that try to brute force SSH accounts

    We’ve noticed that most of our servers have been under heavy attack from random IP addresses to break via SSH.

    With the help of the last post on how to ban an IP, and the following python script, you’ll be able to have a cronjob that runs once or twice a day and automagically bans all the offending ips from ever trying to brute force their way in ever again.

    touch and make executable a file called “detect_ssh_hostiles”

    touch detect_ssh_hostiles
    chmod +x detect_ssh_hostiles
    

    Then copy the following code inside:

    # Usage:
    # python detect_ssh_hostiles [auth.log file path]
    #
    # Requirement: There should be "ban_ip" and "unban_ip" command availability on the path
    #
    # Note: you gotta have read permissions on the auth.log file and sudo
    #       permissions for the script to ban the ips.
    
    #If an IP meets this number of failed login attemmpts it will be banned
    BAN_THRESHOLD = 7
    SUSPECTS = {}
    
    #Put here IP addresses you trust, could be making genuine login errors
    SAFE_IPS = ['81.73.111.49','101.73.111.160','72.31.171.235','72.36.23.234','82.36.180.210','202.132.82.16']
    
    import os
    import sys
    import re
    
    BANNED = {}
    def loadBanned():
      '''
      This function will load all the banned IPS into the BANNED Dict.
      It will also count how many times (by mistake) the same IP has
      been banned, and it will unban it, so that it will appear only once.
      '''
      global BANNED
      command = 'sudo iptables --list --numeric'
      try:
        p = os.popen(command,'rb')
      except Exception,e:
        print e
        sys.exit(1)
    
      line = '-'
    
      while line != '':
        line = p.readline().strip()
    
        if line.startswith("DROP"):
          parts = line.split()
          ip = parts[3]
    
          #add hit or register banned ip
          if BANNED.has_key(ip):
            BANNED[ip]+=1
          else:
            BANNED[ip]=1
    
      #Make sure banned IPs are banned only once
      for ip in BANNED:
        if BANNED[ip] > 1:
          print "IP %s has been banned %d times" % (ip, BANNED[ip])
          n=BANNED[ip]-1
          while n > 0:
            os.system("unban_ip %s" % ip)
            print ("unban_ip %s" % ip)
            n=n-1
    
      p.close()
    
    # ---- here we go ----
    loadBanned()
    
    #read auth log
    logfile = '/var/log/auth.log'
    
    if len(sys.argv)==2:
      logfile = sys.argv[1]
    
    command = 'grep "Failed password for " %s' % logfile
    #print command
    
    try:
      p = os.popen(command,'rb')
    except Exception,e:
      print e
      sys.exit(1)
    
    line = "123"
    
    while line != '':
      line = p.readline()
    
      #Sample line:
      # May 25 03:29:49 main sshd[6933]: Failed password for root from 202.118.236.132 port 54863 ssh2
      pattern = "(.*)(froms)(d+.d+.d+.d+)(.*)"
      matchObject = re.match(pattern, line)
    
      suspect = None
      if matchObject is not None:
        suspect = matchObject.groups()[2]
    
        #skip safe IPs
        if suspect in SAFE_IPS:
          continue
    
        if SUSPECTS.has_key(suspect):
          #add a hit
          SUSPECTS[suspect] += 1
        else:
          #add first hit
          SUSPECTS[suspect] = 1
    
    p.close() #close the pipe
    
    print "=="*30
    
    import time
    t = time.localtime()
    #(2008, 6, 6, 9, 35, 21, 4, 158, 1)
    
    timestr = "%d-%d-%d@%d:%d:%d" % (t[0],t[1],t[2],t[3],t[4],t[5])
    print timestr
    print "--"*30
    if len(SUSPECTS) > 0:
      for suspect in SUSPECTS:
        if SUSPECTS[suspect] >= BAN_THRESHOLD and not BANNED.has_key(suspect):
          print "Banning %s with %d attempts" % (suspect,SUSPECTS[suspect])
          BANNED[suspect]=1
          os.system("ban_ip %s" % suspect)
        elif BANNED.has_key(suspect):
          print "Ip %s has already been banned" % (suspect)
        else:
          print "Suspect candidate? %s with %d attempts" % (suspect,SUSPECTS[suspect])
    else:
      print "Found no suspects to ban"
    
    print "=="*30

    Then add this as a cronjob of your root user, and it will automatically ban all those IPs that have tried to break in. See the script for configuration. You can always make some IPs immune to banning by adding them on the SAFE_IPS list.

    [Python] PyCon 2008 attendance jumped 70%!

    Chicago, IL (PRWEB) March 27, 2008 — PyCon 2008, the annual community conference for Python developers, business leaders, and fans showcased the tremendous increase in growth among the community last week in Chicago. Attendance jumped 70 percent from 2007.

    Over 1,000 people gathered at the Crowne Plaza Chicago O’Hare Hotel for PyCon 2008. The influx of proposals was so great this year that organizers could only accept half the proposals received, simply due to lack of space. Demand for tutorials, in particular, rose so high that organizers added an extra timeslot full of new sessions. Twenty-five of the 28 tutorials were sold out and there were 22 open source development sprint projects available for attendees to participate in, up from 13 in 2007. More than 270 people took part in the development sprints, which is more than attended the entire first PyCon in 2003.

    The best part about PyCon is the people you get to meet
    “As jam-packed as the week’s events were, a number of attendees and speakers remarked to me that they felt the conference this year was the best yet,” said PyCon 2008 chair David Goodger. “It’s great to see months of work come together and have people really enjoy it.”

    PyCon 2008 is organized and run entirely by volunteers. A true grassroots movement, it is a community conference put on by the community for the community. Many attendees enjoy the numerous sessions, but more importantly, they believe the real value of the conference is the community all being there together.

    “The best part about PyCon is the people you get to meet,” said Chris McAvoy, founder and president of the Chicago Python Users Group, which hosted PyCon 2008. “That, coupled with more than 1,000 attendees, compared to the 600 last year, says to me that Python is really building a market here in Chicago and elsewhere. It’s an exciting time to be a Python developer.”

    PyCon 2009 will also be held in Chicago. For more information about PyCon, please visit http://us.pycon.org.

    About PyCon

    Presented by the Python Software Foundation, the world’s largest Python conference brings together a diverse group of developers, enthusiasts, and organizations to explore new challenges, launch new businesses and forge new connections within the Python community. PyCon provides attendees with the opportunity to delve into the dynamic programming language employed by well-known companies such as Google, Cisco, and the New York Times. PyCon helps people learn new tools and techniques, showcase projects, and meet other Python fans.

    Source: PRWeb.com

    [Python] ip2num / num2ip – Store an IP string as a 4 byte int.

    This is probably everywhere, maybe python also comes with it, but I wanted to have my own implementation, and I’ll leave it here for future reference.

    Basically, sometimes you don’t want to store IPs in Strings cause they take too much space, instead you want to be a good programmer and store them as 4 bytes (IPv4 that is).

    So here’s a couple of functions in python to illustrate the conversion process between string to 4 byte integer, or viceversa:

    def ip2num(ipString):
        if ipString is None:
            raise Exception("Invalid IP")
    
        try:
           octets = [octet.strip() for octet in ipString.split('.')]
        except Exception,e:
            raise e
    
        num = (int(octets[0])<<24) + (int(octets[1])<<16) + (int(octets[2])<<8) + int(octets[3])
        return num
    
    def num2ip(numericIp):
        if numericIp is None or type(numericIp) != int:
            raise Exception("Invalid numeric IP. Must be an integer")
        return str(numericIp >> 24) + '.' + str((numericIp >> 16) & 255) + '.' + str((numericIp >> 8) & 255) + '.' + str(numericIp & 255)