You may be running several independent but similar servers at the same time and wasting time by executing commands in all of them one by one.

Wouldn’t it be nice to send a command to all of them at once? or to monitor all of them at once.

The following script can be used as a building block to more complex automation tasks for a small size set of servers. (If you’re managing over 50 servers, I’d probably consider looking a different way to arrange servers (map/reduce cluster), but if you’re doing something below that number this might suffice)

#!/usr/bin/python                                                                                                                                                                                                                                                      

#########################################################
# Author: Angel Leon (gubatron@gmail.com) - October 2009
#
# Invokes a command locally and invokes the same command
# in all machines under the specified username, servers
#
# Requirement: Have a public ssh_key for that user on all
# the other machines so you don't have to authenticate
# on all the other machines.
#########################################################
import sys
import os

# set the username that has access to all the machines here
user='safeuser'

# add all your server names here
servers=['server1.mydomain.com','server2.mydomain.com','server3.mydomain.com']

if __name__ == "__main__":
  if len(sys.argv) < 2:
    print "Usage: ssh_map_command "
    sys.exit(0)

  cmd= ' '.join(sys.argv[1:])

  #Execute locally first
  print cmd
  os.system(cmd)

  #Execute for all the servers in the list
  for server in servers:
    remote_cmd="ssh %s@%s %s" % (user,server,cmd)
    print remote_cmd
    os.system(remote_cmd)
    print

Save as ssh_map_command and chmod +x it.

Sample uses
Check the average load of all machines at once (then use output to mitigate high load issues)

$ ssh_map_command uptime

Send HUP signal to all your web servers (put it in an alias or other script… and that’s how you start building more complex scripts)

$ ssh_map_command ps aux | grep [l]ighttpd | kill -HUP `awk {'print $2'}`

Check if processes are alive, check memory usage on processes across different machines, grep remote all logs at once, svn up on all machines, rsync from one to many, hey, you can even tail -f and grep all the logs at once, you can go nuts with this thing. Depends on what you need to do.

Requirements

• For convenience you should probably set up ssh public/private keys between the main machine and the other machines (or they can all trust each other and you can execute on any) so that you can execute commands on all the machines without having to enter passwords.
• Put the script on your $PATH.
• python

Security Advisory
Make sure only the desired user has read/write/execute access to it and keep your private ssh keys safe (preferably only read and execute for the owner, and no permissions whatsoever to anybody else chmod 500 ssh_mod_map), if possible change them as often as possible, for it may become a big security whole if an attacker can manage to write code on this script, specially if you have cronjobs invoking it. Your attacker would only need to change code here to mess up all of your machines.

Disclaimer and Call for Knowledge
Please, if someone knows of a standard way to map commands to multiple servers, please let me know in the comment section, in my case I needed a solution and I wrote a quick and dirty python script and tried to secure it as best as I could, by no means I’m saying that this is the best solution to mapping commands, in fact I believe it might be the least efficient way, however it works good enough for my personal needs.

Just Type…
sudo dpkg-reconfigure tzdata

…and follow the instructions on screen.

The process should look something like the following:

Select your Region

Select a city on your time zone

You’re done.

Tip
You can always check the status of your configuration using
sudo debconf-show tzdata

You could for example map that command via ssh to several machines and grep for “*”, that way you could easily spot servers with wrong timezones very quickly.

Type the following on your cmd line (or make into an alias)

cat ~/.bash_history | sort | uniq -c | sort -r | head

In my case they are (for this week)

ls
fg
svnSync (script I created)
stats_fetch; stats_display (other scripts)
cd
crontab -e
ps aux | grep
ssh_map_command (another script)
python
emacs -nw

I often want to see how much bandwidth is consumed by my network interfaces and how this is happening. There’s plenty of tools available in the linux world to monitor your network activity.

If you’re a Ubuntu or Debian user you can try the ones I use by installing the following packages.

sudo apt-get install iptraf ethstatus dstat iftop ifstat nload bwm-ng

Then try each one of those on your command line and have fun exploring what they can do.

If they’re not on the path of your user account, then try them using sudo.
eg. sudo iftop

If this short list of the tools I use does not satisfy you, I suggest you read this more comprehensive list of monitoring tools

After I received comments from ispak on Flickr I made a few fixes.

ispak pointed out that it was a bad idea reading one byte at the time, also I had a gay ass try/catch that didn’t catch any exception :p

So now I read 16 byte chunks, and I also take care of the file ending. The previous version used to print the file ending with a bunch of null bytes. Now it stops reading at the end, and formats the output accordingly.

Here’s the new source:

//HexViewer.java
import java.io.*;

public final class HexViewer {
    public final static void printFile(String filePath) {
        try {
            File f = new File(filePath);
            BufferedInputStream bis =
                new BufferedInputStream(new FileInputStream(f));

            byte[] chunk = null;
            int readStatus = 0;
            while (true) {
                chunk = new byte[16];
                readStatus = bis.read(chunk, 0, 16);
                char[] line = new char[16];

                if (readStatus == -1)
                    break;

                for (byte i=0; i < readStatus; i++) {
                    int readByte = (chunk[i] < 0) ? (-1 * (int) chunk[i]) : chunk[i];
                    String paddingZero = (readByte < 16) ? "0" : "";
                    System.out.print(paddingZero + Integer.toHexString(readByte).toUpperCase() + " ");
                    line[i] = (readByte >= 33 && readByte <= 126) ? (char) readByte : '.';
                }

                //We add some padding to print the text line right below the one above.
                String padding = new String();
                if (readStatus < 16) {
                    for (byte i=0; i < 16-readStatus; i++) {
                        padding += "   ";
                    }
                }

                System.out.println(padding + new String(line));
            }
        } catch (Exception e1) { e1.printStackTrace(); }
    }

    public final static void main(String[] args) {
        if (args.length == 0)
            return;

        printFile(args[0]);
    }
}

And see how it now handles file endings when the file size is not a multiple of 16 :p

I recently requested an upgrade on one of our dedicated server’s uplink speed, we only had a 10Mbps Uplink, we requested an upgrade to 100Mbps to serve a lot more.

How do you verify the upgrade has been done correctly?

As root, issue the following comand:

# mii-tool
eth0: negotiated 10baseT-FD, link ok

If it doesn’t work (for debian or ubuntu), make sure you have installed the net-tools package (The NET-3 networking toolkit)