ubuntu packages for a kick ass web server

Copy and paste the following list on a file, say “packages.txt”. To install all just do:

sudo apt-get install $(cat packages.txt)

accountsservice
acpid
adduser
ant
ant-optional
apache2-utils
apparmor
apport
apport-symptoms
apt
apt-transport-https
apt-utils
apt-xapian-index
aptitude
at
base-files
base-passwd
bash
bash-completion
bc
bind9-host
bsdmainutils
bsdutils
busybox-initramfs
busybox-static
byobu
bzip2
ca-certificates
ca-certificates-java
cloud-init
cloud-initramfs-growroot
cloud-initramfs-rescuevol
cloud-utils
command-not-found
command-not-found-data
console-setup
consolekit
coreutils
cpio
cpp
cpp-4.6
crda
cron
cryptsetup-bin
curl
dash
dbus
dbus-x11
dconf-gsettings-backend
dconf-service
debconf
debconf-i18n
debianutils
diffutils
dmidecode
dmsetup
dnsutils
dosfstools
dpkg
dstat
e2fslibs
e2fsprogs
ed
eject
emacs
emacs23
emacs23-bin-common
emacs23-common
emacsen-common
euca2ools
file
findutils
fontconfig
fontconfig-config
fonts-ubuntu-font-family-console
friendly-recovery
ftp
fuse
gamin
gcc-4.6-base
gconf-service
gconf-service-backend
gconf2
gconf2-common
geoip-database
gettext-base
gir1.2-glib-2.0
gir1.2-gudev-1.0
gnupg
gpgv
grep
groff-base
grub-common
grub-gfxpayload-lists
grub-legacy-ec2
grub-pc
grub-pc-bin
grub2-common
gvfs
gvfs-common
gvfs-daemons
gvfs-libs
gzip
hdparm
hicolor-icon-theme
hostname
htop
icedtea-6-jre-cacao
icedtea-6-jre-jamvm
icedtea-7-jre-cacao
icedtea-7-jre-jamvm
icedtea-netx
icedtea-netx-common
ifupdown
info
initramfs-tools
initramfs-tools-bin
initscripts
insserv
install-info
iproute
iptables
iputils-ping
iputils-tracepath
irqbalance
isc-dhcp-client
isc-dhcp-common
iso-codes
iw
java-common
kbd
keyboard-configuration
klibc-utils
krb5-locales
landscape-client
landscape-common
language-selector-common
laptop-detect
less
libaccountsservice0
libacl1
libapr1
libaprutil1
libapt-inst1.4
libapt-pkg4.12
libasn1-8-heimdal
libasound2
libasyncns0
libatasmart4
libatk-wrapper-java
libatk-wrapper-java-jni
libatk1.0-0
libatk1.0-data
libattr1
libavahi-client3
libavahi-common-data
libavahi-common3
libavahi-glib1
libbind9-80
libblkid1
libbonobo2-0
libbonobo2-common
libboost-iostreams1.46.1
libbsd0
libbz2-1.0
libc-bin
libc6
libcairo-gobject2
libcairo2
libcanberra0
libcap-ng0
libck-connector0
libclass-accessor-perl
libclass-isa-perl
libcomerr2
libcroco3
libcryptsetup4
libcups2
libcurl3
libcurl3-gnutls
libcwidget3
libdatrie1
libdb5.1
libdbus-1-3
libdbus-glib-1-2
libdconf0
libdevmapper-event1.02.1
libdevmapper1.02.1
libdns81
libdrm-intel1
libdrm-nouveau1a
libdrm-radeon1
libdrm2
libedit2
libelf1
libept1.4.12
libevent-2.0-5
libexpat1
libffi6
libflac8
libfontconfig1
libfontenc1
libfreetype6
libfribidi0
libfuse2
libgamin0
libgc1c2
libgcc1
libgconf-2-4
libgconf2-4
libgcrypt11
libgd2-noxpm
libgdbm3
libgdk-pixbuf2.0-0
libgdk-pixbuf2.0-common
libgdu0
libgeoip1
libgif4
libgirepository-1.0-1
libgl1-mesa-dri
libgl1-mesa-glx
libglapi-mesa
libglib2.0-0
libgmp10
libgnome-keyring-common
libgnome-keyring0
libgnome2-0
libgnome2-common
libgnomevfs2-0
libgnomevfs2-common
libgnutls26
libgpg-error0
libgpm2
libgssapi-krb5-2
libgssapi3-heimdal
libgtk-3-0
libgtk-3-bin
libgtk-3-common
libgtk2.0-0
libgtk2.0-bin
libgtk2.0-common
libgudev-1.0-0
libhcrypto4-heimdal
libheimbase1-heimdal
libheimntlm0-heimdal
libhx509-5-heimdal
libice-dev
libice6
libidl-common
libidl0
libidn11
libio-string-perl
libisc83
libisccc80
libisccfg82
libiw30
libjasper1
libjpeg-turbo8
libjpeg8
libjs-jquery
libjson0
libk5crypto3
libkeyutils1
libklibc
libkrb5-26-heimdal
libkrb5-3
libkrb5support0
liblcms2-2
libldap-2.4-2
libllvm3.0
liblocale-gettext-perl
liblockfile-bin
liblockfile1
libltdl7
liblvm2app2.2
liblwres80
liblzma5
libm17n-0
libmagic1
libmount1
libmpc2
libmpfr4
libmysqlclient18
libncurses5
libncursesw5
libnewt0.52
libnfnetlink0
libnih-dbus1
libnih1
libnl-3-200
libnl-genl-3-200
libnspr4
libnss3
libnss3-1d
libogg0
liborbit2
libotf0
libp11-kit0
libpam-ck-connector
libpam-modules
libpam-modules-bin
libpam-runtime
libpam0g
libpango1.0-0
libparse-debianchangelog-perl
libparted0debian1
libpcap0.8
libpci3
libpciaccess0
libpcre3
libpcsclite1
libpipeline1
libpixman-1-0
libplymouth2
libpng12-0
libpolkit-agent-1-0
libpolkit-backend-1-0
libpolkit-gobject-1-0
libpopt0
libpthread-stubs0
libpthread-stubs0-dev
libpulse0
libpython2.7
libreadline5
libreadline6
libroken18-heimdal
librsvg2-2
librtmp0
libsasl2-2
libsasl2-modules
libselinux1
libsgutils2-2
libsigc++-2.0-0c2a
libslang2
libsm-dev
libsm6
libsndfile1
libsqlite3-0
libss2
libssl1.0.0
libstdc++6
libsub-name-perl
libswitch-perl
libtasn1-3
libtdb1
libterm-readkey-perl
libterm-readline-perl-perl
libtext-charwidth-perl
libtext-iconv-perl
libtext-wrapi18n-perl
libthai-data
libthai0
libtiff4
libtimedate-perl
libtinfo5
libtorrent14
libudev0
libusb-0.1-4
libusb-1.0-0
libuuid1
libvorbis0a
libvorbisenc2
libvorbisfile3
libwind0-heimdal
libwrap0
libx11-6
libx11-data
libx11-dev
libx11-doc
libx11-xcb1
libxapian22
libxau-dev
libxau6
libxaw7
libxcb-glx0
libxcb-render0
libxcb-shape0
libxcb-shm0
libxcb1
libxcb1-dev
libxcomposite1
libxcursor1
libxdamage1
libxdmcp-dev
libxdmcp6
libxerces2-java
libxext6
libxfixes3
libxft2
libxi6
libxinerama1
libxml-commons-external-java
libxml-commons-resolver1.1-java
libxml2
libxmlrpc-core-c3
libxmu6
libxmuu1
libxpm4
libxrandr2
libxrender1
libxt-dev
libxt6
libxtst6
libxv1
libxxf86dga1
libxxf86vm1
libyaml-0-2
lighttpd
linux-firmware
linux-image-3.2.0-25-virtual
linux-image-3.2.0-26-virtual
linux-image-3.2.0-27-virtual
linux-image-virtual
linux-virtual
locales
lockfile-progs
login
logrotate
lsb-base
lsb-release
lshw
lsof
ltrace
m17n-contrib
m17n-db
makedev
man-db
manpages
mawk
memcached
memtest86+
mercurial
mercurial-common
mime-support
mlocate
module-init-tools
mount
mountall
mtools
mtr-tiny
multiarch-support
mysql-common
nano
ncurses-base
ncurses-bin
net-tools
netbase
netcat-openbsd
nethogs
ntfs-3g
ntpdate
openjdk-6-jre
openjdk-6-jre-headless
openjdk-6-jre-lib
openjdk-7-jdk
openjdk-7-jre
openjdk-7-jre-headless
openjdk-7-jre-lib
openssh-client
openssh-server
openssl
os-prober
parted
passwd
patch
pciutils
perl
perl-base
perl-modules
php5
php5-cgi
php5-cli
php5-common
php5-mysql
plymouth
plymouth-theme-ubuntu-text
policykit-1
policykit-1-gnome
popularity-contest
powermgmt-base
ppp
pppconfig
pppoeconf
procps
psmisc
python
python-apport
python-apt
python-apt-common
python-boto
python-chardet
python-cheetah
python-configobj
python-crypto
python-dbus
python-dbus-dev
python-debian
python-gdbm
python-gi
python-gnupginterface
python-httplib2
python-keyring
python-launchpadlib
python-lazr.restfulclient
python-lazr.uri
python-m2crypto
python-minimal
python-newt
python-oauth
python-openssl
python-pam
python-paramiko
python-pkg-resources
python-problem-report
python-pycurl
python-serial
python-simplejson
python-software-properties
python-twisted-bin
python-twisted-core
python-twisted-names
python-twisted-web
python-wadllib
python-xapian
python-yaml
python-zope.interface
python2.7
python2.7-minimal
readline-common
resolvconf
rsync
rsyslog
rtorrent
screen
sed
sensible-utils
sgml-base
shared-mime-info
sound-theme-freedesktop
spawn-fcgi
ssh-import-id
strace
sudo
sysv-rc
sysvinit-utils
tar
tasksel
tasksel-data
tcpd
tcpdump
telnet
time
tmux
ttf-dejavu-core
ttf-dejavu-extra
tzdata
tzdata-java
ubuntu-keyring
ubuntu-minimal
ubuntu-standard
ucf
udev
udisks
ufw
unattended-upgrades
update-manager-core
update-notifier-common
upstart
ureadahead
usbutils
util-linux
uuid-runtime
vim
vim-common
vim-runtime
vim-tiny
w3m
wget
whiptail
whoopsie
wireless-regdb
wireless-tools
wpasupplicant
x11-common
x11-utils
x11proto-core-dev
x11proto-input-dev
x11proto-kb-dev
xauth
xfsprogs
xkb-data
xml-core
xorg-sgml-doctools
xtrans-dev
xz-lzma
xz-utils
zlib1g

These was the list of packages I ended up installing on a new web server machine until I stopped needing to put more things in it. I’ll come back and update this list in a few months.

I’ve put this list together because my AWS EC2 image instantiation didn’t work, so I did the old school solution, just install the same packages on a new machine and configure the computer, oh well.

ubuntu/debian abort: error: _ssl.c:504: error:14090086:SSL [FIXED]

Trying to clone or update a repo, and you get this error?
[bash]
hg pull -u
abort: error: _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed
[/bash]

Quick fix, go to .hgrc and put this on your [web] section

[bash]
[web]
cacerts=
[/bash]

Fixed, remember to leave a tip if you wasted hours on this and I just saved your ass 😉 j/k, but no seriously, leave a tip, pleease.

lighttpd, allow “Access-Control-Allow-Origin:*” headers on the server status page

Maybe there’s someone out there who needs to read the output of lighttpd’s status for monitoring purpose like me tonight, and also, like me, you want to do this using JavaScript, but your browser gives you this nasty error:

XMLHttpRequest cannot load http://otherSubdomain.server.com/lighttpd-status-url-you-have-configured. Origin http://requestingSubdomain.server.com is not allowed by Access-Control-Allow-Origin.

lighttpd allows you to add a custom header for all requests by adding this in a given context:

[perl]setenv.add-response-header = ( "Access-Control-Allow-Origin" => "*" )[/perl]

For this to work, you must enable the mod_setenv.

But if you don’t enable this module, before you enable your mod_status module, you will never see the custom headers come out of your lighttpd HTTP response header output.

So make sure you enable mod_setenv like this:

[perl]
server.modules = (
"mod_fastcgi",
"mod_auth",
"mod_access",
"mod_alias",
"mod_accesslog",
# "mod_simple_vhost",
"mod_rewrite",
"mod_redirect",
"mod_setenv", #before mod_status, very important!
"mod_status",
# "mod_evhost",
"mod_compress",

[/perl]

The header output of your lighttpd status page should look like this now:

[bash]
Access-Control-Allow-Origin:*
Content-Length:5952
Content-Type:text/html
Date:Wed, 30 Nov 2011 01:27:04 GMT
Server:lighttpd/1.4.28
[/bash]

Hope this helps you.

[SOLVED] Updated to Ubuntu 11.10 on VirtualBox and now you get a blank screen when you reboot?

So you used the update manager to upgrade to the new Ubuntu 11.10 that came out a couple days ago and when you’re finished with the process, next time it boots you get a blank screen.

Do this:

1. Switch to a terminal.
Press “Ctrl+Alt+F1” if you’re on Windows.
or
Press “Cmd + F1” or “Ctrl + Alt_option + F1” if you on MacOSX.

You will now be on a text based terminal (TTY1), and you will see that your VM has booted fine, you just didn’t have a graphical environment, therefore the blank screen.

2. Login in with your account on this text based terminal by entering your credentials.

3. Install virtualbox-ose-guest-utils.
(Yes, I also had my virtual guest additions installed before so that I could have full screen resolution, but the problem seems to be related to this and now ubuntu has a package on their repos that you need.)

sudo apt-get install virtualbox-ose-guest-utils [enter]

4. Reboot
sudo reboot

You’re done.

Have the latest HAProxy as a Ubuntu Service

So you need to use HAProxy and you love the convenience of binary packages on repos, but when you install the version HAProxy available in the repos you realize that it is way too old for what you need.

Then you download the latest HAProxy, compile it, configure it, but it’s a bit of a pain in the ass to not have the convenience of having haproxy be automatically restarted as a service like those available on /etc/init.d

This post teaches you how to have haproxy as a Ubuntu/Debian service.

First copy or symlink this script to your /etc/init.d/ folder (you’ll need root permissions to do this)

[bash]
#!/usr/bin/env bash
# haproxyd
# Script to start|stop|restart haproxy from /etc/init.d/
# By Gubatron.

HAPROXY_PATH=/path/to/haproxy-X.Y.Z
HAPROXY_DAEMON=$HAPROXY_PATH/haproxy

test -x $HAPROXY_DAEMON || exit 0

set -e

function getHaproxyPID() {
PID=`ps aux | grep ‘haproxy -f’ | grep -v "grep" | awk ‘{ print $2 }’`
}

case $1 in
start)
echo "Starting haproxy…"
$HAPROXY_DAEMON -f $HAPROXY_PATH/haproxy.cfg
;;
restart)
echo "Hot restart of haproxy"
getHaproxyPID
COMMAND="$HAPROXY_DAEMON -f $HAPROXY_PATH/haproxy.cfg -sf $PID"
echo $COMMAND
`$COMMAND`
;;
stop)
echo "Stopping haproxy"
getHaproxyPID
COMMAND="kill -9 $PID"
echo $COMMAND
`$COMMAND`
;;
*)
echo "Usage: haproxyd {start|restart|stop}" >&2
exit 1
;;
esac

exit 0
[/bash]

This script, on it’s own can be used as
[bash]
./haproxyd start
./haproxyd restart
./haproxyd stop
[/bash]

But you want this script registered on all the right runlevels of the operating system.

With Ubuntu/Debian there’s a utility called update-rc.d to register /etc/init.d/ scripts very easily.

Once the script above is available on /etc/init.d do the following

[bash]
cd /etc/init.d
sudo update-rc.d haproxyd defaults
[/bash]

The script should now be registered on all the right runlevels and you should be able to invoke it as a service like

[bash]
sudo service haproxyd <command>
[/bash]

Ubuntu: WiCD Network Manager “Connection Failed: Bad Password” [SOLVED]

If you’re familiar with correctly entering your WPA2 password after an Ubuntu update and now your Netbook won’t connect using WiCD, I got the solution that worked for me.

Uninstall network-manager and restart WiCD

Open a terminal and type:
[shell]
$ sudo apt-get remove network-manager

$ sudo /etc/init.d/wicd restart
[/shell]

You may have to restart wicd daemon twice in case you don’t see the wireless networks right away.

startKeychain – bash utility to start ssh-agent

For my (and your) future reference, here’s a function to put on your .bashrc or .bash_profile, you can invoke it later at any time to start/re-start your ssh-agent.


[bash]
function startKeychain {
killall ssh-agent
rm ~/.keychain/*
keychain id_rsa
HOSTNAME=`hostname`
source ~/.keychain/${HOSTNAME}-sh
}
[/bash]

Then at any time, the “command” startKeychain will be available on your command line.

Output should look something like this:
[bash]

gubatron@gubatron-desktop:~$ startKeychain

KeyChain 2.6.8; http://www.gentoo.org/proj/en/keychain/
Copyright 2002-2004 Gentoo Foundation; Distributed under the GPL

* Initializing /home/gubatron/.keychain/gubatron-desktop-sh file…
* Initializing /home/gubatron/.keychain/gubatron-desktop-csh file…
* Initializing /home/gubatron/.keychain/gubatron-desktop-fish file…
* Starting ssh-agent
* Initializing /home/gubatron/.keychain/gubatron-desktop-sh-gpg file…
* Initializing /home/gubatron/.keychain/gubatron-desktop-csh-gpg file…
* Initializing /home/gubatron/.keychain/gubatron-desktop-fish-gpg file…
* Starting gpg-agent
* Adding 1 ssh key(s)…
Identity added: /home/gubatron/.ssh/id_rsa (/home/gubatron/.ssh/id_rsa)

[/bash]

Comments are welcome to improve it, I’m not an ssh-agent expert, but this seems to do the work.

[SOLVED] Issue with KDE 4.4.2 and Dolphin always asking my ssh passwords whenever I browsed folder I checked out from a remote subversion repository.

[SOLVED] Eclipse can’t see my Android Device on Ubuntu

Are you seeing this on eclipse when you plug your Android device to your Ubuntu box?

Serial Number: ??????????
AVD Name: N/A
Target: unknown
State: ??

Here’s the solution:

1. Create a script to fix this next time it happens, let’s call it “android_device_reset” and save it on a folder contained on your $PATH environment variable.

[bash]
#!/bin/bash
# android_device_reset script
sudo adb kill-server
sudo service udev stop
sudo adb start-server
sudo adb devices
[/bash]

Save it and make it executable
[bash]chmod +x android_device_reset[/bash]

2. Open this file /etc/udev/rules.d/51-android.rules

Make sure it looks something like this
[bash]
SUBSYSTEMS=="usb", SYSFS{idVendor}=="0bb4", MODE=="0666"
SUBSYSTEMS=="usb", SYSFS{idVendor}=="22b8", MODE=="0666"
[/bash]

Each line represents a different android device. If you have just one, the file should be one line long.

On that example I’ve configured the rules for a Motorola Droid and a Nexus One.
If you need to know the idVendor numbers for your Android device go here
developer.android.com/guide/developing/device.html#VendorIds

3. Whenever the problem happens, just open a terminal and type
[bash]android_device_reset[/bash]

It’ll ask you for your password, only administrative users will be able to execute the script.

Hope this helps.

Quick N Dirty way to Map Commands to remote servers via ssh

You may be running several independent but similar servers at the same time and wasting time by executing commands in all of them one by one.

Wouldn’t it be nice to send a command to all of them at once? or to monitor all of them at once.

The following script can be used as a building block to more complex automation tasks for a small size set of servers. (If you’re managing over 50 servers, I’d probably consider looking a different way to arrange servers (map/reduce cluster), but if you’re doing something below that number this might suffice)

[code lang=”python”]
#!/usr/bin/python

#########################################################
# Author: Angel Leon (gubatron@gmail.com) – October 2009
#
# Invokes a command locally and invokes the same command
# in all machines under the specified username, servers
#
# Requirement: Have a public ssh_key for that user on all
# the other machines so you don’t have to authenticate
# on all the other machines.
#########################################################
import sys
import os

# set the username that has access to all the machines here
user=’safeuser’

# add all your server names here
servers=[‘server1.mydomain.com’,’server2.mydomain.com’,’server3.mydomain.com’]

if __name__ == "__main__":
if len(sys.argv) < 2:
print "Usage: ssh_map_command <cmd>"
sys.exit(0)

cmd= ‘ ‘.join(sys.argv[1:])

#Execute locally first
print cmd
os.system(cmd)

#Execute for all the servers in the list
for server in servers:
remote_cmd="ssh %s@%s %s" % (user,server,cmd)
print remote_cmd
os.system(remote_cmd)
print
[/code]

Save as ssh_map_command and chmod +x it.

Sample uses
Check the average load of all machines at once (then use output to mitigate high load issues)
[code lang=”shell”]$ ssh_map_command uptime[/code]

Send HUP signal to all your web servers (put it in an alias or other script… and that’s how you start building more complex scripts)
[code lang=”shell”]$ ssh_map_command ps aux | grep [l]ighttpd | kill -HUP `awk {‘print $2’}`[/code]

Check if processes are alive, check memory usage on processes across different machines, grep remote all logs at once, svn up on all machines, rsync from one to many, hey, you can even tail -f and grep all the logs at once, you can go nuts with this thing. Depends on what you need to do.

Requirements

Security Advisory
Make sure only the desired user has read/write/execute access to it and keep your private ssh keys safe (preferably only read and execute for the owner, and no permissions whatsoever to anybody else chmod 500 ssh_mod_map), if possible change them as often as possible, for it may become a big security whole if an attacker can manage to write code on this script, specially if you have cronjobs invoking it. Your attacker would only need to change code here to mess up all of your machines.

Disclaimer and Call for Knowledge
Please, if someone knows of a standard way to map commands to multiple servers, please let me know in the comment section, in my case I needed a solution and I wrote a quick and dirty python script and tried to secure it as best as I could, by no means I’m saying that this is the best solution to mapping commands, in fact I believe it might be the least efficient way, however it works good enough for my personal needs.

Ubuntu/Debian Quick Reference: How To Change Your Server’s UTC Timezone on the command line

Just Type…
sudo dpkg-reconfigure tzdata

…and follow the instructions on screen.

The process should look something like the following:


Select your Region


Select a city on your time zone


You’re done.

Tip
You can always check the status of your configuration using
sudo debconf-show tzdata

You could for example map that command via ssh to several machines and grep for “*”, that way you could easily spot servers with wrong timezones very quickly.