HOW TO ENABLE PHP FPM LOG OUTPUT

This one had me for the longest time.

If you happen to be running a web server with php-fpm, sometimes you will run across an HTTP 500 error and all you will get will be a blank screen.

You will look at your server’s error log, your vhost error’s log and you will see nothing.

At this point you will want to enable logging on php fpm to see what’s up.

So you will go to your /etc/php/7.0/fpm/php-fpm.conf

you will start searching for “log”, and you will come across
error_log = /var/log/php7.0-fpm.log
you will tail -f that log file, and nothing will come up.

you will go back to that config file, you will play with your log levels, and nothing, and that’s because there’s this fucking obscure setting on your pool configuration that you’d never think of.

Let’s say you’re using the default www.conf pool config file (the one sitting at /etc/php/7.0/fpm/pool.d/www.conf), open it and look for “workers”, you will see this:

; Redirect worker stdout and stderr into main error log. If not set, stdout and
; stderr will be redirected to /dev/null according to FastCGI specs.
; Note: on highloaded environement, this can cause some delay in the page
; process time (several ms).
; Default Value: no
;catch_workers_output = yes

uncomment catch_workers_output=yes
and restart your php-fpm service, tail -f your log and you will see the stack trace you’re looking for.

You’re welcome.

How to resize an EBS (xfs formatted) partition

First of all, create a snapshot of your EBS volume. Then out of that snapshot you will be able to create your new volume.

However, when you detach the old one from your instance and attach the new one, you will still see the old available space with df

look at my /dev/xvdf/ available space (after mounting the new EBS volume)

ubuntu@ip-10-47-167-74:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1       16G  5.9G  9.1G  40% /
udev            7.4G   12K  7.4G   1% /dev
tmpfs           1.5G  176K  1.5G   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            7.4G     0  7.4G   0% /run/shm
/dev/xvdf        20G   19G  1.7G  92% /media/ebs/data  <<< this one

still 20G, I mounted a 80G one!

This drive has been formatted to use an xfs file system. In order to resize it, this is the command I used:

sudo xfs_growfs -d /media/ebs/data

output should be something like this

ubuntu@ip-10-47-167-74:~$ sudo xfs_growfs -d /media/ebs/data
meta-data=/dev/xvdf              isize=256    agcount=4, agsize=1310720 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=5242880, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
data blocks changed from 5242880 to 20971520

now let’s see the df -h output

ubuntu@ip-10-47-167-74:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1       16G  5.9G  9.1G  40% /
udev            7.4G   12K  7.4G   1% /dev
tmpfs           1.5G  176K  1.5G   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            7.4G     0  7.4G   0% /run/shm
/dev/xvdf        80G   19G   62G  23% /media/ebs/data

WordPress: Cannot create directory error, got everything chmoded to 777?

So you went as far as chmod’ing your entire folder structure to 777 (very unsafe), you’ve hacked wp/wp-admin/includes/file.php

return new WP_Error( 'mkdir_failed_ziparchive', __( get_current_user() . ' Could not create directory. ('.$_dir.')' )...

to print out exactly what directory it cannot create and what user is trying to create the folder, everything is correct, yet it won’t create the fucking folder?

the issue might be your vsftpd configuration!

Go to /etc/vsftpd.con and make sure that you have this setting uncommented:

write_enable=YES

restart your vsftpd (sudo service vsftpd restart) and try upgrading again, you’ll want to hit that ‘Donate Bitcoin’ button if I saved your ass today.

Cheers

AWS troubleshooting: how to fix a broken EBS volume (bad superblock on xfs)

As great as EBS volumes are on Amazon Web Services, they can break and not ever mount again, even though your data could still be there intact, a simple corruption on the filesystem structure can cause a lot of damage. On this post I teach you how to move all that data onto a new EBS drive, so keep calm and read slowly.

So, you try to mount your drive after some updates and you get an error like this on dmesg | tail:

[56439860.329754] XFS (xvdf): Corruption detected. Unmount and run xfs_repair

so you unmount your drive, invoke xfs_repair and you get this…

$ sudo xfs_repair -n /dev/xvdf
Phase 1 - find and verify superblock...
bad primary superblock - bad magic number !!!

attempting to find secondary superblock...
..........................................

and no good secondary superblock is found.

Don’t panic, this is what you have to do next to solve this issue:

  1. Go to your AWS dashboard, EC2 section.
  2. Click on “Volumes”
  3. Find the broken volume.
  4. Create a snapshot of the broken volume (this takes a while)
  5. Create a new volume the same size (or larger than) your old drive out of the snapshot you just created (this takes a while)
  6. Attach your new volume to the same EC2 instance (no need to reboot or anything), if the old drive was mapped to /dev/xvdf, the new one will be mapped to /dev/xvdg (see how the last letter increases alphabetically)

Now here’s a gotcha, Amazon will not create your new drive using the same file system type (xfs), for some reason it will create it using the ext2 filesystem.

$ sudo file -s /dev/xvdg
/dev/xvdg: Linux rev 1.0 ext2 filesystem data (mounted or unclean), UUID=2e35874f-1d21-4d2d-b42b-ae27966e0aab (large files)

Here you have two options:
1. Live with the new ext2 file system, make sure your /etc/fstab is updated to look something like this:
/dev/xvdg /path/to/mount/to auto defaults,nobootwait,noatime 0 0

or 2. copy the contents of your drive to a temporary location, usually inside /mnt which has plenty of space from that ephemeral drive the ec2 instances come to, and then mkfs.xfs the new volume, and then copy the contents back… (which what I did, as I chose to create a larger drive and the ext2 format that came on the new volume only recognized the size of the snapshot)

Hope this saved your ass, leave a note if I did.

Remember to never do any irreversible action until you have a disk snapshot, try your best to never lose data.

ubuntu packages for a kick ass web server

Copy and paste the following list on a file, say “packages.txt”. To install all just do:

sudo apt-get install $(cat packages.txt)

accountsservice
acpid
adduser
ant
ant-optional
apache2-utils
apparmor
apport
apport-symptoms
apt
apt-transport-https
apt-utils
apt-xapian-index
aptitude
at
base-files
base-passwd
bash
bash-completion
bc
bind9-host
bsdmainutils
bsdutils
busybox-initramfs
busybox-static
byobu
bzip2
ca-certificates
ca-certificates-java
cloud-init
cloud-initramfs-growroot
cloud-initramfs-rescuevol
cloud-utils
command-not-found
command-not-found-data
console-setup
consolekit
coreutils
cpio
cpp
cpp-4.6
crda
cron
cryptsetup-bin
curl
dash
dbus
dbus-x11
dconf-gsettings-backend
dconf-service
debconf
debconf-i18n
debianutils
diffutils
dmidecode
dmsetup
dnsutils
dosfstools
dpkg
dstat
e2fslibs
e2fsprogs
ed
eject
emacs
emacs23
emacs23-bin-common
emacs23-common
emacsen-common
euca2ools
file
findutils
fontconfig
fontconfig-config
fonts-ubuntu-font-family-console
friendly-recovery
ftp
fuse
gamin
gcc-4.6-base
gconf-service
gconf-service-backend
gconf2
gconf2-common
geoip-database
gettext-base
gir1.2-glib-2.0
gir1.2-gudev-1.0
gnupg
gpgv
grep
groff-base
grub-common
grub-gfxpayload-lists
grub-legacy-ec2
grub-pc
grub-pc-bin
grub2-common
gvfs
gvfs-common
gvfs-daemons
gvfs-libs
gzip
hdparm
hicolor-icon-theme
hostname
htop
icedtea-6-jre-cacao
icedtea-6-jre-jamvm
icedtea-7-jre-cacao
icedtea-7-jre-jamvm
icedtea-netx
icedtea-netx-common
ifupdown
info
initramfs-tools
initramfs-tools-bin
initscripts
insserv
install-info
iproute
iptables
iputils-ping
iputils-tracepath
irqbalance
isc-dhcp-client
isc-dhcp-common
iso-codes
iw
java-common
kbd
keyboard-configuration
klibc-utils
krb5-locales
landscape-client
landscape-common
language-selector-common
laptop-detect
less
libaccountsservice0
libacl1
libapr1
libaprutil1
libapt-inst1.4
libapt-pkg4.12
libasn1-8-heimdal
libasound2
libasyncns0
libatasmart4
libatk-wrapper-java
libatk-wrapper-java-jni
libatk1.0-0
libatk1.0-data
libattr1
libavahi-client3
libavahi-common-data
libavahi-common3
libavahi-glib1
libbind9-80
libblkid1
libbonobo2-0
libbonobo2-common
libboost-iostreams1.46.1
libbsd0
libbz2-1.0
libc-bin
libc6
libcairo-gobject2
libcairo2
libcanberra0
libcap-ng0
libck-connector0
libclass-accessor-perl
libclass-isa-perl
libcomerr2
libcroco3
libcryptsetup4
libcups2
libcurl3
libcurl3-gnutls
libcwidget3
libdatrie1
libdb5.1
libdbus-1-3
libdbus-glib-1-2
libdconf0
libdevmapper-event1.02.1
libdevmapper1.02.1
libdns81
libdrm-intel1
libdrm-nouveau1a
libdrm-radeon1
libdrm2
libedit2
libelf1
libept1.4.12
libevent-2.0-5
libexpat1
libffi6
libflac8
libfontconfig1
libfontenc1
libfreetype6
libfribidi0
libfuse2
libgamin0
libgc1c2
libgcc1
libgconf-2-4
libgconf2-4
libgcrypt11
libgd2-noxpm
libgdbm3
libgdk-pixbuf2.0-0
libgdk-pixbuf2.0-common
libgdu0
libgeoip1
libgif4
libgirepository-1.0-1
libgl1-mesa-dri
libgl1-mesa-glx
libglapi-mesa
libglib2.0-0
libgmp10
libgnome-keyring-common
libgnome-keyring0
libgnome2-0
libgnome2-common
libgnomevfs2-0
libgnomevfs2-common
libgnutls26
libgpg-error0
libgpm2
libgssapi-krb5-2
libgssapi3-heimdal
libgtk-3-0
libgtk-3-bin
libgtk-3-common
libgtk2.0-0
libgtk2.0-bin
libgtk2.0-common
libgudev-1.0-0
libhcrypto4-heimdal
libheimbase1-heimdal
libheimntlm0-heimdal
libhx509-5-heimdal
libice-dev
libice6
libidl-common
libidl0
libidn11
libio-string-perl
libisc83
libisccc80
libisccfg82
libiw30
libjasper1
libjpeg-turbo8
libjpeg8
libjs-jquery
libjson0
libk5crypto3
libkeyutils1
libklibc
libkrb5-26-heimdal
libkrb5-3
libkrb5support0
liblcms2-2
libldap-2.4-2
libllvm3.0
liblocale-gettext-perl
liblockfile-bin
liblockfile1
libltdl7
liblvm2app2.2
liblwres80
liblzma5
libm17n-0
libmagic1
libmount1
libmpc2
libmpfr4
libmysqlclient18
libncurses5
libncursesw5
libnewt0.52
libnfnetlink0
libnih-dbus1
libnih1
libnl-3-200
libnl-genl-3-200
libnspr4
libnss3
libnss3-1d
libogg0
liborbit2
libotf0
libp11-kit0
libpam-ck-connector
libpam-modules
libpam-modules-bin
libpam-runtime
libpam0g
libpango1.0-0
libparse-debianchangelog-perl
libparted0debian1
libpcap0.8
libpci3
libpciaccess0
libpcre3
libpcsclite1
libpipeline1
libpixman-1-0
libplymouth2
libpng12-0
libpolkit-agent-1-0
libpolkit-backend-1-0
libpolkit-gobject-1-0
libpopt0
libpthread-stubs0
libpthread-stubs0-dev
libpulse0
libpython2.7
libreadline5
libreadline6
libroken18-heimdal
librsvg2-2
librtmp0
libsasl2-2
libsasl2-modules
libselinux1
libsgutils2-2
libsigc++-2.0-0c2a
libslang2
libsm-dev
libsm6
libsndfile1
libsqlite3-0
libss2
libssl1.0.0
libstdc++6
libsub-name-perl
libswitch-perl
libtasn1-3
libtdb1
libterm-readkey-perl
libterm-readline-perl-perl
libtext-charwidth-perl
libtext-iconv-perl
libtext-wrapi18n-perl
libthai-data
libthai0
libtiff4
libtimedate-perl
libtinfo5
libtorrent14
libudev0
libusb-0.1-4
libusb-1.0-0
libuuid1
libvorbis0a
libvorbisenc2
libvorbisfile3
libwind0-heimdal
libwrap0
libx11-6
libx11-data
libx11-dev
libx11-doc
libx11-xcb1
libxapian22
libxau-dev
libxau6
libxaw7
libxcb-glx0
libxcb-render0
libxcb-shape0
libxcb-shm0
libxcb1
libxcb1-dev
libxcomposite1
libxcursor1
libxdamage1
libxdmcp-dev
libxdmcp6
libxerces2-java
libxext6
libxfixes3
libxft2
libxi6
libxinerama1
libxml-commons-external-java
libxml-commons-resolver1.1-java
libxml2
libxmlrpc-core-c3
libxmu6
libxmuu1
libxpm4
libxrandr2
libxrender1
libxt-dev
libxt6
libxtst6
libxv1
libxxf86dga1
libxxf86vm1
libyaml-0-2
lighttpd
linux-firmware
linux-image-3.2.0-25-virtual
linux-image-3.2.0-26-virtual
linux-image-3.2.0-27-virtual
linux-image-virtual
linux-virtual
locales
lockfile-progs
login
logrotate
lsb-base
lsb-release
lshw
lsof
ltrace
m17n-contrib
m17n-db
makedev
man-db
manpages
mawk
memcached
memtest86+
mercurial
mercurial-common
mime-support
mlocate
module-init-tools
mount
mountall
mtools
mtr-tiny
multiarch-support
mysql-common
nano
ncurses-base
ncurses-bin
net-tools
netbase
netcat-openbsd
nethogs
ntfs-3g
ntpdate
openjdk-6-jre
openjdk-6-jre-headless
openjdk-6-jre-lib
openjdk-7-jdk
openjdk-7-jre
openjdk-7-jre-headless
openjdk-7-jre-lib
openssh-client
openssh-server
openssl
os-prober
parted
passwd
patch
pciutils
perl
perl-base
perl-modules
php5
php5-cgi
php5-cli
php5-common
php5-mysql
plymouth
plymouth-theme-ubuntu-text
policykit-1
policykit-1-gnome
popularity-contest
powermgmt-base
ppp
pppconfig
pppoeconf
procps
psmisc
python
python-apport
python-apt
python-apt-common
python-boto
python-chardet
python-cheetah
python-configobj
python-crypto
python-dbus
python-dbus-dev
python-debian
python-gdbm
python-gi
python-gnupginterface
python-httplib2
python-keyring
python-launchpadlib
python-lazr.restfulclient
python-lazr.uri
python-m2crypto
python-minimal
python-newt
python-oauth
python-openssl
python-pam
python-paramiko
python-pkg-resources
python-problem-report
python-pycurl
python-serial
python-simplejson
python-software-properties
python-twisted-bin
python-twisted-core
python-twisted-names
python-twisted-web
python-wadllib
python-xapian
python-yaml
python-zope.interface
python2.7
python2.7-minimal
readline-common
resolvconf
rsync
rsyslog
rtorrent
screen
sed
sensible-utils
sgml-base
shared-mime-info
sound-theme-freedesktop
spawn-fcgi
ssh-import-id
strace
sudo
sysv-rc
sysvinit-utils
tar
tasksel
tasksel-data
tcpd
tcpdump
telnet
time
tmux
ttf-dejavu-core
ttf-dejavu-extra
tzdata
tzdata-java
ubuntu-keyring
ubuntu-minimal
ubuntu-standard
ucf
udev
udisks
ufw
unattended-upgrades
update-manager-core
update-notifier-common
upstart
ureadahead
usbutils
util-linux
uuid-runtime
vim
vim-common
vim-runtime
vim-tiny
w3m
wget
whiptail
whoopsie
wireless-regdb
wireless-tools
wpasupplicant
x11-common
x11-utils
x11proto-core-dev
x11proto-input-dev
x11proto-kb-dev
xauth
xfsprogs
xkb-data
xml-core
xorg-sgml-doctools
xtrans-dev
xz-lzma
xz-utils
zlib1g

These was the list of packages I ended up installing on a new web server machine until I stopped needing to put more things in it. I’ll come back and update this list in a few months.

I’ve put this list together because my AWS EC2 image instantiation didn’t work, so I did the old school solution, just install the same packages on a new machine and configure the computer, oh well.

Reference: Linux Network/Bandwidth Monitoring CLI Tools

I often want to see how much bandwidth is consumed by my network interfaces and how this is happening. There’s plenty of tools available in the linux world to monitor your network activity.

If you’re a Ubuntu or Debian user you can try the ones I use by installing the following packages.

sudo apt-get install iptraf ethstatus dstat iftop ifstat nload bwm-ng

Then try each one of those on your command line and have fun exploring what they can do.

If they’re not on the path of your user account, then try them using sudo.
eg. sudo iftop

If this short list of the tools I use does not satisfy you, I suggest you read this more comprehensive list of monitoring tools