[SYSADMIN] Serve your WordPress cached pages directly with lighttpd and not PHP

Optimizing Your WordPress Cache Loads in Lighttpd.

If you don’t configure your wordpress virtual host properly in lighttpd, your wordpress cache will still make use of PHP.

Wouldn’t it be nice if all those cached requests were served directy from the webserver as the static files that they are, bypassing the CPU/memory load PHP can have, and use those resources for otherthings?

Install and Enable mod_magnet

For this to occur with lighttpd, you will need mod_magnet, so assuming you’re on a Ubuntu/Debian based linux distro let’s make sure we have it installed.

sudo apt-get install lighttpd-mod-magnet

Then let’s make sure it’s enabled, you can do this manually on your lighttpd.conf by adding “mod_magnet” to the list of enabled modules…

server.modules = (
        "mod_fastcgi",
        "mod_access",
        "mod_alias",
        "mod_accesslog",
        "mod_compress",
        "mod_rewrite",
        "mod_redirect",
        "mod_status",
        "mod_proxy",
        "mod_setenv",
        "mod_magnet"
)

or you can do it the lighty way:

sudo lighty-enable-mod magnet

(this simply makes a symlink to the 10-magnet.conf file inside /etc/lighttpd/conf-enabled which lighty will check upon startup)

The cache logic script that will be executed by lighttpd

Now, on your wordpress directory, create a file called rewrite.lua and paste the following script in it:

function log(str)
   -- wanna tail -f a log to see what's happening    
   fp = io.open("/path/to/some/lua.log","a+")
   fp:write(str .. "\n")
   fp:flush()
   fp:close()
end

function serve_html(cached_page)
    if (lighty.stat(cached_page)) then
        lighty.env["physical.path"] = cached_page
        return true
    else
        return false
    end
end

function serve_gzip(cached_page)
    if (lighty.stat(cached_page .. ".gz")) then
        lighty.header["Content-Encoding"] = "gzip"
        lighty.header["Content-Type"] = ""
        lighty.env["physical.path"] = cached_page .. ".gz"
        return true
    else
        return false
    end
end

if (lighty.env["uri.scheme"] == "http") then
    ext = ".html"
else
    ext = "-https.html"
end

cached_page = lighty.env["physical.doc-root"] .. "/wp-content/cache/supercache/" .. lighty.request["Host"] .. lighty.env["request.orig-uri"]
cached_page = string.gsub(cached_page, "//", "/")
cached_page = string.gsub(cached_page, lighty.request["Host"] .. "/index.php", lighty.request["Host"])

attr = lighty.stat(cached_page)

if (attr) then
    query_condition = not (lighty.env["uri.query"] and string.find(lighty.env["uri.query"], ".*s=.*"))
    user_cookie = lighty.request["Cookie"] or "no_cookie_here"
    cookie_condition = not (string.find(user_cookie, ".*comment_author.*") or (string.find(user_cookie, ".*wordpress.*") and not string.find(user_cookie,"wordpress_test_cookie")))

    if (query_condition and cookie_condition) then
        accept_encoding = lighty.request["Accept-Encoding"] or "no_acceptance"

        if (string.find(accept_encoding, "gzip")) then
            if not serve_gzip(cached_page) then 
                serve_html(cached_page) 
            end
        else
            serve_html(cached_page)
        end
        --log('cache-hit: ' .. cached_page)
    end
else
    --log('cache-miss: ' .. cached_page)
end

Configuring your vhost in lighttpd for WordPress redirects and direct cache serves without php.

Then on your vhost configuration in lighttpd.conf add the following towards the end.
(Fix paths if you have to)

var.wp_blog = 1

magnet.attract-physical-path-to = ( server.document-root + "/rewrite.lua" )

url.rewrite-if-not-file = (
   "^/(wp-.+).*/?" => "$0",
   "^/(sitemap.xml)" => "$0",
   "^/(xmlrpc.php)" => "$0",
   "^/(.+)/?$" => "/index.php/$1"
  )

Restart your lighttpd sudo service lighttpd restart

Now watch how your PHP processes breathe a lot better and you page loads are insanely faster.

You’re welcome πŸ™‚

How to stop Twitter SMS notifications from 40404

Simply send a text message back to 40404 with the following text:

STOP TWITTER

If for some reason this no longer works, then just add the 40404 number to your blocked caller’s list.

For me the “STOP TWITTER” text message did it. I was receiving text messages even though I didn’t have any mobile notifications set in my twitter account.

Hope this helps, and if it does say hi on twitter @gubatron.

Cheers

Things to think about when tuning libtorrent for high performance

An user on the libtorrent mailing list had the following problem:

I’m doing some testing for a libtorrent application that will use a small number of peers (often just 1 seed and 1 downloaded) and high bandwidth delay product links (in the hundreds of Mb/s and 100ms+ round trip). I’m unable to get more than 20Mbps with a single peer on a 140ms RTT link (simulated delay with no packet loss). If I take the network simulator delay down to 0, I can get almost a full 1000Mbps on the same system. I’ve tried playing with everything that sounded relevant in session_settings, but nothing seems to make any improvement at all.

Is this the best that can be expected out of uTP on a high latency link? Or is there some combination of parameters that would improve the single peer throughput? Alternatively, is there a way to get libtorrent to make multiple connections between the same 2 peers?

If you’re facing a similar situation here are some things you could adjust according to Arvid Norberg lead engineer of the libtorrent project.

“Did you increase the socket buffer sizes on both ends?”

int recv_socket_buffer_size;
int send_socket_buffer_size;

“There’s also buffer sizes at the bittorrent level:”

int send_buffer_low_watermark;
int send_buffer_watermark;
int send_buffer_watermark_factor;

“And there are buffers at the disk layer:”

int max_queued_disk_bytes;
int max_queued_disk_bytes_low_watermark;

If the performance issue happens with uTP but not TCP though, it’s probably
just the first ones that matters.

Also, the uTP implementation needs a system call for each packet sent and
received (i.e. it does not use sendmmsg()/recvmmsg()) which also makes it
more expensive than TCP, but that would primarily cause additional CPU
usage, and only slow-downs once a CPU core is pegged.

command line speed test, see how fast is your server’s connection

Save the following script in a file called speed_test

#!/bin/bash

# Requirements
# sudo apt-get install lftp iperf

lftp -e 'pget http://releases.ubuntu.com/14.04.3/ubuntu-14.04.3-desktop-amd64.iso; exit; '

make sure the file is executable: sudo chmod +x speed_test

Once you have installed lftp and iperf make sure you have the script somewhere in your $PATH.

The script basically downloads an ubuntu iso and does the math.

The output looks like this on a AWS m3.large instance:

$ speed_test
1054871586 bytes transferred in 14 seconds (70.37M/s)

Multiply by 8 to convert 70.37M/s to Megabits per second = 562.96 Mbit/s

AWS’s download speeds for m3.large instances is half a gigabit in January 2016. (or is that the upload speed of the Ubuntu ISO server?)

IntelliJ / Adroid Studio / CLion / GoLand / PyCharm / PHPStorm keymap shortcuts I can’t live without

Download settings.jar (Import with `File -> Import Settings`)

I got to IntelliJ after years of using Emacs and Eclipse. Here’s how my keymap looks now that I work with Jetbrains tools, these are the keyboard shortcuts I can’t live without as a programmer.

 

Legend (Mac keyboard)
⌘ Command
⇧ Shift

⌘1 Show Intent Actions (Things like assign to local variable)
⌘- Decrease font size
⌘= Increase font size
⌘0 Reset font size
βŒ˜β†“ Scroll to botom
βŒ˜β†‘ Scroll to top
⌘f Find in text
⌘. Find next highlighted error
^⇧F Find in path (searches in all files, like a grep)
βŒ˜β‡§G Find Usages (Find all Occurrences of selected symbol [variable, method, class, identifier])
⌘I Show current function/method parameter info
F3 Jump to source (go inside function’s code)
⌘E Show list of recent files
⌘B Build
⌘L Jump to line number
⌘[ Back
⌘] Forward
⌘T Show implementations of an interface.
βŒ˜β‡§F Format (selected) code
βŒ˜β‡§O Organize/Optimize imports
⌘βŒ₯R Rename current symbol
^K Kill line
⌘D Duplicate line
Ctrl+G on selected token – repeats selection and enables multiple cursors
Alt+Up | Alt+Down – Move the current line up or down

Debugger shortcuts
F5 Step Into
F6 Step Over
F7 Step Out
F9 Resume Program

^K Cut up to Line End. (Deletes everything to the right of where the text cursor or caret is)

How to create a list that holds different object types using `void*` in C.

I remember being in school back around 1998 and not knowing enough about C to do this. After coding in other languages, then going back to C++ and understanding at a lower level how references and pointers work, this was pretty easy to figure out.

In this exercise I store elements of different types in a forward linked list.
In order to know what to dereference as I iterate through the list’s elements, I’ve put a “.type” field, which has an int value representing the type of the object stored.

The “.value” is simply a void*, which lets me store a pointer of any kind, and it works pretty well.

Here’s the code for your enjoyment, I hope this is useful to C apprentices.

The example shows how you can store native types like int, or more complex char* or even a struct person* (which is the more useful probably to your purposes)

It’s a good exercise to see the uses of the “address of” operator “&”, which is used to initialize pointers (the ‘&’ can also be used differently to create references, which I call in my mind as ‘aliases’, but this is not shown in this example)

I also play with a not so popular syntax to access a pointer’s sub-fields:

(*myPointer).structField == myPointer->structField

to teach you that the -> is a short hand for dereferencing a pointer and accessing one of its fields.

//                                                                                                                                                                                          
// An exercise to play with a struct that stores anything using a void* field.                                                                                                              
//                                                                                                                                                                                          

#include <stdio.h>

#define TRUE 1

int TYPE_INT = 0;
int TYPE_STRING = 1;
int TYPE_BOOLEAN = 2;
int TYPE_PERSON = 3;

struct node {
  struct node* next;
  int type;
  void* value;
};

struct person {
  char* name;
  int age;
};

int main(int args, char **argv) {

  struct person aPerson;
  aPerson.name = "Angel";
  aPerson.age = 35;

  // Define a linked list of objects.                                                                                                                                                       
  // We use that .type field to know what we're dealing                                                                                                                                     
  // with on every iteration. On .value we store our values.                                                                                                                                
  struct node nodes[] = {
    { .next = &nodes[1], .type = TYPE_INT    , .value=1                   },
    { .next = &nodes[2], .type = TYPE_STRING , .value="anyfing, anyfing!" },
    { .next = &nodes[3], .type = TYPE_PERSON , .value=&aPerson            },
    { .next = NULL     , .type = TYPE_BOOLEAN, .value=TRUE                }
  };

  // We iterate through the list                                                                                                                                                            
  for ( struct node *currentNode = &nodes[0]; currentNode;  currentNode = currentNode->next) {
    int currentType = (*currentNode).type;
    if (currentType == TYPE_INT) {
      printf("%s: %dn", "- INTEGER", (*currentNode).value); // just playing with syntax, same as currentNode->value                                                                        
    } else if (currentType == TYPE_STRING) {
      printf("%s: %sn", "- STRING", currentNode->value);
    } else if (currentType == TYPE_BOOLEAN) {
      printf("%s: %dn", "- BOOLEAN (true:1, false:0)", currentNode->value);
    } else if (currentType == TYPE_PERSON) {
        // since we're using void*, we end up with a pointer to struct person, which we *dereference                                                                                        
        // into a struct in the stack.                                                                                                                                                      
        struct person currentPerson = *(struct person*) currentNode->value;
        printf("%s: %s (%d)n","- TYPE_PERSON", currentPerson.name, currentPerson.age);
      }
  }

    return 0;
}

The output is this:

- INTEGER: 1
- STRING: anyfing, anyfing!
- TYPE_PERSON: Angel (35)
- BOOLEAN (true:1, false:0): 1

How to make a “foreach” function in JavaScript

I thought this would be a simple exercise in case of having to interview someone for a JavaScript position.

“How would you make your own ‘foreach’ in JavaScript”

I came up with the following solution:

// 
// collection: A list of objects.
// onElementIterationCallback: The function to be called on every element iterated
//    taking the following parameters:    foo(collection : [T], currentIndex : int)
function foreach(collection, onElementIterationCallback) {
    for (var i in collection) {
        if (collection.hasOwnProperty(i)) {
            onElementIterationCallback(collection[i], i);
        }
    }
}

This is how you’d use it:

var sumOfAges = 0;
var people = [ {name:"Angel", age:35},
               {name:"Paulina", age:33},
               {name:"Nicole", age:16}]

foreach(people, function (person, currentOffset) {
   console.log("("+ currentOffset + ") iterating on " + 
               person.name + ", age: " + person.age);
   sumOfAges += person.age; 
});

console.log(sumOfAges);

The expected output would be:

(0) iterating on Angel, age: 35
(1) iterating on Paulina, age: 33
(2) iterating on Nicole, age: 16
84

Hope you enjoyed, just a simple exercise of lists, creativity and callbacks.

Manifest on Internet’s Evolution

(Forgotten draft from May 31st 2010)

I’m a little sad to see some of the paths the internet is taking lately. I personally think it’s enough about the infatuation that people have with “Social Networking”, enough, it’s boring and it makes us waste precious time as a race.

Technology is coming together for you to re-assess those technological ambitions that were not possible 10 years ago. Our networks are faster than ever, our home computers are several orders of magnitude more powerful, with multicore processing, gigs of RAM, powerful video cards, and what are we doing? Sending 140 chars at the time, and wasting time on Facebook a huge but simple friend database that can be resumed for external access to a simple 6 file “SDK”.

I write this (ongoing) manifest to inspire you to not waste your time on building the next social bullshit web “service”. I want you to think of yourself as the entrepreneur or scientist that truly changed the world for the better in a little way, not the one that made the entire world waste it’s time for you to make some bucks.

Everyone is in the Clouds, wake up, use those powerful CPUs
It’s like we forgot that we have powerful CPUs and hardware in our computers, like we forgot all the amazing technology we’ve created over the years and fell for the o-mighty-cloud. While at it we’re stuck with HTML5 and Javascript of all languages.

If we’re indeed going to rebuild and create new things for the browser, why the hell do we have to do it with such a limited set of technologies. Since I think we’ll be stuck in it, it’s time to create better IDEs and debugging tools for it, or start making those new browsers support the latest iterations in ECMA, to have proper Object Oriented support, a freaking binary primitive data type, threading and lower level I/O.

And why are we all buying this cloud crap, it’s like going back to mainframe/dummy terminal all over again, and paying for it. If things keep going this way, I might as well go back to a Pentium II with 64mb ram.

It’s about how we imagine the future
Think the future is not happening the way you thought it would be? Re-imagine, do something about it.

Stop cloning crap, build on top of it If you’re too late in the game
Competition is good, but you should recognize when you’re too late to enter the game unless you have something truly compelling and new to offer, on which case you might try to do business with the market leaders. For example, we don’t need another Twitter clone. Twitter was not the first on their space, Jaiku was there before (google even bought them) but they didn’t win. Remember what happened to Pwnce?

How to resize an EBS (xfs formatted) partition

First of all, create a snapshot of your EBS volume. Then out of that snapshot you will be able to create your new volume.

However, when you detach the old one from your instance and attach the new one, you will still see the old available space with df

look at my /dev/xvdf/ available space (after mounting the new EBS volume)

ubuntu@ip-10-47-167-74:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1       16G  5.9G  9.1G  40% /
udev            7.4G   12K  7.4G   1% /dev
tmpfs           1.5G  176K  1.5G   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            7.4G     0  7.4G   0% /run/shm
/dev/xvdf        20G   19G  1.7G  92% /media/ebs/data  <<< this one

still 20G, I mounted a 80G one!

This drive has been formatted to use an xfs file system. In order to resize it, this is the command I used:

sudo xfs_growfs -d /media/ebs/data

output should be something like this

ubuntu@ip-10-47-167-74:~$ sudo xfs_growfs -d /media/ebs/data
meta-data=/dev/xvdf              isize=256    agcount=4, agsize=1310720 blks
         =                       sectsz=512   attr=2
data     =                       bsize=4096   blocks=5242880, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0
log      =internal               bsize=4096   blocks=2560, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0
data blocks changed from 5242880 to 20971520

now let’s see the df -h output

ubuntu@ip-10-47-167-74:~$ df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1       16G  5.9G  9.1G  40% /
udev            7.4G   12K  7.4G   1% /dev
tmpfs           1.5G  176K  1.5G   1% /run
none            5.0M     0  5.0M   0% /run/lock
none            7.4G     0  7.4G   0% /run/shm
/dev/xvdf        80G   19G   62G  23% /media/ebs/data