[SYSADMIN] Serve your WordPress cached pages directly with lighttpd and not PHP

Optimizing Your WordPress Cache Loads in Lighttpd.

If you don’t configure your wordpress virtual host properly in lighttpd, your wordpress cache will still make use of PHP.

Wouldn’t it be nice if all those cached requests were served directy from the webserver as the static files that they are, bypassing the CPU/memory load PHP can have, and use those resources for otherthings?

Install and Enable mod_magnet

For this to occur with lighttpd, you will need mod_magnet, so assuming you’re on a Ubuntu/Debian based linux distro let’s make sure we have it installed.

sudo apt-get install lighttpd-mod-magnet

Then let’s make sure it’s enabled, you can do this manually on your lighttpd.conf by adding “mod_magnet” to the list of enabled modules…

server.modules = (
        "mod_fastcgi",
        "mod_access",
        "mod_alias",
        "mod_accesslog",
        "mod_compress",
        "mod_rewrite",
        "mod_redirect",
        "mod_status",
        "mod_proxy",
        "mod_setenv",
        "mod_magnet"
)

or you can do it the lighty way:

sudo lighty-enable-mod magnet

(this simply makes a symlink to the 10-magnet.conf file inside /etc/lighttpd/conf-enabled which lighty will check upon startup)

The cache logic script that will be executed by lighttpd

Now, on your wordpress directory, create a file called rewrite.lua and paste the following script in it:

function log(str)
   -- wanna tail -f a log to see what's happening    
   fp = io.open("/path/to/some/lua.log","a+")
   fp:write(str .. "\n")
   fp:flush()
   fp:close()
end

function serve_html(cached_page)
    if (lighty.stat(cached_page)) then
        lighty.env["physical.path"] = cached_page
        return true
    else
        return false
    end
end

function serve_gzip(cached_page)
    if (lighty.stat(cached_page .. ".gz")) then
        lighty.header["Content-Encoding"] = "gzip"
        lighty.header["Content-Type"] = ""
        lighty.env["physical.path"] = cached_page .. ".gz"
        return true
    else
        return false
    end
end

if (lighty.env["uri.scheme"] == "http") then
    ext = ".html"
else
    ext = "-https.html"
end

cached_page = lighty.env["physical.doc-root"] .. "/wp-content/cache/supercache/" .. lighty.request["Host"] .. lighty.env["request.orig-uri"]
cached_page = string.gsub(cached_page, "//", "/")
cached_page = string.gsub(cached_page, lighty.request["Host"] .. "/index.php", lighty.request["Host"])

attr = lighty.stat(cached_page)

if (attr) then
    query_condition = not (lighty.env["uri.query"] and string.find(lighty.env["uri.query"], ".*s=.*"))
    user_cookie = lighty.request["Cookie"] or "no_cookie_here"
    cookie_condition = not (string.find(user_cookie, ".*comment_author.*") or (string.find(user_cookie, ".*wordpress.*") and not string.find(user_cookie,"wordpress_test_cookie")))

    if (query_condition and cookie_condition) then
        accept_encoding = lighty.request["Accept-Encoding"] or "no_acceptance"

        if (string.find(accept_encoding, "gzip")) then
            if not serve_gzip(cached_page) then 
                serve_html(cached_page) 
            end
        else
            serve_html(cached_page)
        end
        --log('cache-hit: ' .. cached_page)
    end
else
    --log('cache-miss: ' .. cached_page)
end

Configuring your vhost in lighttpd for WordPress redirects and direct cache serves without php.

Then on your vhost configuration in lighttpd.conf add the following towards the end.
(Fix paths if you have to)

var.wp_blog = 1

magnet.attract-physical-path-to = ( server.document-root + "/rewrite.lua" )

url.rewrite-if-not-file = (
   "^/(wp-.+).*/?" => "$0",
   "^/(sitemap.xml)" => "$0",
   "^/(xmlrpc.php)" => "$0",
   "^/(.+)/?$" => "/index.php/$1"
  )

Restart your lighttpd sudo service lighttpd restart

Now watch how your PHP processes breathe a lot better and you page loads are insanely faster.

You’re welcome 🙂

command line speed test, see how fast is your server’s connection

Save the following script in a file called speed_test

#!/bin/bash

# Requirements
# sudo apt-get install lftp iperf

lftp -e 'pget http://releases.ubuntu.com/14.04.3/ubuntu-14.04.3-desktop-amd64.iso; exit; '

make sure the file is executable: sudo chmod +x speed_test

Once you have installed lftp and iperf make sure you have the script somewhere in your $PATH.

The script basically downloads an ubuntu iso and does the math.

The output looks like this on a AWS m3.large instance:

$ speed_test
1054871586 bytes transferred in 14 seconds (70.37M/s)

Multiply by 8 to convert 70.37M/s to Megabits per second = 562.96 Mbit/s

AWS’s download speeds for m3.large instances is half a gigabit in January 2016. (or is that the upload speed of the Ubuntu ISO server?)

How to create a list that holds different object types using `void*` in C.

I remember being in school back around 1998 and not knowing enough about C to do this. After coding in other languages, then going back to C++ and understanding at a lower level how references and pointers work, this was pretty easy to figure out.

In this exercise I store elements of different types in a forward linked list.
In order to know what to dereference as I iterate through the list’s elements, I’ve put a “.type” field, which has an int value representing the type of the object stored.

The “.value” is simply a void*, which lets me store a pointer of any kind, and it works pretty well.

Here’s the code for your enjoyment, I hope this is useful to C apprentices.

The example shows how you can store native types like int, or more complex char* or even a struct person* (which is the more useful probably to your purposes)

It’s a good exercise to see the uses of the “address of” operator “&”, which is used to initialize pointers (the ‘&’ can also be used differently to create references, which I call in my mind as ‘aliases’, but this is not shown in this example)

I also play with a not so popular syntax to access a pointer’s sub-fields:

(*myPointer).structField == myPointer->structField

to teach you that the -> is a short hand for dereferencing a pointer and accessing one of its fields.

//                                                                                                                                                                                          
// An exercise to play with a struct that stores anything using a void* field.                                                                                                              
//                                                                                                                                                                                          

#include <stdio.h>

#define TRUE 1

int TYPE_INT = 0;
int TYPE_STRING = 1;
int TYPE_BOOLEAN = 2;
int TYPE_PERSON = 3;

struct node {
  struct node* next;
  int type;
  void* value;
};

struct person {
  char* name;
  int age;
};

int main(int args, char **argv) {

  struct person aPerson;
  aPerson.name = "Angel";
  aPerson.age = 35;

  // Define a linked list of objects.                                                                                                                                                       
  // We use that .type field to know what we're dealing                                                                                                                                     
  // with on every iteration. On .value we store our values.                                                                                                                                
  struct node nodes[] = {
    { .next = &nodes[1], .type = TYPE_INT    , .value=1                   },
    { .next = &nodes[2], .type = TYPE_STRING , .value="anyfing, anyfing!" },
    { .next = &nodes[3], .type = TYPE_PERSON , .value=&aPerson            },
    { .next = NULL     , .type = TYPE_BOOLEAN, .value=TRUE                }
  };

  // We iterate through the list                                                                                                                                                            
  for ( struct node *currentNode = &nodes[0]; currentNode;  currentNode = currentNode->next) {
    int currentType = (*currentNode).type;
    if (currentType == TYPE_INT) {
      printf("%s: %dn", "- INTEGER", (*currentNode).value); // just playing with syntax, same as currentNode->value                                                                        
    } else if (currentType == TYPE_STRING) {
      printf("%s: %sn", "- STRING", currentNode->value);
    } else if (currentType == TYPE_BOOLEAN) {
      printf("%s: %dn", "- BOOLEAN (true:1, false:0)", currentNode->value);
    } else if (currentType == TYPE_PERSON) {
        // since we're using void*, we end up with a pointer to struct person, which we *dereference                                                                                        
        // into a struct in the stack.                                                                                                                                                      
        struct person currentPerson = *(struct person*) currentNode->value;
        printf("%s: %s (%d)n","- TYPE_PERSON", currentPerson.name, currentPerson.age);
      }
  }

    return 0;
}

The output is this:

- INTEGER: 1
- STRING: anyfing, anyfing!
- TYPE_PERSON: Angel (35)
- BOOLEAN (true:1, false:0): 1

How to make a “foreach” function in JavaScript

I thought this would be a simple exercise in case of having to interview someone for a JavaScript position.

“How would you make your own ‘foreach’ in JavaScript”

I came up with the following solution:

// 
// collection: A list of objects.
// onElementIterationCallback: The function to be called on every element iterated
//    taking the following parameters:    foo(collection : [T], currentIndex : int)
function foreach(collection, onElementIterationCallback) {
    for (var i in collection) {
        if (collection.hasOwnProperty(i)) {
            onElementIterationCallback(collection[i], i);
        }
    }
}

This is how you’d use it:

var sumOfAges = 0;
var people = [ {name:"Angel", age:35},
               {name:"Paulina", age:33},
               {name:"Nicole", age:16}]

foreach(people, function (person, currentOffset) {
   console.log("("+ currentOffset + ") iterating on " + 
               person.name + ", age: " + person.age);
   sumOfAges += person.age; 
});

console.log(sumOfAges);

The expected output would be:

(0) iterating on Angel, age: 35
(1) iterating on Paulina, age: 33
(2) iterating on Nicole, age: 16
84

Hope you enjoyed, just a simple exercise of lists, creativity and callbacks.

First Impressions Lenovo Yoga Pro and Windows 8.1 with a 2 in 1 device.

After now almost 2 weeks of heavy duty use of this machine, I must say Windows 8.1 is not fully baked when it comes to its tiled/touch/app store experience, however it’s not the nightmare I expected.

The FrostWire 5.7.0. currently circulating on the Internet was built on this machine, the experience was quite pleasant as a workstation, and then I carried it around for the North American Bittorrent conference and during that time it was a very convenient tablet while I was doing all the social networking during the event.

WordPress: Cannot create directory error, got everything chmoded to 777?

So you went as far as chmod’ing your entire folder structure to 777 (very unsafe), you’ve hacked wp/wp-admin/includes/file.php

return new WP_Error( 'mkdir_failed_ziparchive', __( get_current_user() . ' Could not create directory. ('.$_dir.')' )...

to print out exactly what directory it cannot create and what user is trying to create the folder, everything is correct, yet it won’t create the fucking folder?

the issue might be your vsftpd configuration!

Go to /etc/vsftpd.con and make sure that you have this setting uncommented:

write_enable=YES

restart your vsftpd (sudo service vsftpd restart) and try upgrading again, you’ll want to hit that ‘Donate Bitcoin’ button if I saved your ass today.

Cheers

Metrics Gathering notes from Workshop by Aaron Ginn

Aaron Ginn from StumbleUpon.com
October 10th, 2013 @ Miami Science Museum / Refresh Miami

Most important metrics?
Number of installations.
Revenues.

Types of metrics:
– Quantitatie vs -Qualitative.

Idea:
– On Wizard, let them know how to be in touch for Tech Support
via @frostwire and facebook.com/FrostWire Official.
Task for Marcelina to design that screen on the wizard.

QUANTITATIVE
The funnel (Pirate Metric)
Acquisition > Activations > Retention > Referral > Revenue.

QUALITATIVE
User Testing (In Progress)
Surveys
Study Groups
User Interviews

Stages for companies:

Early Stage involves discovery, validation. Then get Seeded out of pocket and angels.
Qualitative metrics help here.

Later Stage creation and building – Series A, profitability.
Quantitative metrics.

During creation, you want to start doing A/B to grow and see what works.

PRODUCT DASHBOARD
Engagement
User’s worth (LTV)
Growth rate

Qualities of a Good Metric
Measurements that are aggressively attacking your product.

Actionable – short feedback loop.
Product Centric – attack core assumptions of your product
Ratio – Compare against competitors performance, industry numbers.
Explainable – simpler the better
Timeless
(Checkout “Flurry” reports

Key Metrics per business

Marketplace: fulfillment rate, WOW (week over week) growth on both sides, % repeat purchase

Social Product: DAU/MAU, WOW daily registration, % ad impressions

B2B – sales funnel metrics, LTV

B2C – sales funnel metrics, LTV, WOW growth

LTV = churn * avg revenue per customer

Virality = measure the share cycle time is important. The faster, the deeper impact at sub-levels. Time between invites and activations.
Look for vitality equations.

Create your own AARRR

Acquisition – Number of new downloads, newsletter signups, twitter followers, Facebook page likes, community forum signups, Number of displays of our thank you page
Activation – Number of first uses. Displays of our promotions.
Retention – Number of sessions, Number of core actions, Newsletter unsubscribes, Facebook unlikes
Referral – Measure social activity, buzz, cult (pictures, fan art)
Revenue – Daily reports from all revenue channels (some hourly)

Qualitative Testing Schedule

One user a day. Watch them.
Two user sessions a month.
Three usability studies a month.

Important Factors in Quantitative Testing
is it Statistically Significant?
sample size.
sensitivity of the test.
duration of the test.
velocity. How many tests can you run in a period, pretty important. You need to test and get results as fast as possible, to learn more and more.
conversion.

Good habits
Test the hell out of MVPs.
Kill tech that slows you down.
Validate Instrumentation First. (make sure your test is running well)
The Goal is testing velocity.
Think about your next developer/employee.

Tools we use
Qualitative Testing Tools:
FT User Researcher
UserTesting.com
MouseOver Videos
User Studies

Quantitative Testing Tools
AB testing
Daily metrics emails
Product dashboards
Team metrics

Rapid A/B Testing Workflow

Test Iteration Implementations -> Rapid implementations.
Full Implementations -> Winner only (clean losers)
Code Fragmentation -> Clean unused test code.

Branch Based Workflow (git based)

Test Iteration Implementation -> Rapid
Full Impl -> Winner only

What will we do differently after this?
Added some new metrics I was missing thanks to this in relationship to acquisition and activations.
Will try new tools like UserTesting.com
Will look at what tech might be slowing us down.
Will not build more visualization tools of test data.

How can we all own a robotic workforce?

As people become useless, how can capitalism survive?

As technology advances exponentially the fields of robotics and artificial intelligence will start taking over our jobs. More and more people already can’t find work, yet the nation is still very productive.
Yes, a lot in part due to globalization and cheaper foreign labor, but those cheap labor countries will only remain competitive against fully automated factories for so long.

Self driving vehicles will eventually be here, it’s just a matter of time before we think of driving ourselves as the most barbaric thing we ever did, our kids won’t believe we actually drove ourselves.

Self driving vehicles will take the jobs of millions of people, think bus drivers, truck drivers, taxi drivers, and these robotic vehicles will do their job infinitely better, not getting tired and for way less money.

Think Fedex, UPS, USPS announcing their first robotic delivery fleets, an then in a matter of years they start reducing their costs exponentially by letting go of +90% of their human workforce, shipping worldwide becomes faster and cheaper than ever.

If things keep going the way they are as they happen on the internet, only a handful of people will own powerful computers, shrinking economies due to technological efficiency and more wealth will be concentrated in a few.

If the same thing happens to robotized labor, we’ll find ourselves soon in civil war with so many “useless” people unemployed.

But I think there’s another end to it, while keeping capitalism.

If we are all smart about it, as we see robotization happening, we must make the decision to change a few things, without turning completely into a socialist country which in the case of latin america’s implementation of socialism, the result is usually an all too powerful government and oppressed citizens.

I believe that instead of handouts (like having a government that provides everything to people who are just useless thanks to robots being so useful) we should look at this as an opportunity to make everyone in society a stake holder of the robotic workforce.

If things would happen the right way, robots would be doing our job and we’d be getting paid for it while we play outside.

The only way this could happen would be to distribute the ownership of all the robots to as many citizens as possible. The implementation of such distributed ownership is the part that I still don’t have quite clear and I’d love for you to give me some ideas on how this could come true.

Perhaps it could all start with a law to avoid the ultimate inequality gap, as things are today the rich will be the only ones to own the majority of robots, ending up with a robotic workforce that produces goods that nobody will be able to afford since they’re all unemployed, then people would start talking about a communist revolution and we don’t want that, we need capitalism evolved. A nation made from a lot of rich people out of robotic production and a renassaince of American product exports.

I think if the people now living off of unemployment checks would own a portion of the robotic workforce, they would instead get larger and larger divident checks that come off their robot’s productivity (I see this as some sort of Robotic ETF that pays monthly dividends).

This ownership could initially come from a portion of their tax contributions/returns (through some initial law that would make as many americans as possible robotic-workforce-entrepreneurs by force), maybe there could be incentives toward robotic equity, such as people who continue to get a high level education getting tax credits or rebates on the price of the robotic labor ETF.

The first country to figure this out will be king, itwould have a society mainly composed of a highly educated and almost rich middle class, people that don’t have to spend 8 hours a day doing work that’s done 1000 times better by their robotic counterparts.

A society that would have a surplus of money they could use to travel the world, study, research, or invest in whatever they please, being highly educated an peaceful we’d find times to take mankind to far more interesting directions like clean and free energy research, space exploration and mining, medical advances, the pursuit of much longer life spans, and solving real problems affecting developing nations.

But the question remains, how do you motivate the currently rich elite to not eat/cook the whole robotic labor pie, or to somehow share it so that we all get to own a piece of it and live worry free, happily unemployed because robots are the ones that are supposed to work for us?