How to build your Docker image using the same Dockerfile regardless of the host architecture

Problem

If you are now using docker on a Mac M1 (arm64 platform), you don’t want to use amd64 as the architecture for your Linux Images.

You could have 2 lines on your Dockerfile and comment each one depending on where you’re building the image

Dockerfile

# Building on Apple Silicon host
FROM --platform=linux/arm64 ubuntu:20.04

# Building on Intel/x86_64 host
#FROM --platform=linux/amd64 ubuntu:20.04

Eventually this becomes very annoying.

Solution

You can pass a build time argument when you invoke docker build

Put this on your Dockerfile:

ARG BUILDPLATFORM
FROM --platform=linux/$BUILDPLATFORM ubuntu:20.04

On your docker_build_image.sh script:

export BUILDPLATFORM=`uname -m`
docker build --build-arg BUILDPLATFORM=${BUILDPLATFORM} -t myimagename .

How to enable source highlighting when doing `less mycodefile.ext`

How to enable source highlighting when doing less mycodefile.ext

  1. Install source-highlight
    sudo apt install source-highlight

  2. Configure it on your .bash_profile

lessWithSourceHighlightSetup() {
  # location of the script may vary
  src_hilite_pipe_script=`dpkg -L libsource-highlight-common | grep lesspipe`
  export LESSOPEN="| ${src_hilite_pipe_script} %s"
  export LESS=' -R '
}


lessWithSourceHighlightSetup
  1. Use it on any code file less -N /path/to/mycode.ext

Screen Shot 2019-11-12 at 9 55 32 AM

[linux/ubuntu] How to suppress useless mod_openssl/lighttpd error messages from appearing in /var/log/syslog

Sometimes you have a bunch of useless errors creating unnecessary disk I/O on your server, disk I/O that should be used towards serving your user’s requests efficiently.

In this case a site running on lighttpd keeps logging several times per second the following message, creating too much noise and making it hard to see meaningful things I should pay attention to could appear on /var/log/syslog.

Aug  7 19:36:03 ip-172-30-1-251 lighttpd[287019]: message repeated 44 times: [ 2020-08-07 19:36:02: (mod_openssl.c.1796) SSL: 1 error:14209102:SSL routines:tls_early_post_process_client_hello:unsupported protocol]

I tried disabling syslog error messages for SSL, and all syslog output on the lighttpd configuration to no avail. Good thing you can configure rsyslog in Linux to do amazing things with log messages before they make it into the log.

To silence this message, all I had to do was edit an rsyslog config file to filter out my undesired message, and restart the service (no need to restart your host os)

  1. Edited /etc/rsyslog.d/50-default.conf before any mention of /var/log/syslog, to have the following condition (ideally at the top of the config file):

[pastacode lang=”bash” manual=”if%20%24msg%20contains%20’tls_early_post_process_client_hello’%20then%20stop” message=”” highlight=”” provider=”manual”/]

  1. Restarted the rsyslog service, no more noise on /var/log/syslog
     sudo service rsyslog restart

nginx server configuration for a wordpress instance served from a URL’s subdirectory

You want to serve a wordpress instance on a website’s domain url but not at the path’s root, you want it under a sub-directory, for example “blog”, the same as this blog:

https://www.gubatron.com/blog 

Here’s how my NGINX’s server block for ‘www.gubatron.com’ looks like at the moment (https/ssl hasn’t been configured yet)

[pastacode lang=”javascript” path_id=”fc6f967d7689931c81e9e9691b714f29″ file=”” highlight=”” lines=”” provider=”gist”/]

Here is the equivalent in lighttpd, too bad lighttpd has no plans for HTTP2, it’s much friendlier and flexible to configure than nginx in my humble opinion.

[pastacode lang=”javascript” path_id=”d6e2f73026cf61ac74e1d45ce8a401b5″ file=”” highlight=”” lines=”” provider=”gist”/]

I used to host this website and wordpress on lighttpd, lighttpd’s config file is very powerful, it’s all based on matching server variables and applying rules, I will miss it dearly, things like having a compressed file cache and it’s flexibility, but I have to move on to nginx if I want to use http2, the lighttpd has no plans for http2 support and it’s just much faster and efficient than http 1.1.

Fix high CPU usage by WordPress and MySQL

Today one of our wordpress sites had very high server load and it was being caused by MySQL

So I went to the mysql console, and looked up the process list:

So this guy is appearing a lot
SELECT option_name, option_value FROM wp_options WHERE autoload = 'yes';

Let’s see how it’s behaving with explain
explain SELECT option_name, option_value FROM wp_options WHERE autoload = 'yes';

It’s scanning 226k rows to get its search results!

Probably some moronic plugin is doing this and wordpress does not add an index on that table. The solution is simple, let’s add an index!

ALTER TABLE wp_options ADD INDEX (`autoload`);

Now let’s run explain again

From scanning 226k it went down to 408!, 3 orders of magnitude drop.

And now the CPU load went below 4%, crisis averted.

HOW TO ENABLE PHP FPM LOG OUTPUT

This one had me for the longest time.

If you happen to be running a web server with php-fpm, sometimes you will run across an HTTP 500 error and all you will get will be a blank screen.

You will look at your server’s error log, your vhost error’s log and you will see nothing.

At this point you will want to enable logging on php fpm to see what’s up.

So you will go to your /etc/php/7.0/fpm/php-fpm.conf

you will start searching for “log”, and you will come across
error_log = /var/log/php7.0-fpm.log
you will tail -f that log file, and nothing will come up.

you will go back to that config file, you will play with your log levels, and nothing, and that’s because there’s this fucking obscure setting on your pool configuration that you’d never think of.

Let’s say you’re using the default www.conf pool config file (the one sitting at /etc/php/7.0/fpm/pool.d/www.conf), open it and look for “workers”, you will see this:

; Redirect worker stdout and stderr into main error log. If not set, stdout and
; stderr will be redirected to /dev/null according to FastCGI specs.
; Note: on highloaded environement, this can cause some delay in the page
; process time (several ms).
; Default Value: no
;catch_workers_output = yes

uncomment catch_workers_output=yes
and restart your php-fpm service, tail -f your log and you will see the stack trace you’re looking for.

You’re welcome.

Upgrading your wordpress blog to PHP 7.0 on Ubuntu Xenial

If you’re about to upgrade your Ubuntu server to 16.04 (Xenial) you might want to take advantage of the new PHP 7.0 which is as fast or more than Facebook’s HHVM, or perhaps during the upgrade process a few things may have broken and perhaps that’s why you’re here

Make sure the following packages are installed.

sudo apt install php7.0-cli php7.0-common php7.0-curl php7.0-fpm php7.0-json php7.0-readline php7.0-mbstring php7.0-xml php7.0-mysql

Update your php-fpm web server configuration

I run lighttpd, but you’re more likely running nginx or apache.
If you use php-fpm and you’ve configured your pool to be accessed via unix socket, you will have to update your server configuration from the old socket path: "/var/run/php5-fpm.sock"
to the new one "/var/run/php/php7.0-fpm.sock"

this is how it looks for lighttpd:

fastcgi.server = ( ".php" =>
(( "socket" => "/var/run/php/php7.0-fpm.sock",
"broken-scriptfilename" => "enable",
"allow-x-send-file" => "enable")))

[SYSADMIN] Serve your WordPress cached pages directly with lighttpd and not PHP

Optimizing Your WordPress Cache Loads in Lighttpd.

If you don’t configure your wordpress virtual host properly in lighttpd, your wordpress cache will still make use of PHP.

Wouldn’t it be nice if all those cached requests were served directy from the webserver as the static files that they are, bypassing the CPU/memory load PHP can have, and use those resources for otherthings?

Install and Enable mod_magnet

For this to occur with lighttpd, you will need mod_magnet, so assuming you’re on a Ubuntu/Debian based linux distro let’s make sure we have it installed.

sudo apt-get install lighttpd-mod-magnet

Then let’s make sure it’s enabled, you can do this manually on your lighttpd.conf by adding “mod_magnet” to the list of enabled modules…

server.modules = (
        "mod_fastcgi",
        "mod_access",
        "mod_alias",
        "mod_accesslog",
        "mod_compress",
        "mod_rewrite",
        "mod_redirect",
        "mod_status",
        "mod_proxy",
        "mod_setenv",
        "mod_magnet"
)

or you can do it the lighty way:

sudo lighty-enable-mod magnet

(this simply makes a symlink to the 10-magnet.conf file inside /etc/lighttpd/conf-enabled which lighty will check upon startup)

The cache logic script that will be executed by lighttpd

Now, on your wordpress directory, create a file called rewrite.lua and paste the following script in it:

function log(str)
   -- wanna tail -f a log to see what's happening    
   fp = io.open("/path/to/some/lua.log","a+")
   fp:write(str .. "\n")
   fp:flush()
   fp:close()
end

function serve_html(cached_page)
    if (lighty.stat(cached_page)) then
        lighty.env["physical.path"] = cached_page
        return true
    else
        return false
    end
end

function serve_gzip(cached_page)
    if (lighty.stat(cached_page .. ".gz")) then
        lighty.header["Content-Encoding"] = "gzip"
        lighty.header["Content-Type"] = ""
        lighty.env["physical.path"] = cached_page .. ".gz"
        return true
    else
        return false
    end
end

if (lighty.env["uri.scheme"] == "http") then
    ext = ".html"
else
    ext = "-https.html"
end

cached_page = lighty.env["physical.doc-root"] .. "/wp-content/cache/supercache/" .. lighty.request["Host"] .. lighty.env["request.orig-uri"]
cached_page = string.gsub(cached_page, "//", "/")
cached_page = string.gsub(cached_page, lighty.request["Host"] .. "/index.php", lighty.request["Host"])

attr = lighty.stat(cached_page)

if (attr) then
    query_condition = not (lighty.env["uri.query"] and string.find(lighty.env["uri.query"], ".*s=.*"))
    user_cookie = lighty.request["Cookie"] or "no_cookie_here"
    cookie_condition = not (string.find(user_cookie, ".*comment_author.*") or (string.find(user_cookie, ".*wordpress.*") and not string.find(user_cookie,"wordpress_test_cookie")))

    if (query_condition and cookie_condition) then
        accept_encoding = lighty.request["Accept-Encoding"] or "no_acceptance"

        if (string.find(accept_encoding, "gzip")) then
            if not serve_gzip(cached_page) then 
                serve_html(cached_page) 
            end
        else
            serve_html(cached_page)
        end
        --log('cache-hit: ' .. cached_page)
    end
else
    --log('cache-miss: ' .. cached_page)
end

Configuring your vhost in lighttpd for WordPress redirects and direct cache serves without php.

Then on your vhost configuration in lighttpd.conf add the following towards the end.
(Fix paths if you have to)

var.wp_blog = 1

magnet.attract-physical-path-to = ( server.document-root + "/rewrite.lua" )

url.rewrite-if-not-file = (
   "^/(wp-.+).*/?" => "$0",
   "^/(sitemap.xml)" => "$0",
   "^/(xmlrpc.php)" => "$0",
   "^/(.+)/?$" => "/index.php/$1"
  )

Restart your lighttpd sudo service lighttpd restart

Now watch how your PHP processes breathe a lot better and you page loads are insanely faster.

You’re welcome 🙂

command line speed test, see how fast is your server’s connection

Save the following script in a file called speed_test

#!/bin/bash

# Requirements
# sudo apt-get install lftp iperf

lftp -e 'pget http://releases.ubuntu.com/14.04.3/ubuntu-14.04.3-desktop-amd64.iso; exit; '

make sure the file is executable: sudo chmod +x speed_test

Once you have installed lftp and iperf make sure you have the script somewhere in your $PATH.

The script basically downloads an ubuntu iso and does the math.

The output looks like this on a AWS m3.large instance:

$ speed_test
1054871586 bytes transferred in 14 seconds (70.37M/s)

Multiply by 8 to convert 70.37M/s to Megabits per second = 562.96 Mbit/s

AWS’s download speeds for m3.large instances is half a gigabit in January 2016. (or is that the upload speed of the Ubuntu ISO server?)