EBook Pricing Failure

14-04-2016

This is insane. I'd quite like to read the new Mitch Rapp novel, (RIP, Vince Flynn). I generally go for the Kindle edition because despite the limited flexibility:

  • "90% finished" but the book has ended. What?
  • Difficult/impossible to lend
  • What happens to it when I die?!

There are a few features that make up for it:

  • The Kindle app is always in my pocket
  • Text search
  • Lower price

I'll stop you there.

With zero marginal cost, zero lead time, no dependency on mass print-runs and the oh so awesome ability to enforce repressive DRM to forbid me to read it in a way you don't like, I have to ask why certain publishers insist on pricing the Kindle Edition higher than the print editions.

Luckily there are many different options. As Tim Worstall eloquently puts it,

A really good play with oneself is just as much something one can do with one's leisure hours as reading an overly expensive book

Likewise, I could borrow the book, buy a different book or if so inclined, find the book on a less than 100% legitimate site and additionally benefit from no DRM.

So there you go, Simon & Schuster, I'll paraphrase Tim Worstall and perhaps somewhat less eloquently suggest that you go fuck yourself.

PHP profiling saved one of my meagre website's users 13 hours per month

26-09-2014

That surprised me too. But at 100,000 pageviews per month, and looking at an average load time of 0.5s before and 0.03s after my two minor optimisations, I make it just over 13 hours. What happens to that extra time now? Do I get more pageviews or do Facebook and Twitter see the majority of the benefit?

Skip to the howto

Before finding where my code was slow I didn't know where to look. Now I don't need to guess. Over 3 seconds crunching MySQL:

After a few minutes I'd to cut out a load of calls and optimise the ones I did still need:

What is xhprof?

The xhprof PHP extension itself was developed and open-sourced by Facebook a few years ago. It analyzes your PHP as it runs and reports on time taken to run each function. It's within the PECL repository so in theory it's a straight foward install. The front end I used is a slightly modified version by Paul Reinheimer (GitHub).

How to install xhprof on a standard LAMP system

This assumes you've got a standard CentOS installation with Apache, MySQL and PHP already installed. Take a look over here to see how to do this.

There's not much to the install that Google can't help with. I had an error when I tried "pecl install xhprof" but this worked:

yum install php-devel php-pear
pecl install channel://pecl.php.net/xhprof-0.9.4

Enable the extension within php.ini, or more commonly in /etc/php.d/*.ini

cat >/etc/php.d/xhprof.ini <<EOF
[xhprof]
extension=xhprof.so
xhprof.output_dir="/tmp/xhprof"
EOF

Instaling the GUI front-end is a slightly more involved: Downloading some PHP; modify a config file; create a database and table.

You can either put the code in an existing DocumentRoot or create a new vhost and give it its own subdomain. I've explaind the new vhost option but to simplfy you could unzip it straight into /var/www/html (or wherever your existing DocumentRoot is) and access using your existing domain: http://example.com/xhprof-master.

curl https://github.com/preinheimer/xhprof/archive/master.zip -L -o /tmp/xhprof.zip
cd /var/www
unzip /tmp/xhprof.zip

Create and modify the config file to reflect your local system. Make a new database user.

cp /var/www/xhprof-master/xhprof-lib/config.sample.php /var/www/xhprof-master/xhprof-lib/config.php

Modify the new config.php with Vim, Emacs, gedit or your editor of choice. Make sure to add database and server details, and adding your IP address to the $controlIPs array. If you're not sure, click here to find your IP.

Create a vhost for this site:

cat >//etc/httpd/conf.d/vhost_xhprof.conf <<EOF
<VirtualHost 203.0.113.123:80>
        ServerName profiler.example.com
        DocumentRoot /var/www/xhprof
</VirtualHost>
EOF

Optionally, to automatically run the profiling tool include this within the relevant vhost in your httpd config file.

php_admin_value auto_prepend_file "/var/www/xhprof-master/external/header.php"

That's about it for server setup so reload Apache's config files:

service httpd reload

Create a username, database and table for your data logging:

mysql -u root -p
mysql>CREATE DATABASE xhprof;
mysql>use xhprof;
mysql>GRANT ALL PRIVILEGES ON xhprof.* To 'xhprof'@'127.0.0.1' IDENTIFIED BY 'password' WITH GRANT OPTION;
mysql>CREATE TABLE `details` (
 `id` char(17) NOT NULL,
 `url` varchar(255) default NULL,
 `c_url` varchar(255) default NULL,
 `timestamp` timestamp NOT NULL default CURRENT_TIMESTAMP on update CURRENT_TIMESTAMP,
 `server name` varchar(64) default NULL,
 `perfdata` MEDIUMBLOB,
 `type` tinyint(4) default NULL,
 `cookie` BLOB,
 `post` BLOB,
 `get` BLOB,
 `pmu` int(11) unsigned default NULL,
 `wt` int(11) unsigned default NULL,
 `cpu` int(11) unsigned default NULL,
 `server_id` char(3) NOT NULL default 't11',
 `aggregateCalls_include` varchar(255) DEFAULT NULL,
 PRIMARY KEY  (`id`),
 KEY `url` (`url`),
 KEY `c_url` (`c_url`),
 KEY `cpu` (`cpu`),
 KEY `wt` (`wt`),
 KEY `pmu` (`pmu`),
 KEY `timestamp` (`timestamp`)
 ) ENGINE=MyISAM DEFAULT CHARSET=utf8;

And finally, either manually include header.php or, using the auto_prepend_file directive above, load a page and checkout the profile. By default a page will be randomly profiled 1 in 100 page-loads but you can explicitly request it by appending ?_profile=1 to your url. Eg: http://example.com/page?_profile=1. Once the page has loaded you'll see an extra link at the bottom of the page called "Profiler output" Follow this to see the profile. (This won't work if your site is beind a CDN so you might need to modify xhprof/external/header.php)

The next bit might require some real effort. Take a look at the profile and find the slow bits. Then make them better!

Have I missed anything? Let me know!

How to backup your cloud server to Amazon S3

15-08-2014

By Rob Stevenson

Following on from how to setup a web server on ElasticHosts, this covers backing up your server. It's not quite as ElasticHosts-centric and should work on most linux environments, virtual or real, with very little modification.

Amazon Web Services

You'll need to sign up fist. Go to aws.amazon.com and click Sign Up. I'll leave it there because the official documentation covers it pretty well. You'll need to Sign up for AWS and creat an IAM user. Make a note of the Access Key ID and Secret Access Key as you'll need these later to connect and copy files.

Once you're setup on AWS, head to the S3 section and create a bucket. Give it a meaningful name (eg myserverbackup) and pick a geographical region based on pricing and proximity to your existing servers.

Now head back to the User Identity and Access Management (IAM) area and find your user. Click on it and scroll down to permissions. Click Attach User Policy and paste the below into it, modifying it to the name of your bucket.

{
  "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": ["s3:ListBucket"],
            "Resource": [ "arn:aws:s3:::myserverbackup"]
        },
        {
            "Effect": "Allow",
            "Action": [ "s3:PutObject", "s3:GetObject", "s3:DeleteObject"],
            "Resource": [ "arn:aws:s3:::myserverbackup/*"]
        }
    ]
}

That's the AWS part done for now. To access your backups online you can go back to the S3 admin interface.

Setup s3cmd to upload files

s3cmd is a Python tool to put files on and retrieve files from AWS S3 buckets. It requires Python 2.4 or above, plus python-dateutil and optionally python-magic. Python usually comes with Linux but you may need to install the dependancies. On CentOS it goes like this:

yum -y install python-dateutil python-magic

Next download s3cmd. I tested with 1.5.0 rc1 but feel free to try a later one if it exists - let me know how it goes!

curl -L "http://downloads.sourceforge.net/project/s3tools/s3cmd/1.5.0-rc1/s3cmd-1.5.0-rc1.tar.gz?r=&ts=1408106691&use_mirror=heanet" -o s3cmd-1.5.0-rc1.tar.gz

Open, and optionally, move this package somewhere sensible

tar zxf s3cmd-1.5.0-rc1.tar.gz
mkdir /opt/s3
mv s3cmd-1.5.0-rc1/S3 s3cmd-1.5.0-rc1/s3cmd /opt/s3
ln -s /opt/s3/s3cmd /usr/local/bin/s3cmd

Configure it with the access keys you got from the IAM user setup. This will ask you for your IAM Access Key ID and Secret Access Key along with optional password and gpg key will protect your data against reading by Amazon staff or anyone who may get access to your them while they're stored at Amazon S3.

s3cmd --configure s3://myserverbackup

Next grab this backup script and put it somewhere sensible.

curl https://gist.github.com/merob/4f8923b014ffb3248f84/download# -o daily.backup.tgz
tar zxf daily.backup.tgz
mv gist4f8923b014ffb3248f84-*/daily.backup ~/bin
rmdir gist4f8923b014ffb3248f84-*

Take a look at the file daily.backup, there are a few things that need to be customized.

First create a file with a password in it that corresponds to the user which has access to backup your databases

echo mypassword > ~/.dbrootpw

Add the name of your s3 bucket and modify any config or directories that you want to be backed up. Here's a snippet of daily.backup:

DBPW=`cat /root/.dbrootpw`              # location of file which _just_ contains database root password
S3PATH="s3://nameofs3bucket"            # s3 bucket name
 
                                        # all lists separated by white space
ACTIVEUSERS="root"                      # any usernames who have homedirs to backup
DIRS=`ls -d /var/www/*`                 # list all directories to be backed up. (Will be backed up in separate files)
                                        # below: any custom config files
CONFIG="/etc/httpd/conf/httpd.conf /etc/httpd/conf.d /etc/postfix /etc/sysconfig/network /etc/sysconfig/iptables /etc/ssh/sshd_config /etc/aliases"

Add this to your crontab, ideally at an off-peak time of day

(crontab -l ; echo "7 1 * * * /root/bin/daily.backup
> ") | sort - | uniq - | crontab -

And that's it. It's probably worth checking that it works by running it manually, then logging on to AWS and checking your bucket contains the files you expect.

/root/bin/daily.backup

How to setup a web server on ElasticHosts

24-07-2014

By Rob Stevenson

This post details steps to create a set of LAMP servers that will serve a php website with MySQL sitting behind CloudFlare's CDN, Cache and optimizer.

Skip to the start

server topology

Over the last few years I've been maintaining a virtual private server at a cloud hosting company. I've built it up from a minimal stock CentOS 5 image and it's mostly been doing well. Recently it's got a bit cluttered as I've added more and more to it and I've hit a few memory issues and less than brilliant performance. Nothing against the current host, but it was time to upgrade and I happened to stumble upon ElasticHosts at the exact point I was ready to go.

What makes this host appealing to me over my previous host is their new Elastic Containers. Rather than fixing down a system to a set CPU speed, fixed RAM and disk space, this technology allows seamless burts of any of these over your set package. No reboot necessary.

Basically a container isn't a full virtual machine. It's a sand-boxed environment which sees all the available resource and is allocated it dynamclally by the underlying system on demand, up to a pre-determined (by me) maximum. So rather than paying for the 3GB of RAM I need to cope with a weekly peak-usage, I pay for 768MB and for the two hours per week when I need more, it uses it at a cost of a few pence.

[[email protected] ~]# free -m
            total   used    free  shared  buffers  cached
Mem:       128878  12789  116089       0       10    3853
-/+ buffers/cache:  8925  119952
Swap:       49151      0   49151

[[email protected] ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root       477G  110G  366G  24% /

[[email protected] ~]# du -sh / 2>/dev/null
610M    /

Total use: 12MHz, 768MB RAM, 4GB SSD. 1 IP. 1 VLAN. Subscriptions: 500MHz, 768MB, 5GB SSD. 1 IP. 1 VLAN.

Base image

I've based this on a pre-build minimal CentOS 6 image. There are a few common features so I created a base container which I dupliacted to create the other systems.

From the control panel hit 'Add', 'Server (Container)...' and give it a name. Select max sizes (default 2000MHz, 1024MB is probably fine), pre-installed system and pick centos-6 from the dropdown.

Setup a Virtual LAN

Add: Private VLAN
Connect

By default, you can login using ssh with the username toor. The IP address & password you'll need is shown on your control panel. Mac and Linux users can fire up a terminal and type:

ssh 203.0.113.123 -l toor

Windows users will need some extra software: download PuTTY plus PuTTYgen and Pageant for later, too.

Next we need to do some standard maintenence setup; update the system, name it and set the time zone. This example uses Europe/London but follow this link for others.

yum update -y
ln -sf /usr/share/zoneinfo/Europe/London /etc/localtime
cat >/etc/sysconfig/clock <<EOF
ZONE="Europe/London"
UTC=true
ARC=false
EOF

cat >>/etc/sysconfig/network <<EOF
HOSTNAME=base.example.com
EOF

Setup the VLAN. Create a new interface config file - Also pick an ip address.

NEWMAC=`ifconfig eth1 | grep HWaddr | awk '{print $5}'`
cat >/etc/sysconfig/network-scripts/ifcfg-eth1 <<EOF
DEVICE=eth1
BOOTPROTO="static"
HWADDR="$NEWMAC"
NM_CONTROLLED="yes"
ONBOOT="yes"
TYPE="Ethernet"
NETMASK=255.255.0.0
IPADDR=172.16.x.d
EOF
Network security

Prevent access to unused ports with iptables. At this stage we just need to block all access except for ssh.

cat >/etc/sysconfig/iptables <<EOF
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT
-A INPUT -p icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
-A INPUT -j REJECT --reject-with icmp-host-prohibited
-A FORWARD -j REJECT --reject-with icmp-host-prohibited
COMMIT
EOF

And start service and make sure it starts on boot:

service iptables start
chkconfig iptables on

Configure the ssh daemon to prevent password logins and setup a private key. This is optional but makes connecting easier and more secure. From your computer, generate an ssh key if you haven't already. If you don't know, ~/.ssh/id_rsa will probably exist if you already have one. If you're using Windows, PuTTYgen will create a key for you and Pageant will allow you to manage it without having to type in your key's password every time you connect. Once you've got your key, grab the public key (id_rsa.pub), and paste it into the SSH Keys section at the bottom of the profile/authentication page.

This will create two files ~/.ssh/id_rsa and ~/.ssh/id_rsa.pub. id_rsa should be kept very private while id_rsa.pub should be added to ~/.ssh/authorized_keys of any system you'd like to access with this key.

ssh-keygen -t rsa
cat .ssh/id_rsa.pub 

You might need to restart your server before your new key works. Or you can manually add it to ~/.ssh/authorized_keys, but I'm not sure if this file is special on ElasticHosts since it's added to automatically when you add keys to your profile.

You should also disable password logins as soon as you've confirmed you can login with your private key instead of passwords. To do this modify your sshd_config file and restart the ssh daemon.

patch /etc/ssh/sshd_config <<EOF
--- sshd_config         2014-08-06 11:32:04.916253729 +0100
+++ sshd_config.new     2014-08-06 11:23:06.848325452 +0100
@@ -63,7 +63,7 @@
 # To disable tunneled clear text passwords, change to no here!
 #PasswordAuthentication yes
 #PermitEmptyPasswords no
-PasswordAuthentication yes
+PasswordAuthentication no

 # Change to no to disable s/key passwords
 #ChallengeResponseAuthentication yes
@@ -94,7 +94,7 @@
 # PAM authentication, then enable this but set PasswordAuthentication
 # and ChallengeResponseAuthentication to 'no'.
 #UsePAM no
-UsePAM yes
+UsePAM no
EOF

service sshd restart
Email

Setup Postfix so we can receive email destined for the root account. (Courtesy of TecloTech's blog). I've used a Google Apps email account because it makes dealing with spam a non-issue, but this guide, or minor variation of this, should work for most email systems. Install Cyrus to allow secure connections, then create, hash (and secure) a file which will contain valid gmail credentials.

yum install cyrus-sasl-plain
cat >/etc/postfix/sasl_passwd <<EOF
smtp.gmail.com:587 [email protected]:emailpassword
EOF
chmod 600 /etc/postfix/sasl_passwd
postmap hash:/etc/postfix/sasl_passwd

Add some details to Postfix's config file

cat >>/etc/postfix/main.cf <<EOF
relayhost = smtp.gmail.com:587
smtp_tls_security_level = secure
smtp_tls_mandatory_protocols = TLSv1
smtp_tls_mandatory_ciphers = high
smtp_tls_secure_cert_match = nexthop

smtp_tls_CAfile = /etc/postfix/cacert.pem

smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options = noanonymous
EOF

And create a certificate

cd /etc/pki/tls/certs
make hostname.pem
cp /etc/pki/tls/certs/hostname.pem /etc/postfix/cacert.pem
service postfix restart

Redirect local email to a real place

cat >>/etc/aliases <<EOF
root: [email protected]
EOF
newaliases

Test that it all works. This should be delivered to [email protected]

sendmail root <<EOF
Hello World!
EOF

Web server

First shutdown the base image and take a copy. Fire it up and modify the network config:

patch /etc/sysconfig/network <<EOF
--- network	2014-07-23 15:19:02.850816786 +0100
+++ network.new	2014-07-27 22:09:13.810328498 +0100
@@ -1,2 +1,2 @@
-HOSTNAME=base.example.com
+HOSTNAME=web1.example.com
 NETWORKING=yes
EOF

Update the VLAN network to reflect hardware address and IP address

NEWMAC=`ifconfig eth1 | grep HWaddr | awk '{print $5}'`
patch /etc/sysconfig/network-scripts/ifcfg-eth1 <<EOF
--- ifcfg-eth1  2014-07-23 15:19:31.146956710 +0100
+++ ifcfg-eth1.new      2014-07-28 13:20:38.404805597 +0100
@@ -1,8 +1,8 @@
 DEVICE=eth1
 BOOTPROTO="static"
-HWADDR="aa:aa:aa:aa:aa:aa"
+HWADDR="$NEWMAC"
 NM_CONTROLLED="yes"
 ONBOOT="yes"
 TYPE="Ethernet"
 NETMASK=255.255.0.0
-IPADDR=172.16.x.d
+IPADDR=172.16.x.b
EOF

ifdown eth1
ifup eth1

Install and start web server (Apache)

yum install httpd 
chkconfig httpd on
service httpd start

Open up the firewall to http requests (port 80)

patch /etc/sysconfig/iptables <<EOF
--- iptables    2014-07-28 14:03:45.874945252 +0100
+++ iptables.new        2014-07-28 14:03:58.745376763 +0100
@@ -8,6 +8,7 @@
 -A INPUT -p icmp -j ACCEPT
 -A INPUT -i lo -j ACCEPT
 -A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
+-A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
 -A INPUT -j REJECT --reject-with icmp-host-prohibited
 -A FORWARD -j REJECT --reject-with icmp-host-prohibited
 COMMIT
EOF
service iptables restart

If you need any other applications, install them too. In this example, I'm using php any mysqli

yum install php php-mysqli

The default web root is /var/www/html. Unless running multiple sites, this doesn't need to change. So put something in there to test that everything's working

cat >/var/www/html/index.php <<EOF
<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>Hello World!</title>
</head>

<body>
 <h1>Hello World!</h1>
 <?php echo "<h2>Hello from php!</h2>"; ?>
</body>
</html>
EOF

Database

This is optional, dependent on requirements. A database server could be setup on its own container or on the same container as the web server. If you're likely to be running at any kind of scale, it might be worth setting it up separately but if it will be small, it may as well stay on the same server.

If on a different system, take another copy of base, then setup hostname and VLAN as before:

patch /etc/sysconfig/network <<EOF
--- network     2014-07-23 15:19:02.850816786 +0100
+++ network.new 2014-07-27 22:09:13.810328498 +0100
@@ -1,2 +1,2 @@
-HOSTNAME=base.example.com
+HOSTNAME=db.example.com
 NETWORKING=yes
EOF

Update the VLAN network to reflect hardware address and IP address

NEWMAC=`ifconfig eth1 | grep HWaddr | awk '{print $5}'`
patch /etc/sysconfig/network-scripts/ifcfg-eth1 <<EOF
--- ifcfg-eth1  2014-07-23 15:19:31.146956710 +0100
+++ ifcfg-eth1.new      2014-07-28 13:20:38.404805597 +0100
@@ -1,8 +1,8 @@
 DEVICE=eth1
 BOOTPROTO="static"
-HWADDR="aa:aa:aa:aa:aa:aa"
+HWADDR="$NEWMAC"
 NM_CONTROLLED="yes"
 ONBOOT="yes"
 TYPE="Ethernet"
 NETMASK=255.255.0.0
-IPADDR=172.16.x.d
+IPADDR=172.16.x.c
EOF

ifdown eth1
ifup eth1

Next, where on the web server or a dedicated database server, install and configure mysql:

yum install mysql mysql-server
chkconfig mysqld on
service mysqld start

Create a database and user and change the root password. It's worth adding that granting all privileges is a particularly bad plan for security. You should only give your users the minimal privileges required to do the job.

mysql_secure_installation
mysql -u root <<EOF
SET PASSWORD = PASSWORD('dbrootpassword');
CREATE DATABASE databasename;
GRANT ALL PRIVILEGES ON databasename.*
  To 'databaseuser'@'172.16.x.c'
  IDENTIFIED BY 'dbpassword' WITH GRANT OPTION;
FLUSH PRIVILEGES;
EOF

Set the timezone:

mysql_tzinfo_to_sql /usr/share/zoneinfo | mysql -u root -p mysql
cat >>/etc/my.cnf <<EOF
default-time-zone=Europe/London
EOF
patch /etc/sysconfig/iptables <<EOF
--- iptables    2014-07-28 14:03:45.874945252 +0100
+++ iptables.new        2014-07-28 14:03:58.745376763 +0100
@@ -8,3 +8,4 @@
 -A INPUT -p icmp -j ACCEPT
 -A INPUT -i lo -j ACCEPT
+-A INPUT -m state --state NEW -m tcp -s 172.16.x.c -p tcp --dport 3306 -j ACCEPT
 -A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT
EOF
service iptables restart

That's it. Point your browser to the IP address (listed in the control panel) of the container running Apache (httpd), and hopefully you should see your 'Hello World!' page. Eg: 203.0.113.123

Hello World Example

CloudFlare

Another optional bit - but I can't see any great reason not to! It will even shortly be supporting https as standard.

Signup here. From memory, this is straigh forward enougt not to bother documenting. Let me know if I'm wrong! The Gist is to hand over dns control of your domain to CloudFlare. Check all your dns entries have been copied over and select which records you want CloudFlare to server. Magic, and easy!

Google's Project Ara Modular Smart Phone

05-03-2014

By Rob Stevenson

Oooooh

A new era in mobile phone design? Huge potential to change the way we upgrade and think about phones.

The chassis has a small built-in battery to enable users to hot-swap parts and Google are aiming for it to cost just $15. They also hope to make a 'Grey Phone' available with a display, low-end processor, battery and WiFi support for $50, which is the crossover point between feature phones and smartphones.
Project Ara Phone with some removed modules

Source 3g.co.uk

comments powered by Disqus