mod_mime + php = hacked site.

So I learned something about mod_mime today that made my jaw drop.

The default way of telling apache to parse a php file looks something like this:

AddHandler php5-script .php

If you install php via the command line on RHEL 4,5 or 6, this is how it sets it up.

What I didn't know is that mod_mime expands the match (.php) to anywhere in the file name.
So test.php or test.php.csv or test.php.jpg would all be passed to the php handler to be executed.

Facepalm

That's a big deal when your application accepts file uploads and is only type-checking the last file extension.
Magento, expression engine, wordpress, etc…

The workaround is to only apply the php handler to files which end in ".php"

<FilesMatch \.php$>
    SetHandler php5-script
</FilesMatch>

And for a little extra 'security', disable php for a directory if you're accepting uploads.

<Directory "/var/www/html/example/uploads">
    php_flag engine off
</Directory>

Which I'm going to go back and change on any server I've ever setup.
I learned this tid-bit from a security advisory from magento.

UPDATE: You may also see a lot of folks who recommend turning on "open_basedir" in php to lock thinks down.
There is a cavet there too. When "open_basdir" is in use, php disables the realpathcache. 
This makes loading/including files very slow.

Steps to migrate a code repository from CVS to GIT

CVS has served us well for many years. The interface in eclipse is so good it was really hard to justify switching to something else. 

That day has come. This tutorial was done on a mac from the command line interface. GIT comes with mac os now and cvs2git was avalible via the brew install system.
This process should also work on linux command line too.

Requirements:

  • A linux/Unix computer.
  • Install git http://git-scm.com/downloads
  • Install cvs2svn http://cvs2svn.tigris.org/cvs2git.html
  • Direct access to CVS repository files.

Step 1: Make a copy of your raw CVS repository.
This doesn't work correctly with a working copy.
For this example its on my desktop.

~/Desktop/cvs-local

Step 2: Let csv2git to process the repository.

cvs2git –blobfile="~/Desktop/blob.dat" –dumpfile="~/Desktop/dump.dat" –username="dkilgo" ~/Desktop/cvs-local/my-sample-project

If your cvs repository is corrupt, you may recieve warnings from cvs2git like the following:

ERROR: A CVS repository cannot contain both ~/Desktop/cvs-local/my-sample-project/help/internal/compile/Release_text0.htm,v and ~/Desktop/cvs-local/my-sample-project/help/internal/compile/Attic/Release_text0.htm,v
ERROR: A CVS repository cannot contain both ~/Desktop/cvs-local/my-sample-project/help/internal/compile/Release_text1.htm,v and ~/Desktop/cvs-local/my-sample-project/help/internal/compile/Attic/Release_text1.htm,v

In CVS when a file is deleted, it moves into the attic folder. Its not techniclly correct for both files to exist. You'll have to examine both files with a text editor and see which one has the highest revision number and delete the older file.
All the times I encountered this error, the Attic version was older.
Once you've resolved all the errors, re-run the cvs2git command.

A successful run will provide something like the following output:

Time for pass1 (CollectRevsPass): 5.119 seconds.
—– pass 2 (CleanMetadataPass) —–
Converting metadata to UTF8…
Done
Time for pass2 (CleanMetadataPass): 0.026 seconds.
—– pass 3 (CollateSymbolsPass) —–
Checking for forced tags with commits…
Done
Time for pass3 (CollateSymbolsPass): 0.011 seconds.
—– pass 4 (FilterSymbolsPass) —–
Filtering out excluded symbols and summarizing items…
Done
Time for pass4 (FilterSymbolsPass): 74.65 seconds.
—– pass 5 (SortRevisionsPass) —–
Sorting CVS revision summaries…
Done
Time for pass5 (SortRevisionsPass): 0.179 seconds.
—– pass 6 (SortSymbolsPass) —–
Sorting CVS symbol summaries…
Done
Time for pass6 (SortSymbolsPass): 0.110 seconds.
—– pass 7 (InitializeChangesetsPass) —–
Creating preliminary commit sets…
Done
Time for pass7 (InitializeChangesetsPass): 1.975 seconds.
—– pass 8 (BreakRevisionChangesetCyclesPass) —–
Breaking revision changeset dependency cycles…
Done
Time for pass8 (BreakRevisionChangesetCyclesPass): 0.727 seconds.
—– pass 9 (RevisionTopologicalSortPass) —–
Generating CVSRevisions in commit order…
Done
Time for pass9 (RevisionTopologicalSortPass): 0.547 seconds.
—– pass 10 (BreakSymbolChangesetCyclesPass) —–
Breaking symbol changeset dependency cycles…
Done
Time for pass10 (BreakSymbolChangesetCyclesPass): 1.134 seconds.
—– pass 11 (BreakAllChangesetCyclesPass) —–
Breaking CVSSymbol dependency loops…
Done
Time for pass11 (BreakAllChangesetCyclesPass): 1.681 seconds.
—– pass 12 (TopologicalSortPass) —–
Generating CVSRevisions in commit order…
Done
Time for pass12 (TopologicalSortPass): 1.520 seconds.
—– pass 13 (CreateRevsPass) —–
Mapping CVS revisions to Subversion commits…
Creating Subversion r1 (Project initialization)
Creating Subversion r2 (commit)
Creating Subversion r3 (commit)
Creating Subversion r4 (commit)



Done
Time for pass13 (CreateRevsPass): 1.538 seconds.
—– pass 14 (SortSymbolOpeningsClosingsPass) —–
Sorting symbolic name source revisions…
Done
Time for pass14 (SortSymbolOpeningsClosingsPass): 0.854 seconds.
—– pass 15 (IndexSymbolsPass) —–
Determining offsets for all symbolic names…
Done.
Time for pass15 (IndexSymbolsPass): 0.373 seconds.
—– pass 16 (OutputPass) —–
Time for pass16 (OutputPass): 1.455 seconds.

cvs2svn Statistics:
——————
Total CVS Files:              1417
Total CVS Revisions:          2982
Total CVS Branches:           1482
Total CVS Tags:              36376
Total Unique Tags:              59
Total Unique Branches:           2
CVS Repos Size in KB:        25792
Total SVN Commits:             579
First Revision Date:    Thu May 17 08:25:14 2007
Last Revision Date:     Fri Aug 17 11:13:14 2013
——————
Timings (seconds):
——————
 5.12   pass1    CollectRevsPass
 0.03   pass2    CleanMetadataPass
 0.01   pass3    CollateSymbolsPass
74.65   pass4    FilterSymbolsPass
 0.18   pass5    SortRevisionsPass
 0.11   pass6    SortSymbolsPass
 1.97   pass7    InitializeChangesetsPass
 0.73   pass8    BreakRevisionChangesetCyclesPass
 0.55   pass9    RevisionTopologicalSortPass
 1.13   pass10   BreakSymbolChangesetCyclesPass
 1.68   pass11   BreakAllChangesetCyclesPass
 1.52   pass12   TopologicalSortPass
 1.54   pass13   CreateRevsPass
 0.85   pass14   SortSymbolOpeningsClosingsPass
 0.37   pass15   IndexSymbolsPass
 1.46   pass16   OutputPass
91.90   total

Step 3: Create a git repository and import the data into it.

mkdir sample-git
cd sample-git
git init
cat ../blob.dat ../dump.dat | git fast-import

Which will create the following output:

git-fast-import statistics:
———————————————————————
Alloc'd objects:       5000
Total objects:         3057 (      1285 duplicates                  )
      blobs  :         1754 (      1270 duplicates       1254 deltas of       1668 attempts)
      trees  :          956 (        15 duplicates        758 deltas of        892 attempts)
      commits:          347 (         0 duplicates          0 deltas of          0 attempts)
      tags   :            0 (         0 duplicates          0 deltas of          0 attempts)
Total branches:          44 (         6 loads     )
      marks:     1073741824 (      3371 unique    )
      atoms:            997
Memory total:          2594 KiB
       pools:          2360 KiB
     objects:           234 KiB
———————————————————————
pack_report: getpagesize()            =       4096
pack_report: core.packedGitWindowSize = 1073741824
pack_report: core.packedGitLimit      = 8589934592
pack_report: pack_used_ctr            =        802
pack_report: pack_mmap_calls          =         41
pack_report: pack_open_windows        =          1 /          1
pack_report: pack_mapped              =   31753114 /   31753114
———————————————————————

Have a look in the git repo directory and you'll find its empy. Where's all your stuff?
Its in the .git directory. You don't have a working copy checked out yet.

Do "git checkout" to get a working copy of your repo.

Now you have a functional git repository with all your CVS comments, tags and branches.

Side note:

Changing to git ment updating our deployment scripts as well.
The old way with CVS looked like this:

cvs -d /home/cvs/devel -q export -r $tag -d $stageDir/sample-project smaple-project

Git has a little more overhead in that by default the cloned repo contains all the branches and commits, not just the tagged/current version.
That means what it checks out is usually a bit bigger than what would be picked up by the CVS export.
To workaround that, you can set the –depth param to limit git where it only cares about the last version of each file and folder.

/usr/local/bin/git clone –depth 1 –branch $tag git\@myserver.mydomain.com:sample-project.git $stageDir/sample-project/

I always remove the revision control files before deployment. Its a pain with CVS and SVN. With git, its a snap because they're all in the same folder.

rm -f -r $stageDir/sample-project/.git

Open Webalizer DNS cache with PHP

Problem:

Webalizer produces a cache file for DNS to speed up the process for resolving addresses to names but no utilities exist to browse this kind of file.

Background:

Webalizer is a log file analyser which can provide basic site usage stats by processing your apache log files. Its a standard utility in a RHEL envirnment. One feature of it is caching DNS information to speed up resolving addresses for reporting purposes. It uses a Berkeley DB to store these key/value pairs for quick lookup. Unfortunatly there really arn’t an gui tools for getting a look at this data. On my machine, its a 70MB file and I’d really like to get a look at that data.

So there are a couple of ways to do this.
MySQL comes with a bdb storage engine which can read this kind of file.

Or, you could use PHP and its driver to load this information.
I chose the latter.

Environment:
I’m starting with a base install of ubuntu 11 and zend server.
Using zend server makes the process more complicated but makes the process more standardized.  ZS uses the same paths for files on every OS it installs on.

Install dependent software.

Zend doesn’t come with this module by default, so we’ll need to compile it.
Make sure you’ve got the basics needed to compile under ubuntu installed.


aptitude install install build-essential checkinstall autoconf

Next we need to make sure that libdb4 is installed so we can reference it in php.

aptitude install libdb4.8-dev db4.8-util db4.8-doc libdb4.8

And finaly, we need the header files for zend server’s php.

sudo aptitude install php-5.3-source-zend-server

Sanity check – Lets make sure that the module isn’t installed.


php --ri dba
Extension 'dba' not present.

Great. Lets build it.


cd /usr/local/zend/share/php-source/php-5.3.9/ext/dba/
/usr/local/zend/bin/phpize
./configure --with-php-config=/usr/local/zend/bin/php-config --with-db4
make
make install
echo "extension=dba.so" > /usr/local/zend/etc/conf.d/dba.ini

lets check that again


php --ri dba
dba

DBA support => enabled
Supported handlers => gdbm cdb cdb_make db4 inifile flatfile

Directive => Local Value => Master Value
dba.default_handler => flatfile => flatfile

Done. We can now access the dba functions from the command line.
They’ll also get picked up by apache the next time you restart.

Build a quick script to read the dns cache and output it as a comma separated list.


<?php
/**
* Requires the dba module be enabled with db4 support.
*/
$path = dirname(__FILE__) . '/dns_cache.db';
$logPath = $path . '.txt';
echo 'Loading ' . $path . chr(10);

$row = 0;
$h = dba_open($path,'r','db4');
$fh = fopen($logPath,'w+');

if(!$h){
echo 'Failed to open db';
die();
}

echo "Exporting Key/Values pairs...\n";
$key = dba_firstkey($h);
$value = dba_fetch($key,$h);
$value = filter_var($value,FILTER_SANITIZE_URL);
$key = filter_var($key,FILTER_SANITIZE_URL);
fwrite($fh, "$key,$value\n");
$row++;

while($key = dba_nextkey($h)){
$value = dba_fetch($key,$h);
$value = filter_var($value,FILTER_SANITIZE_URL);
$key = filter_var($key,FILTER_SANITIZE_URL);
fwrite($fh, "$key,$value\n");
$row++;
}
echo "Printed $row key/value pairs." . chr(10);
echo 'Done' . chr(10);
dba_close($h);

I called my script test.php.
Then run it from the command line.


php -r ./test.php

Loading /home/{MASKED}/dns_cache.db
Exporting Key/Values pairs...
Printed 703268 key/value pairs.
Done

Not sure if this is the best way to get at this data but its the route I chose and it worked.
Your mileage may vary.

XRDP is great

xrdp.org hosts an open-source implementation of Microsoft’s remote desktop protocol server. This is a great tool long overdue on GNU/Linux.

Remote Desktop Protocol (RDP) is a proprietary protocol developed by Microsoft, which provides a user with a graphical interface to another computer. Unlike VNC which simply forwards screen shots and keyboard input from another computer, A RDP session is a virtual terminal creating a separate instance of X.org server running a desktop separate from other logged in users. No users need to be logged in to create a new RDP session.
This means the session can be configured by the client so the screen is displayed in the native resolution of the client. Using a RDP client at full screen feels like your using a native operating system. Its also more efficient in terms of bandwidth needed to communicate video and keyboard data.

Installing in ubuntu 11.10 is easy. Its a package which can be installed with apt-get.

sudo apt-get install xrdp

Great, but I encountered a couple of minor issues.
For one after connecting with a client, I was only able to see the desktop. I did not get tool bars or any way to interact with the terminal.
Turns out its an easy fix.

cd /home/youruser
echo "gnome-session --session=ubuntu-2d" > .xsession
sudo /etc/init.d/xrdp restart

This tells xrdp to use ubuntu-2d for your remote session.
This is all I needed to get my menus to show up. Note that restarting xrdp will disconnect your session.

The next issue is pressing the ‘d’ key causes all active windows to be hidden. This can be corrected from the GUI.

  • Click on the ubuntu menu and search for ‘system settings’.
  • Click on ‘keyboard’.
  • Click on the ‘shortcuts’ tab, then the ‘navigation’ tab and find ‘Hide all normal windows’. I changed mine to be ‘alt + d’ .

Once these easy tweaks are done, you’ll have your self a very usable remote desktop server.

O to Dev in 10 seconds with Ubuntu

The end result is to take a based install of ubuntu 10.x desktop and make it ready for php web development.

These commands download about 500MB of software.

Run it line by line as root or make a bash script out of it.


#!/bin/bash

#All of these steps must be done as root.
if [ "$(whoami)" != 'root' ]; then
echo "This script must be run as root."
exit 1;
fi

#Add the zend repo to apt.
echo "deb http://repos.zend.com/zend-server/deb server non-free" >> /etc/apt/sources.list

#Add zend's signing key to the apt key ring so we can use the zend repo.
wget http://repos.zend.com/deb/zend.key -O- | sudo apt-key add -

#Add yogarine's repo so we can download the latest version of eclipse and the php development tools.
add-apt-repository ppa:yogarine/eclipse/ubuntu

# Update your repo cached software list.
apt-get update

#Install eclipse with php development tools (latest), zend server (apache, php, and php control panel), mysql (cli client and server) in a single command.
apt-get install eclipse-pdt zend-server-ce-php-5.3 php-5.3-extra-extensions-zend-server-ce mysql-server mysql-client phpmyadmin

Post install Tasks:

Reboot your computer.
Eclipse will install openJDK and you must restart to complete the installation.

Visit http://127.0.0.1:10081 to complete the setup of your zend server control panel.

Setup your document root.
I usually make a directory in var for my workspace and point eclipse to that location like so.

cp /var/www/*.php ~/workspace/
sudo rm -f -r /var/www
#Replace $USER with your login name.
sudo ln -s -v ~/home/$USER/workspace/ /var/www

To access the debugger from PDT, add the following get variables to your request:


http://localhost/test/info.php?debug_host=127.0.0.1%2C127.0.0.1&start_debug=1&debug_port=10000&original_url=http%3A%2F%2Flocalhost%2Ftest%2Finfo.php&send_sess_end=1&debug_stop=1&debug_start_session=1&debug_no_cache=1310991085348&debug_session_id=1000