CakePHP: Autotest using Watchr

Posted on May 9th, 2010 in Web Development | 4 Comments »

I was taking a look at ZenTest a while ago, especially the autotest functionality. As stated in the ZenTest site, autotest:

  • Improves feedback by running tests continuously.
  • Continually runs tests based on files you’ve changed.
  • Get feedback as soon as you save. Keeps you in your editor allowing you to get stuff done faster.

To add more bang to your buck, you can get continuous notifications from growl every time you save a file, alerting you if any of your previous test cases have failed.

Googling around I wanted to see if this can be tied into testing with CakePHP. If I did a horrible job googling and there is already something out there that mimics this functionality please send me a link. Anyways, I stumbled across Watchr. Watchr is a tool that monitors a directory tree, and triggers a user defined action whenever an observed file is modified. At that point trigger a CakePHP test suite command line argument and send the results to growl. And there we have it, continuous testing with growl notifications on CakePHP using Watchr.

Step 1: Install the Watchr gem: http://github.com/mynyml/watchr

Step 2: Upload images that growl will use to display passed/failed cases

Step 3: Set up a watchr configuration file in app/config/watchr.rb

Step 4: Every time you develop have watchr runing. Type the following in your root directory:

watchr app/config/watchr.rb

End Result:

Pass:

Fail:

Failed and Passed images to pass to growl

Create a directory ~/.watchr_images and upload the following images or any image of your choice for failed and passed test cases:

Break down of the watchr.rb configuration file

Define a global variable that points to your cake console script. I wanted to load this variable from my aliases in .profile but couldn’t figure out how (my ruby skills suck):

$cake = "/Applications/MAMP/bin/php5/bin/php /Applications/MAMP/htdocs/yourProject/cake/console/cake.php"

Define a function that will parse out the details you want to pass to growl. I only cared about the number of passes and fails.

def growl(message)
  message = message.split('---------------------------------------------------------------')[3].split('Time taken by tests')[0]
  growlnotify = `which growlnotify`.chomp
  image = message.include?('fails') ? "~/.watchr_images/failed.png" : "~/.watchr_images/passed.png"
  options = "-w -n Watchr --image '#{File.expand_path(image)}' -m '#{message}'"
  system %(#{growlnotify} #{options} &)
end

Define a function that will run a command-line program and passes the result to growl

def run(cmd) 
  puts(cmd)
  result = `#{cmd}`
  growl result rescue nil
  puts result
end

Define a function that calls the test case of the pertaining model or controller that just got saved:

def test_changed_model_or_controller(file)
  type = file.split('/')[1]
  name = file.split('/')[2].split('.')[0]
  run "#{$cake} testsuite app case #{type}/#{name}" 
end

Define a function that calls the test case of the pertaining test case that just got saved:

def test_changed_test_case(file)
  type = file.split('/')[3]
  name = file.split('/')[4].split('.')[0]
  run "#{$cake} testsuite app case #{type}/#{name}"
end

Define a function that runs the entire test suite for changes in config files, app_controller and app_model:

def test_app()
  run "#{$cake} testsuite app all"
end

The heart of the script, the watchers. Here we state which files to monitor using regular expression pattern matching paths.

watch('app/(models|controllers)/(.*).php')  { |m| test_changed_model_or_controller(m[0]) }
watch('app/tests/cases/(models|controllers)/(.*).test.php')  { |m| test_changed_test_case(m[0]) }
watch('app/config/(.*).php') { |m| test_app() }
watch('app/(app_controller|app_model).php')  { |m| test_app() }

That’s it. Run the watcher via “watchr app/config/watchr.rb” as you develop and your test cases will continuously run giving you growl update notifications.

Download the watchr.rb config file


Sources:
Watchr Readme
Watchr – Most of my time was spent here modifying this script and applying it to a CakePHP scenario
Setting up Watchr and Rails
Watchr: A Flexible, Generic Alternative to AutoTest – This was the first article I stumbled across that got me started on Watchr
CakePHP – Running Tests in the Command Line

Web Scraping using PHP and XPath

Posted on March 30th, 2010 in Web Development | 1 Comment »

Yet another thing on my to-do list has been to either get someone to either start updating the product catalog on the Tech And House site, or to get a scraper to extract the information on the Frigidaire site. Didn’t realize how easy it was to whip up an extraction script using XPath! First, if you don’t already have the XPather add-on for Firefox, download it. Makes determining the XPath to elements on the page very easy:



Using the code below, scraping data from pretty much any site shouldn’t be too difficult. Modifying the XPaths in frigidaire_scraper.php and its extract_specs() method are where the main changes will occur to build a scraper for another site.

File 1: scraper.php

<?php
class Scraper {
  var $oldSetting;
  var $html;
 
  function scraper($targetUrl) {
    $this->oldSetting = libxml_use_internal_errors( true ); 
    libxml_clear_errors(); 
 
    $this->html = new DOMDocument(); 
    $this->html->loadHtmlFile($targetUrl); 
  }
 
  function extract($links_xpath,$extract,$include_blanks=true) {
    $return = array();
 
    $xpath = new DOMXPath($this->html); 
    $items = $xpath->query($links_xpath);
 
    foreach ($items as $item) {        
    	$newDom = new DOMDocument;
    	$newDom->appendChild($newDom->importNode($item,true));
 
    	$xpath = new DOMXPath($newDom); 
    	$extraction = trim($xpath->query($extract)->item(0)->nodeValue);
 
    	if ($include_blanks==true)
    	  array_push($return,$extraction);
    	else if ($extraction!="")
    	  array_push($return,$extraction);
    }
 
    return $return;
  }
}
?>

File 2: frigidaire_scraper.php

<?php
require ("scraper.php");
 
class FrigidaireScraper extends Scraper {
 
  var $category_links;
  var $product_links;
  var $products;
  var $keys;
 
  function FrigidaireScraper($targetUrl) {
    parent::scraper($targetUrl);
  }
 
  function extract_category_links() {
    $this->category_links = $this->extract("//div[@id='left-nav']/a[@class='left-nav-item-sub']","@href");
  }
 
  function extract_product_links() {
    $this->product_links = array();
    foreach($this->category_links as $link): 
      parent::scraper($link);
      $products = $this->extract("//table[@id='ProductTable']//h4/a","@href");
      $this->product_links = array_merge($this->product_links,$products);
    endforeach;
  }
 
  function extract_specs() {
    $this->keys = array();
    $this->products = array();
 
    foreach($this->product_links as $link):  
      parent::scraper($link);
      $model = explode(": ", array_pop($this->extract("//div[@id='main-inner']//h5","text()[1]")));
      $model = $model[1];
      $msrp = explode(" ", array_pop($this->extract("//div[@id='main-inner']//h5","text()[3]")));
      $msrp = $msrp[1];
 
      $specs = $this->extract("//div[@id='ctl00_CPHMain_TabContainer1_TabSpecifications']/div[2]/div/div[1]/div","text()",false);
 
      $this->products[$model]["Price"] = $msrp;
      $this->products[$model]["Model"] = $model;
      $this->products[$model]["Link"] = $link;
 
      foreach($specs as $spec):
        $spec = explode(": ", $spec);
        $this->products[$model][$spec[0]] = $spec[1];
      endforeach;
 
      $this->keys = array_unique(array_merge($this->keys,array_keys($this->products[$model])));
    endforeach;
  }
}
?>

File 3: scrape_it.php (bring it all together)

<?php
require_once("frigidaire_scraper.php");
 
$scraper = new FrigidaireScraper("http://www.frigidaire.com");
$scraper->extract_category_links();
$scraper->extract_product_links();
$scraper->extract_specs();
 
//echo a simple tab delimited file structure
foreach ($scraper->keys as $key):
  echo $key."\t";
endforeach;
 
echo "\n";
 
foreach ($scraper->products as $row):    
  foreach ($scraper->keys as $key):
    if (isset($row[$key]))
      echo $row[$key] . "\t";
    else echo "\t";
  endforeach;
  echo "\n";
endforeach;
?>

Scrape Output:

Installing ImageMagicK and its dependencies

Posted on February 8th, 2010 in Web Development | No Comments »

To get barcode-generator for rails working I needed to get ImageMagicK installed. I had some problems installing it using the instructions on the site. I was using the following posts to get ImageMagicK installed:

RMagicK from source on Snow Leopard
Installing ImageMagicK & RmagicK on Leopard
RMagick & ImageMagick from Source: Mac OSX Leopard

Problem 1: For some reason curl was not working. I could not decompress the files. When attempting to decompress, I would get a file with an extension of .cpgz. Attempting to open that file, I would get a zip file, unzipping that file I would get a .cpgz file. Still don’t know what was up with that! Vicious never ending cycle. Other people with similar issue.

Problem 2: Could not get littlecms and libwmf compiled and installed. Would fail at the make clean step.

Problem 3: ImageMagicK installation instructions using the Apple binary did not work for me.

Below you can find the steps I used to get ImageMagicK compiled and installed with updated links to the latest stable releases – as of today:

wget http://hivelocity.dl.sourceforge.net/project/freetype/freetype2/2.3.11/freetype-2.3.11.tar.gz
tar xzvf freetype-2.3.11.tar.gz
cd freetype-2.3.11
./configure --prefix=/usr/local
make
sudo make install
cd ..
 
wget ftp://ftp.simplesystems.org/pub/libpng/png/src/libpng-1.4.0.tar.gz
tar xzvf libpng-1.4.0.tar.gz
cd libpng-1.4.0
./configure --prefix=/usr/local
make
sudo make install
cd ..
 
wget http://www.ijg.org/files/jpegsrc.v8.tar.gz
tar xzvf jpegsrc.v8.tar.gz
cd jpeg-8
ln -s `which glibtool` ./libtool
export MACOSX_DEPLOYMENT_TARGET=10.6.2
./configure --enable-shared --prefix=/usr/local
make
sudo make install
cd ..
 
wget ftp://ftp.remotesensing.org/pub/libtiff/tiff-3.9.2.tar.gz
tar xzvf tiff-3.9.2.tar.gz
cd tiff-3.9.2
./configure --prefix=/usr/local
make
sudo make install
cd ..
 
wget http://ghostscript.com/releases/ghostscript-8.70.tar.gz
tar xzvf ghostscript-8.70.tar.gz
cd ghostscript-8.70/
./configure  --prefix=/usr/local
make
sudo make install
cd ..
 
wget ftp://ftp.imagemagick.org/pub/ImageMagick/delegates/ghostscript-fonts-std-8.11.tar.gz
tar xzvf ghostscript-fonts-std-8.11.tar.gz
sudo mv fonts /usr/local/share/ghostscript
 
wget ftp://ftp.imagemagick.org/pub/ImageMagick/ImageMagick.tar.gz
tar xzvf ImageMagick.tar.gz
cd ImageMagick-6.5.9-3/
export CPPFLAGS=-I/usr/local/include
export LDFLAGS=-L/usr/local/lib
./configure --prefix=/usr/local --disable-static --with-modules --without-perl --without-magick-plus-plus --with-quantum-depth=8 --with-gs-font-dir=/usr/local/share/ghostscript/fonts
make
sudo make install
cd ..

After installation, check if ImageMagicK was installed correctly. The following command should create a new file called logo.gif with the ImageMagicK logo:

/usr/local/bin/convert logo: logo.gif

For a more comprehensive test, run the following command:

make check

Results:

make check
make  check-am
make  tests/validate  wand/drawtest wand/wandtest
  CC     tests/tests_validate-validate.o
  CCLD   tests/validate
  CC     wand/drawtest.o
  CCLD   wand/drawtest
  CC     wand/wandtest.o
  CCLD   wand/wandtest
make  check-TESTS check-local
PASS: tests/validate-compare.sh
PASS: tests/validate-composite.sh
PASS: tests/validate-convert.sh
PASS: tests/validate-formats-on-disk.sh
PASS: tests/validate-formats-in-memory.sh
PASS: tests/validate-identify.sh
PASS: tests/validate-import.sh
PASS: tests/validate-montage.sh
PASS: tests/validate-stream.sh
PASS: wand/drawtest.sh
PASS: wand/wandtest.sh
===================
All 11 tests passed
===================

MySQL Gem on 64-bit Snow Leopard

Posted on November 28th, 2009 in Web Development | No Comments »

Finally took some time to move a small rails application from my old Macbook Pro to my Macbook Pro running Snow Leopard and MAMP. Setting up autotest which should have just been a few commands ended up being a battle getting the MySQL gem set up correctly.

Autotest setup

What should have been quick and simple…

sudo gem install ZenTest
gem install autotest-rails

Make a file named .autotest in the root directory of the application and add:

require "autotest/growl"

Ended up being…

1) Update the system first.:

sudo gem update --system

Error:

!!! The bundled mysql.rb driver has been removed from Rails 
2.2. Please install the mysql gem and try again: gem install 
mysql.

From this point the battle starts. I have recompiled the MySQL gem many times trying to get it working. Excerpts of a few errors:

Enclosing class/module 'mXML' for class XPointer not known
ERROR:  Error installing mysql:
ERROR: Failed to build gem native extension.
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby extconf.rb
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lm... yes
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lz... yes
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lsocket... no
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lnsl... no
checking for mysql_query() in -lmysqlclient... no
checking for main() in -lmygcc... no
checking for mysql_query() in -lmysqlclient... no
Gem files will remain installed in /Library/Ruby/Gems/1.8/gems/mysql-2.8.1 for inspection.
Results logged to /Library/Ruby/Gems/1.8/gems/mysql-2.8.1/ext/mysql_api/gem_make.out
 
Could not find main page README
Could not find main page README
Could not find main page README
Could not find main page README
ERROR:  Error installing rubynode:
ERROR: Failed to build gem native extension.
 
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/bin/ruby extconf.rb
==================== ERROR =====================
Please set RUBY_SOURCE_DIR to the source path of ruby 1.8.7 (2008-08-11)!
================================================
uninitialized constant MysqlCompat::MysqlRes

The solution…

Finally got it working following the steps in this article. One gotcha of this article – since we are installing on a 64 bit operating system the last step needs to be installed with the -arch x86_64 flag, not -arch i386.

First download the MAMP source here, then:

unzip MAMP_1.7.2_src.zip
cd MAMP_1.7.2_src
tar -xzvf mysql-5.0.41.tar.gz
cd mysql-5.0.41 
./configure --with-unix-socket-path=/Applications/MAMP/tmp/mysql/mysql.sock --without-server --prefix=/Applications/MAMP/Library
make -j2
cp libmysql/.libs/*.dylib /Applications/MAMP/Library/lib/mysql 
mkdir /Applications/MAMP/Library/include
cp -R include /Applications/MAMP/Library/include/mysql
sudo env ARCHFLAGS="-arch x86_64" gem install mysql -- --with-mysql-config=/Applications/MAMP/Library/bin/mysql_config

Git – Ignore already checked in file/directory

Posted on November 14th, 2009 in Web Development | No Comments »

Original Source: stackoverflow.com

This command will cause git to untrack your directory and all files under it without actually deleting them:

git rm -r --cached <your directory>

The -r option causes the removal of all files under your directory.
The –cached option causes the files to only be removed from git’s index, not your working copy. By default git rm would delete .

Renaming files – Adding a suffix to the file name

Posted on October 23rd, 2009 in Web Development | 1 Comment »

The solution provided in Easily Renaming Multiple Files works perfectly well if we want to add a prefix to a file. This solution does not work too well if you want to add a suffix.

Objective: For all jpegs in a folder add the suffix _tn after the file name and before the extension.
Example: file1.jpg = file1_tn.jpg

In the console type the following:

for i in *.jpg ; do mv $i `echo $i | sed 's/\.jpg/\_tn.jpg/'` ; done

If the file name was in CAPS and and you’d rather have your file name in lowercase pipe the following on:

for i in *.jpg ; do mv $i `echo $i | sed 's/\.jpg/\_tn.jpg/' | tr [:upper:] [:lower:]` ; done

Setting up SSH public/private keys on OS X

Posted on October 13th, 2009 in Web Development | No Comments »

Sometimes it’s a little bit of a pain to constantly type your password every time you log in to a server. If you are pushing code, moving files, backing up data, typing your password at each step of the way can be annoying. By setting up public/private keys between your local machine and the remote server you can eliminate that step.

Type the following on the local machine:

ssh-keygen -t dsa

You will see the following after typing the command above. You can press enter and skip the question when you’re asked for a file name, but for the passphrase try not to leave it blank:

Generating public/private dsa key pair.
Enter file in which to save the key (/Users/amitsamtani/.ssh/id_dsa): 
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /Users/amitsamtani/.ssh/id_dsa.
Your public key has been saved in /Users/amitsamtani/.ssh/id_dsa.pub.

If you did not change the file name copy the contents of ~/.ssh/id_dsa.pub to the remote servers ~/.ssh/authorized_keys file. Append the contents to the end of the file.

After this is complete, on your first attempt to ssh into the remote server you will be presented with a prompt by OS X to type the id_dsa.pub passphrase. Type the passphrase into the prompt and save it in your keychain.

Now you can log in to the server without the need of the password.

WordPress – Move existing blog to a development server

Posted on October 5th, 2009 in Web Development | 1 Comment »

First tar/gzip the blog and the database:

tar -cf old_blog.tar blog/
gzip old_blog.tar
mysqldump -h hostname.com -u username -p database_name > dump.sql
gzip dump.sql

Bring the blog and database to your local machine and set it up. For simplicity sake we will assume that the database name, username and password is the same as your live server.

scp user@hostname.com:/path/to/files/old_blog.tar.gz .
scp user@hostname.com:/path/to/files/dump.sql.gz .
tar -xzvf old_blogtar.gz
tar -xzvf dump.sql.gz
mysql -h localhost -u user -p database_name < dump.sql

Log in to your database and change the values of siteurl and home in the wp_options table to point to your local machine’s domain:

update wp_options set option_value = 'http://localhost:8888/domain.com' where option_name in ('siteurl','home);

wget

Posted on October 5th, 2009 in Web Development | No Comments »

I just realized I do not have a copy of wget on my installation of Snow Leopard. If you need it, its just a matter of getting the latest code via curl and installing.

curl -O http://ftp.gnu.org/gnu/wget/wget-latest.tar.gz
tar -xzvf wget-latest.tar.gz
cd wget-1.12
./configure --prefix=/usr/local
make
sudo make install

Now lets use wget in an example:

wget http://wordpress.org/latest.tar.gz
tar -xzvf latest.tar.gz
mv wordpress new_blog

Very nice!

Easily Renaming Multiple Files

Posted on September 18th, 2009 in Web Development | 1 Comment »

Update (2009-10-23): This works well for adding a prefix to files in a folder. To add a suffix to a file read “Renaming files – Adding a suffix to the file name” instead.

Here is a quick way to rename multiple files using bash.

Objective: Rename all image files in a directory in uppercase, to lowercase and append a tn_ prefix.

Step 1: Prepare a working directory

mkdir test
cd test
touch FILE_1.jpg
touch FILE_2.jpg
touch FILE_3.jpg

Step 2: Test first – print out all files in lower case

for i in *.jpg; do echo $i | tr [:upper:] [:lower:]; done

Output

file_1.jpg
file_2.jpg
file_3.jpg

Step 3: Test first – print out all files prefixed with tn_ and lower cased

for i in *.jpg; do echo "tn_"$i | tr [:upper:] [:lower:]; done

Output

tn_file_1.jpg
tn_file_2.jpg
tn_file_3.jpg

Step 4: Rename the files in the directory

for i in *.jpg; do mv $i `echo "tn_"$i | tr [:upper:] [:lower:]`; done

Done!