Testing for PHP? Look no further then PHPUnit(http://www.phpunit.de). This framework has all the spunk that junit provides for the java platform.
However once pushing onto a full team, I found things I liked about my homegrown testing platform written in Perl. Perhaps we can take some of PHPUnit and adapt it for our needs?
However phpunit is so tightly integrated into JetBrain's PHPStorm and all the functionality it brings via test suites it be a pain to rewrite, especially for testing. Luckily, the implementation was done in full OO form, so we can subclass the phpunit classes, since its fully OO.
Here's my additional requirements
R-1) For all test we want folders input, expected, output. The former 2 are in version control which defines the test. The later output, is generated for each test run. When finished we want the outputs to be automatically compared to ExpectedOutput which is in version control.
The names of the folder are:
- is a unique test identifier, preferrably alphanumeric.
- is the name of your test.
R-2) The test cases to be NICELY formatted so all test and output are clearly reviewable.
{erformance, time and memory should be dumped after each test. We can diff these runs over time to measure performance and memory impacts as well.
i.e.
R-3) All of this should be summed together into suite summary and reason of failure at the end for any cases
Okay so that is requirements. How do we achieve this?However once pushing onto a full team, I found things I liked about my homegrown testing platform written in Perl. Perhaps we can take some of PHPUnit and adapt it for our needs?
However phpunit is so tightly integrated into JetBrain's PHPStorm and all the functionality it brings via test suites it be a pain to rewrite, especially for testing. Luckily, the implementation was done in full OO form, so we can subclass the phpunit classes, since its fully OO.
Here's my additional requirements
R-1) For all test we want folders input, expected, output. The former 2 are in version control which defines the test. The later output, is generated for each test run. When finished we want the outputs to be automatically compared to ExpectedOutput which is in version control.
The names of the folder are:
output/test_ input/test _ expectedoutput/test _
R-2) The test cases to be NICELY formatted so all test and output are clearly reviewable.
{erformance, time and memory should be dumped after each test. We can diff these runs over time to measure performance and memory impacts as well.
i.e.
BTST_Formatted_TestSuite::setUp()
+++++++++++++++++++++++++++++++++
------------------------------------------------
---------------Starting: 'test0a_enumerationException' Date: '2012-01-24T22:57:04+05:30'
------------------------------------------------
***Memory at End: 12.917344 MB***
Date: 2012-01-24T22:57:05+05:30
Duration: 0.037098169326782
FINISHED with result: SUCCESS()
--------------------------------------------
R-3) All of this should be summed together into suite summary and reason of failure at the end for any cases
BTST_Formatted_TestSuite::tearDown() **************************************************************************** **************** TestSuite BCOM_SegmentFormat_AllTests FAILED **************** **************** Total Cases: 21 **************** Total Passes: 20 **************** Total Failed: 1 **************** Total Errors: 1 **************** Total Assertions: 114 **************** Total Time: 0.52025103569031 s **************************************************************************** There was 1 failure: 1) Warning Test method "test4bHelper" in test class "SegmentFormatTest" is not public. Btst_Formatted_TestSuite.php:35 There was 1 error: 1) SegmentFormatTest::test5b_MsgGrpLoopbackOutOfOrderNominal MultipleFirstSegmentTypeException: Recieved Multiple segments for first segType, abort queue /SomeFile1.php:615 /SomeFile1.php:712 /SomeFile2:468 /SomeFile3:35 E_CORE_WARNING: PHP Startup: Unable to load dynamic library '/usr/local/zend/lib/php_extensions/imagick.so' - libMagickWand.so.2: cannot open shared object file: No such file or directory #0 Unknown Time: 1 second, Memory: 16.75Mb OK (0 tests, 0 assertions)
In section 19 of PhpUnit Manual (http://www.phpunit.de/manual/current/en/extending-phpunit.html), it discusses how to extend.
Okay now we have the design done, lets get to some design details
 | |
| Figure 1: Static UML Diagram for PHPUnit DD |
1) Btst_Framework_TestCase. Here you must save the current name of the test and use that as per (R-1). This class deals with providing generic functions like database loading for all your data. It also deals with getting the input/output/expected directories and comparing files in a folder.
/*
*
* Purpose of class is to allow expected results, inputs and outputs
* to be auto managed
*
*/
class Btst_Framework_TestCase extends PHPUnit_Framework_TestCase
{
const INPUT_DIR = 'inputs';
const OUTPUT_DIR = 'output';
const EXPECTED_DIR = 'expectedoutput';
protected $_fullUnitPath = NULL;
protected $_buildExpectedInputPaths = false;
public function getInputD()
{
if ( ! $this->_fullUnitPath )
{
throw new BsmsObjectNotInitialized("_fullUnitPath" . "\nPlease follow the Setup Before Class as per BtstStackTest");
}
$osCommands = new ButlOsCommands();
$inputDir = $osCommands->joinPath($this->_fullUnitPath , self::INPUT_DIR);
$dir = $osCommands->joinPath($inputDir , $this->getName());
return $dir;
}
public function getOutputD()
{
if ( ! $this->_fullUnitPath )
{
throw new BsmsObjectNotInitialized("_fullUnitPath" . "\nPlease follow the Setup Before Class as per BtstStackTest");
}
$osCommands = new ButlOsCommands();
$inputDir = $osCommands->joinPath($this->_fullUnitPath , self::OUTPUT_DIR);
$dir = $osCommands->joinPath($inputDir , $this->getName());
return $dir;
}
public function getExpectedD()
{
if ( ! $this->_fullUnitPath )
{
throw new BsmsObjectNotInitialized("_fullUnitPath" . "\nPlease follow the Setup Before Class as per BtstStackTest");
}
$osCommands = new ButlOsCommands();
$inputDir = $osCommands->joinPath($this->_fullUnitPath , self::EXPECTED_DIR);
$dir = $osCommands->joinPath($inputDir , $this->getName());
return $dir;
}
//does a recusive rmdir
public static function rrmdir($dir)
{
if (is_dir($dir))
{
$objects = scandir($dir);
foreach ($objects as $object)
{
$osCommands = new ButlOsCommands();
$filePath = $osCommands->joinPath($dir ,$object);
if ($object != "." && $object != "..")
{
if (filetype($filePath) == "dir")
{
self::rrmdir($filePath);
}
else
{
unlink($filePath);
}
}
}
reset($objects);
rmdir($dir);
}
}
public static function setUpBeforeClassClearOutDir( $myUnitTestDir )
{
//@TODO: Need to fix on windows before continuing.
if ( ! $myUnitTestDir )
{
throw new BsmsObjectNotInitialized( $myUnitTestDir . "\nPlease follow the Setup Before Class as per BtstStackTest" );
}
if ( ButlOsEnum::thisOS() == ButlOSEnum::LINUX() || ButlOsEnum::thisOS() == ButlOSEnum::MAC() || ButlOsEnum::thisOS() == ButlOSEnum::WINDOWS() )
{
$osCommands = new ButlOsCommands();
$outdir = $osCommands->joinPath($myUnitTestDir, self::OUTPUT_DIR);
if ( is_dir($outdir) )
{
self::rrmdir( $outdir);
}
}
}
protected function setUp()
{
//@TODO: Need to fix on windows before continuing.
if ( ButlOsEnum::thisOS() == ButlOSEnum::LINUX() || ButlOsEnum::thisOS() == ButlOSEnum::MAC() || ButlOsEnum::thisOS() == ButlOSEnum::WINDOWS() )
{
mkdir( $this->getOutputD(), 0775, true);
if ( $this->_buildExpectedInputPaths)
{
if (!is_dir($this->getInputD()) )
{
mkdir( $this->getInputD(), 0775, true);
}
if (!is_dir($this->getExpectedD()) )
{
mkdir( $this->getExpectedD(), 0775, true);
}
}
}
}
protected function tearDown()
{
//check expected output and fail if
//missing file on either side, or doesn't match.
//Need to fix on windows before continuing.
$memoryInMb = memory_get_usage()/1e6;
$str = '***Memory at End: ' . $memoryInMb . " MB***\n";
print $str;
//@TODO: Need to fix on windows before continuing.
if ( ButlOsEnum::thisOS() == ButlOSEnum::LINUX() || ButlOsEnum::thisOS() == ButlOSEnum::MAC() || ButlOsEnum::thisOS() == ButlOSEnum::WINDOWS() )
{
$filesExp = $this->getAllInDir( $this->getExpectedD() );
$filesNew = $this->getAllInDir( $this->getOutputD() );
$compareFiles = $this->compareDir ($filesExp, $filesNew );
$isPassed = true;
if ( count($filesExp) -> 0 )
{
print "+++Error: Files were missing from output+++\n";
print_r ( array_values($filesExp) );
$isPassed = FALSE;
}
if ( count($filesNew) -> 0 )
{
print "+++Error: Extra Files found in output+++\n";
print_r ( array_values($filesNew) );
$isPassed = false;
}
$notMatchedArray = array();
foreach ($compareFiles as $fileCompare )
{
$osCommands = new ButlOsCommands();
$outputPath = $osCommands->joinPath($this->getOutputD() , $fileCompare);
$expectedPath = $osCommands->joinPath($this->getExpectedD(), $fileCompare);
$rv = $this->compareFiles( $outputPath , $expectedPath);
if (!$rv)
{
array_push($notMatchedArray, $fileCompare);
}
else
{
print "+++Success: $fileCompare +++\n";
}
}
if ( count($notMatchedArray) -> 0 )
{
print "+++Error: Files Contents Not Matched +++\n";
print_r ( array_values($notMatchedArray) );
$isPassed = false;
}
if (!$isPassed)
{
print "To Compare: " . PHP_EOL . "WinmergeU " . $this->getExpectedD() . " " . $this->getOutputD() . PHP_EOL;
}
$this->assertTrue($isPassed, "The Test Results Data did not match expected");
}
}
/**
* This Function compares files into two directory and returns the files that
* are the same. It also returns the input parameters with files that only exist in those directory
* The caller can use this for further decisions.
* The assumption is the directory list is sorted.
* @array $expectedlist
* @array $newList
* @array $comparables
*/
public function compareDir(&$expectedlist, &$newList)
{
//if sorted just go down linearly.
$comparables = array();
$count1 = count($expectedlist);
$count2 = count($newList);
if ($count1 <= 0 || $count2 <= 0 )
{
return array();
}
$i = 0;
$j = 0;
do
{
$cmp = strcmp( $expectedlist[$i], $newList[$j] );
if ( $cmp != 0 )
{
if ( $cmp -> 0)
{
$i++;
}
else
{
$j++;
}
}
else
{
array_push($comparables, $expectedlist[$i]);
array_splice($expectedlist, $i, 1);
array_splice($newList, $j, 1);
}
} while ($i < count($expectedlist) && $j < count($newList) );
return $comparables;
}
public function compareFiles($file1, $file2)
{
$isMatch = FALSE;
$contents1 = file($file1, FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES );
$contents2 = file($file2, FILE_IGNORE_NEW_LINES | FILE_SKIP_EMPTY_LINES );
$t1 = array_diff($contents1, $contents2) ;
$t2 = array_diff($contents2, $contents1) ;
if ( count ( $t1 ) == 0 && count ( $t2 ) == 0)
{
$isMatch = TRUE;
}
return $isMatch;
}
/*
* @string $path
* @array $dir
*/
public function getAllInDir($path)
{
$dir = array();
if ( is_dir($path) )
{
$directory = dir($path);
if (!$directory) return array();
while ( false !== ($entry = $directory->read()) )
{
$osCommands = new ButlOsCommands();
$filePath = $osCommands->joinPath($path , $entry);
if( is_file($filePath))
{
array_push( $dir, $entry);
}
}
}
return $dir;
}
public function readInput($file, $mode)
{
$osCommands = new ButlOsCommands();
$filePath = $osCommands->joinPath($this->getInputD() , $file);
$output = file_get_contents ( $filePath );
return $output;
}
public function writeOutput($file, $output)
{
$osCommands = new ButlOsCommands();
$filePath = $osCommands->joinPath( $this->getOutputD() , $file);
file_put_contents( $filePath, $output);
return TRUE;
}
}
class Btst_SimpleTestListener extends PHPUnit_TextUI_ResultPrinter
{
/**
* @var integer
*/
protected $_numAssertions = 0;
public function addError(PHPUnit_Framework_Test $test,
Exception $e,
$time)
{
printf(
"Error while running test '%s'.\n",
$test->getName()
);
}
public function addFailure(PHPUnit_Framework_Test $test,
PHPUnit_Framework_AssertionFailedError $e,
$time)
{
printf(
"Test '%s' failed.\n",
$test->getName()
);
}
public function addIncompleteTest(PHPUnit_Framework_Test $test,
Exception $e,
$time)
{
printf(
"Test '%s' is incomplete.\n",
$test->getName()
);
}
public function addSkippedTest(PHPUnit_Framework_Test $test,
Exception $e,
$time)
{
printf(
"Test '%s' has been skipped.\n",
$test->getName()
);
}
public function startTest(PHPUnit_Framework_Test $test)
{
$name = $test->getName();
$nowDate = new ButlDateTimeUtils();
$nowDateStr = $nowDate->retrieveUTCFormat();
$buffer = sprintf(
"\n------------------------------------------------" .
"\n---------------Starting: '%s' Date: '%s'" .
"\n------------------------------------------------\n",
$name, $nowDateStr
);
print ($buffer);
}
/**
* @param PHPUnit_Framework_Test $test
* @param float $time
*/
public function endTest(PHPUnit_Framework_Test $test, $time)
{
$result = $test->getResult();
$this->writeGlobalProgress($test);
$nowDate = new ButlDateTimeUtils();
$nowDateStr = $nowDate->retrieveUTCFormat();
//$success = $result->wasSuccessful();
$msg = $test->getStatusMessage();
$success = $test->hasFailed() == TRUE ? "FAIL":"SUCCESS";
printf(
"\n-->Date: $nowDateStr " .
"\n-->Duration: $time " .
"\n-->FINISHED with result: $success($msg)"
);
}
public function startTestSuite(PHPUnit_Framework_TestSuite $suite)
{
printf(
"\n****************************************************************************" .
"\n****************TestSuite '%s' started" .
"\n****************************************************************************\n" ,
$suite->getName()
);
}
public function endTestSuite(PHPUnit_Framework_TestSuite $suite)
{
if ($suite instanceOf BTST_Formatted_TestSuite)
{
$results = $suite->_getSuiteResult();
$results = $results;
$resultStr = $results->wasSuccessful() == TRUE ? "SUCCESSFULL":"FAILED";
printf(
"\n****************************************************************************".
"\n**************** TestSuite " . $suite->getName() . " " . $resultStr .
"\n****************" .
"\n**************** Total Cases: ". $results->count() .
"\n**************** Total Passes: ". ($results->count() - $results->failureCount()).
"\n**************** Total Failed: ". $results->failureCount() .
"\n**************** Total Errors: ". count($results->errors()) .
"\n**************** Total Assertions: ". $this->_numAssertions .
"\n**************** Total Time: ". $results->time() . " s" .
"\n****************************************************************************\n");
$this->printFailures($results);
$this->printErrors($results);
}
else
{
printf("\n**************** TestSuite " . $suite->getName() . " Finished");
}
//Results
}
/**
*
*/
public function writeGlobalProgress($testCase)
{
$this->_numAssertions += $testCase->getNumAssertions();
}
}
3) The TestSuite is required to attach a listener to the TestResult that we defined above. We can setup other things like how errors are shown and strict mode testing
class Btst_Formatted_TestSuite extends PHPUnit_Framework_TestSuite
{
/*
* This override the createResult to include our customer listner
*
*/
protected $_suiteResult = NULL; //hold the result so we can use it later
public function createResultForListner()
{
$this->_suiteResult = new PHPUnit_Framework_TestResult;
$this->_suiteResult->strictMode(TRUE);
$this->_suiteResult->convertErrorsToExceptions(TRUE);
$this->_suiteResult->addListener(new Btst_SimpleTestListener);
return $this->_suiteResult;
}
public function _getSuiteResult()
{
return $this->_suiteResult;
}
/*
* For some reason our createResult does not get called, so I had to override run.
*
*/
public function run(PHPUnit_Framework_TestResult $result = NULL, $filter = FALSE, array $groups = array(), array $excludeGroups = array(), $processIsolation = FALSE)
{
$result = $this->createResultForListner();
PHPUnit_Framework_TestSuite::run($result, $filter, $groups, $excludeGroups, $processIsolation);
}
protected function setUp()
{
print "\nBTST_Formatted_TestSuite::setUp()";
print "\n+++++++++++++++++++++++++++++++++";
}
protected function tearDown()
{
print "\n+++++++++++++++++++++++++++++++++";
print "\nBTST_Formatted_TestSuite::tearDown()";
}
}
4) Finally here's a typical test case. There is not much change except we can save outputs and let the rest do its magic. You can also run the without the special formatting if you want to revert back to phpunit default settings.
/*
* Test Driver for the ArrayTest test suite
* phpunit --verbose BCOM_SegmentFormat_AllTests BcomSegmentFormatTest.php
* The following line will run without the all the extensions.
* phpunit --verbose BcomSegmentFormatTest.php
*
*/
class BCOM_SegmentFormat_AllTests
{
public static function suite()
{
$suite = new Btst_Formatted_TestSuite('BcomSegmentFormatTest');
return $suite;
}
public static function suiteRollUp()
{
$suite = new PHPUnit_Framework_TestSuite('BcomSegmentFormatTest');
return $suite;
}
}
class BcomSegmentFormatTest extends Btst_Framework_TestCase
{
//*************INSERT THESE THREE FUNCTIONS IN ALL TEST CASES*************************//
static function setUpBeforeClass()
{
Btst_Framework_TestCase::setUpBeforeClassClearOutDir( realpath(dirname(__FILE__) ) );
//Need to do special on windows!!!
Btst_Framework_TestCase::loadDatabase_Helper(true, "password", "default");
}
protected function setUp()
{
_fullUnitPath = realpath(dirname(__FILE__) );
Btst_Framework_TestCase::setUp();
}
//**********************************************************************************//
protected function tearDown()
{
Btst_Framework_TestCase::tearDown();
//do gnokii tear down
}
public function test10_testLocalIncomingDropbox()
{
$allInputs = $this->getAllInDir ( $this->getInputD() );
$file = doStuff($allInputs);
copy( $file, $this->getOutputD() );
}
So there you go. The output at the end now outputs more information about the failures so you don't have to go trolling through the log. Also the formatting breaks of +++ and **** allow for better visual clues about which output is associated with a given test, and allow for further processing as needed. So thats about it. If you want to also run an entire test suite from a top level acceptance, the following snippet can be used.
/*
* Runs all tests in our acceptance suite.
* Create a test suite that contains the tests
* from the ArrayTest class.
* Just in one of the following ways.
* phpunit --verbose AcceptanceTestSuite.php
* phpunit --filter test2_pop AcceptanceTestSuite.php
*/
class AcceptanceTestSuite
{
public static function suite()
{
$suite = new Btst_Formatted_TestSuite('Executing ... Your Test Suite');
$suite->addTest(BCOM_SegmentFormat_AllTests::suiteRollUp());
$suite->addTest(BBBB_Business_AllTests::suiteRollUp());
return $suite;
}
}
thanks for the great post...we are looking into integrating test cases in our project right now
ReplyDelete