Overview

ktest is a tool that makes kphp programs easier to test.

• ktest phpunit can run PHPUnit tests using KPHP
• ktest compare run given script with PHP and KPHP, check that output is identical
• ktest bench run benchmarks using KPHP
• ktest bench-ab run two selected benchmarks using KPHP and compare their results
• ktest bench-php run benchmarks using PHP
• ktest bench-vs-php run benchmarks using both KPHP and PHP, compare the results
• ktest benchstat compute and compare statistics about benchmark results (see benchstat)
• ktest env print ktest-related env variables

Installation

$ composer require --dev vkcom/ktest-script

If composer is not an option or you want to install ktest binary globally, consider these options:

• Download the ktest binary from the latest release
• or build ktest from sources

You may need to set KPHP_ROOT environment variable to point to your KPHP repository folder:

$ git clone https://github.com/VKCOM/kphp.git
$ export KPHP_ROOT=$(pwd)/kphp

Example - phpunit

Imagine that we have an ordinary PHPUnit test:

<?phpusePHPUnit\Framework\TestCase;
useExampleLib\Strings;
class StringsTest extends TestCase {
publicfunctiontestContains() {
$this->assertSame(Strings::contains('foo', 'bar'), false);
$this->assertTrue(Strings::contains('foo', 'foo'));
}
publicfunctiontestHasPrefix() {
$this->assertSame(Strings::hasPrefix('Hello World', 'Hello'), true);
$this->assertFalse(Strings::hasPrefix('Hello World', 'ello'));
}
}

It comes without a surprise that you can run it with phpunit tool:

$ ./vendor/bin/phpunit tests
...... 6 / 6 (100%)
Time: 70 ms, Memory: 4.00 MB
OK (6 tests, 14 assertions)

When you're using phpunit, tests are executed as PHP, not KPHP.

ktest makes it possible to run your phpunit-compatible tests with KPHP:

$ ./vendor/bin/ktest phpunit tests
.... 4 / 6 (66%) OK
.. 6 / 6 (100%) OK
Time: 10.74657386s
OK (6 tests, 14 assertions)

Note that running KPHP tests is slower: a separate binary is compiled per every Test class.

All you need is ktest utility and installed kphpunit package:

$ composer require --dev vkcom/kphpunit

Now let's do something more exciting.

Take a look at this Integers::getFirst method:

<?phpnamespaceFoo\Bar;
class Integers {
/** @param int[] $xs */publicstaticfunctiongetFirst(array$xs) {
return$xs[0];
}
}

It's intended to return the first int array item, or null, if index 0 is unset.

We can write a test for this method:

<?phpusePHPUnit\Framework\TestCase;
useFoo\Bar\Integers;
class IntegersTest extends TestCase {
publicfunctiontestGetFirst() {
$this->assertSame(Integers::getFirst([]), null);
$this->assertSame(Integers::getFirst([1]), 1);
}
}

All tests are passing:

. 1 / 1 (100%)
Time: 36 ms, Memory: 4.00 MB
OK (1 test, 2 assertions)

But if you try to run it with ktest, you'll see how that code would behave in KPHP:

F 1 / 1 (100%) FAIL
Time: 4.59874429s
There was 1 failure:
1) IntegersTest::testGetFirst
Failed asserting that null is identical to 0.
IntegersTest.php:8
FAILURES!
Tests: 1, Assertions: 1, Failures: 1.

Accessing unset array index can yield a "zero value" instead of null.

Running with ktest makes it easier to ensure that your code behaves identically in both PHP and KPHP.

Example - bench

There are 2 main ways to do benchmarking with bench subcommand:

1. Run different benchmarks and see how they relate
2. Run benchmarks by the same name and compare samples with benchstat

Let's assume that you have a function that concatenates 3 strings. You can write a benchmark for it:

<?php// file "BenchmarkConcat3.php"class BenchmarkConcat3 {
privatestatic$strings = [
'foo',
'bar',
'baz',
];
publicfunctionbenchmarkConcat() {
returnself::$strings[0] . self::$strings[1] . self::$strings[2];
}
}

This benchmark can be executed with a bench subcommand:

$ ./vendor/bin/ktest bench BenchmarkConcat3.php
class BenchmarkConcat3
BenchmarkConcat3::benchmarkConcat	106500	372.0 ns/op
ok BenchmarkConcat3 147.153797ms

Suppose that somebody proposed to re-write this function with ob_start() claiming that it would make things faster.

First, we need to collect samples of the current implementation. We need at least 5 rounds, but usually the more - the better (don't get too crazy though, 10 is good enough in most cases).

$ ./vendor/bin/ktest bench -count 5 BenchmarkConcat3.php | tee old.txt

Now we have old implementation results, it's time to roll the a implementation:

<?phpclass BenchmarkConcat3 {
privatestatic$strings = [
'foo',
'bar',
'baz',
];
publicfunctionbenchmarkConcat() {
ob_start();
echoself::$strings[0];
echoself::$strings[1];
echoself::$strings[2];
returnob_get_clean();
}
}

Now we need to collect the new implementation results:

$ ./vendor/bin/ktest bench -count 5 BenchmarkConcat3.php | tee new.txt

When you have 2 sets of samples, it's possible to compare them with benchstat:

$ ./vendor/bin/ktest benchstat old.txt new.txt
name old time/op new time/op delta
Concat 372ns ± 2% 546ns ± 6% +46.91% (p=0.008 n=5+5)

As we can see, the new implementation is, in fact, almost 2 times slower!

TODO

• Mocks

Limitations

• Assert functions can't be used for objects (class instances)
• No custom comparators for assert functions