Potential adoption of run-tests.php replacement

  101721
January 29, 2018 12:57 nazar@mokrynskyi.com (Nazar Mokrynskyi)
Hi internals,

As most of you know, there is this script called run-tests.php which is huge in size and hard to maintain.

I also heard some wanted it to support parallel tests execution and probably other things.

There is a project I've created a long time ago and still maintaining (https://github.com/nazar-pc/phpt-tests-runner) that should be able to run most of the tests without changes already with colored output and the whole thing is a fraction of run-tests.php's size.

How I'd like to work towards mainlining it into php-src in case there is a desire and an agreement on such transition.

If this seems like a good idea, it would be very useful if someone can enumerate the list of features that are currently used/needed from run-tests.php and which replacement should offer, alongside with features on top that might be useful (like parallel execution).

If there was similar discussion already and there is an accepted solution there, let me know.

-- 
Sincerely, Nazar Mokrynskyi
github.com/nazar-pc
  101726
January 29, 2018 22:10 themattficken@gmail.com (Matt Ficken)
This is a reimplementation of run-test.php, not just a wrapper (so its
another parser for the PHPT file format). However, its missing support for
many PHPT sections (ie its PHPT format support is incomplete).

See:
https://github.com/nazar-pc/phpt-tests-runner/blob/master/bin/phpt-tests-runner#L117


I created PFTT as a multi-SAPI (CLI, IIS/FastCGI, CLI-WWW, Apache)
multiplatform test tool reimplementing PHPT and PHPUnit(and support for
other common tasks)...

This may give you some clarification of what's missing
http://git.php.net/?p=pftt2.git;a=blob_plain;f=src/com/mostc/pftt/model/core/EPhptSection.java;hb=refs/heads/master

PFTT is a big heavyweight tool (so PFTT shouldn't be in core).
run-tests.php is nice because its a lightweight tool, not overly complex
and minimal to download, so I can easily do `make test` for a custom
build... its just not as good at seriously pounding PHP across multiple
scenarios, SAPIs, filesystems, etc... (built in part to hammer PHP on
Windows to ensure it works well).  Do we really need a 3rd PHPT engine?

It also supports parallel execution (it tests many different scenarios, so
faster execution is critical)... that's a feature that's hard to get right.

-Regards
-M


On Mon, Jan 29, 2018 at 4:57 AM, Nazar Mokrynskyi <nazar@mokrynskyi.com>
wrote:

> Hi internals, > > As most of you know, there is this script called run-tests.php which is > huge in size and hard to maintain. > > I also heard some wanted it to support parallel tests execution and > probably other things. > > There is a project I've created a long time ago and still maintaining ( > https://github.com/nazar-pc/phpt-tests-runner) that should be able to run > most of the tests without changes already with colored output and the whole > thing is a fraction of run-tests.php's size. > > How I'd like to work towards mainlining it into php-src in case there is a > desire and an agreement on such transition. > > If this seems like a good idea, it would be very useful if someone can > enumerate the list of features that are currently used/needed from > run-tests.php and which replacement should offer, alongside with features > on top that might be useful (like parallel execution). > > If there was similar discussion already and there is an accepted solution > there, let me know. > > -- > Sincerely, Nazar Mokrynskyi > github.com/nazar-pc > > > -- > PHP Internals - PHP Runtime Development Mailing List > To unsubscribe, visit: http://www.php.net/unsub.php > >
  101727
January 29, 2018 22:41 nazar@mokrynskyi.com (Nazar Mokrynskyi)
It was never intended to be a wrapper or to be 100% compatible.
It was created as a simpler and modern replacement with additional features like colored output (tried to add this to run-tests.php initially, even submitted PR with some fixes and refactoring, but found it too complex to deal with) and simpler CLI interface to work with.

It is intentionally missing a few PHPT sections that seem to be redundant and have slight tweaks to some that are present. At the end of the day so many time have passed since run-tests.php was initially written.

For instance, it is painful to work with tests in PHP source tree when they are called 001.phpt, 002.phpt and so on - you need to open each file and read it to see what it intends to test. My tool uses file names instead, which makes browsing source tree with tests more productive and makes --TEST-- section redundant (simply ignored).

This tool is 6.5 times smaller than run-tests.php and imho is better structured, which leads to lower cost for introducing new features when/if needed.

Sincerely, Nazar Mokrynskyi
github.com/nazar-pc

30.01.18 00:10, Matt Ficken пише:
> This is a reimplementation of run-test.php, not just a wrapper (so its another parser for the PHPT file format). However, its missing support for many PHPT sections (ie its PHPT format support is incomplete). > > See: https://github.com/nazar-pc/phpt-tests-runner/blob/master/bin/phpt-tests-runner#L117 > > > I created PFTT as a multi-SAPI (CLI, IIS/FastCGI, CLI-WWW, Apache) multiplatform test tool reimplementing PHPT and PHPUnit(and support for other common tasks)... > > This may give you some clarification of what's missing > http://git.php.net/?p=pftt2.git;a=blob_plain;f=src/com/mostc/pftt/model/core/EPhptSection.java;hb=refs/heads/master > > PFTT is a big heavyweight tool (so PFTT shouldn't be in core).  run-tests.php is nice because its a lightweight tool, not overly complex and minimal to download, so I can easily do `make test` for a custom build.... its just not as good at seriously pounding PHP across multiple scenarios, SAPIs, filesystems, etc... (built in part to hammer PHP on Windows to ensure it works well).  Do we really need a 3rd PHPT engine? > > It also supports parallel execution (it tests many different scenarios, so faster execution is critical)... that's a feature that's hard to get right. > > -Regards > -M > > > On Mon, Jan 29, 2018 at 4:57 AM, Nazar Mokrynskyi <nazar@mokrynskyi.com <mailto:nazar@mokrynskyi.com>> wrote: > > Hi internals, > > As most of you know, there is this script called run-tests.php which is huge in size and hard to maintain. > > I also heard some wanted it to support parallel tests execution and probably other things. > > There is a project I've created a long time ago and still maintaining (https://github.com/nazar-pc/phpt-tests-runner <https://github.com/nazar-pc/phpt-tests-runner>) that should be able to run most of the tests without changes already with colored output and the whole thing is a fraction of run-tests.php's size. > > How I'd like to work towards mainlining it into php-src in case there is a desire and an agreement on such transition. > > If this seems like a good idea, it would be very useful if someone can enumerate the list of features that are currently used/needed from run-tests.php and which replacement should offer, alongside with features on top that might be useful (like parallel execution). > > If there was similar discussion already and there is an accepted solution there, let me know. > > -- > Sincerely, Nazar Mokrynskyi > github.com/nazar-pc <http://github.com/nazar-pc> > > > -- > PHP Internals - PHP Runtime Development Mailing List > To unsubscribe, visit: http://www.php.net/unsub.php > >
  101736
January 30, 2018 22:26 smalyshev@gmail.com (Stanislav Malyshev)
Hi!

> For instance, it is painful to work with tests in PHP source tree when they are called 001.phpt, 002.phpt and so on - you need to open each file and read it to see what it intends to test. My tool uses file names instead, which makes browsing source tree with tests more productive and makes --TEST-- section redundant (simply ignored).
I think --TEST-- should absolutely be supported. Filename is not nearly enough to properly describe the content of the test. Something like "Test whether foo(bar($baz)) crashes if called more than twice (bug #1234)" is not going to be captured in a filename. Filenames and informative descriptions are not the same thing. I'll look more into what other requirements are needed (one suggestion would be try to keep option compatibility with the old tool if possible, so people could use their old habits) and write later, but I think supporting proper test descriptions would definitely be one of them. -- Stas Malyshev smalyshev@gmail.com
  101738
January 31, 2018 10:50 nazar@mokrynskyi.com (Nazar Mokrynskyi)
The idea is to make filenames more descriptive and sufficient for displaying in test results. This doesn't prevent or discourage anyone from including description in test file itself when more details need to be included.

Moreover, all of the characters from mentioned title are allowed for use in file name (at least on Linux) and it would be much more useful to see them instead of 001.phpt or bug-1234.phpt.

Sincerely, Nazar Mokrynskyi
github.com/nazar-pc

31.01.18 00:26, Stanislav Malyshev пише:
> Hi! > >> For instance, it is painful to work with tests in PHP source tree when they are called 001.phpt, 002.phpt and so on - you need to open each file and read it to see what it intends to test. My tool uses file names instead, which makes browsing source tree with tests more productive and makes --TEST-- section redundant (simply ignored). > I think --TEST-- should absolutely be supported. Filename is not nearly > enough to properly describe the content of the test. Something like > "Test whether foo(bar($baz)) crashes if called more than twice (bug > #1234)" is not going to be captured in a filename. Filenames and > informative descriptions are not the same thing. > > I'll look more into what other requirements are needed (one suggestion > would be try to keep option compatibility with the old tool if possible, > so people could use their old habits) and write later, but I think > supporting proper test descriptions would definitely be one of them. >
  101739
January 31, 2018 12:17 rowan.collins@gmail.com (Rowan Collins)
On 31 January 2018 at 10:50, Nazar Mokrynskyi <nazar@mokrynskyi.com> wrote:

> The idea is to make filenames more descriptive and sufficient for > displaying in test results. This doesn't prevent or discourage anyone from > including description in test file itself when more details need to be > included. >
Would you propose to apply a bulk rename of all .phpt files currently in the tree to use their TEST blocks as filenames, or a manual review of each test? If automating, how would you handle tests whose descriptions are not unique within a directory? If manual, what kind of naming scheme would you propose?
> Moreover, all of the characters from mentioned title are allowed for use > in file name (at least on Linux) and it would be much more useful to see > them instead of 001.phpt or bug-1234.phpt.
While most characters are allowed in names, some can be irritating to work with - test descriptions are likely to include things like ", $, * etc which have to be escaped carefully whenever manipulating the files. Different filesystems and environments may not handle the same set of characters, so care would need to be taken to ensure portability of the test suite. It seems like a tool which lists the file name and description for each ..phpt file in a directory would bring most of the advantages of such a mass renaming without the problems. Regards, -- Rowan Collins [IMSoP]
  101743
February 1, 2018 23:02 smalyshev@gmail.com (Stanislav Malyshev)
Hi!

On 1/31/18 2:50 AM, Nazar Mokrynskyi wrote:
> The idea is to make filenames more descriptive and sufficient for > displaying in test results.
Descriptive - sure, that'd be great, why not. Sufficient - won't happen. Filenames are not meant to describe complex things, are not suitable for it and we should not try to put it there. It just won't work.
> This doesn't prevent or discourage anyone from including description in test file itself when more details need to be included.
Thus, the necessity of supporting --TEST--.
> Moreover, all of the characters from mentioned title are allowed for > use in file name (at least on Linux) and it would be much more useful > to see them instead of 001.phpt or bug-1234.phpt.
Yes, but I do not want to have files with such names. And neither, I expect, do our users - weird filenames are hard to work with, break some common tools and require extraordinary care to handle them properly. To illustrate, right now git (tool that is behind Linux kernel, which is behind pretty much everything on the Internet) can't properly process some of our test filenames. We don't need more trouble in this department. We certainly don't need to force people to name their files "Test whether foo(bar($baz)) crashes if it's called more than twice (bug #1234).phpt" - that would be a nightmare to work with. We should not be using filenames for things they weren't meant to be used for. -- Stas Malyshev smalyshev@gmail.com
  101760
February 4, 2018 09:53 smalyshev@gmail.com (Stanislav Malyshev)
Hi!

> There is a project I've created a long time ago and still maintaining (https://github.com/nazar-pc/phpt-tests-runner) that should be able to run most of the tests without changes already with colored output and the whole thing is a fraction of run-tests.php's size.
Some notes: - I already wrote about TEST support, that stays unchanged. - It would be nice to preserve option-compatibility with the old version, if possible. People have scripts and trained keyboard-memory, and migrating would be easier without looking up new options each time. That doesn't mean every option should be supported, but for those that are, it'd be nice to keep the same names. - Would be nice to support in options: -- Passing options to PHP binary (at least -n and -d, but potentially all others) -- Keeping temp files - Section support - I think besides TEST, the following will be needed: * XFAIL: unfortunately, those are still needed * ENV: there are a lot of environment-dependent tests as it seems * FILE_EXTERNAL: is pretty widely used * The group of GET,POST,PUT,POST_RAW,GZIP_POST,GZIP_POST,COOKIE,HEADERS,CGI - necessary for some functionality related to HTTP processing. But most of the tests -- Stas Malyshev smalyshev@gmail.com
  101767
February 5, 2018 10:44 johannes@schlueters.de (Johannes =?ISO-8859-1?Q?Schl=FCter?=)
On So, 2018-02-04 at 01:53 -0800, Stanislav Malyshev wrote:
> - Section support - I think besides TEST, the following will be > needed: > * XFAIL: unfortunately, those are still needed > * ENV: there are a lot of environment-dependent tests as it seems > * FILE_EXTERNAL: is pretty widely used > * The group of > GET,POST,PUT,POST_RAW,GZIP_POST,GZIP_POST,COOKIE,HEADERS,CGI - > necessary > for some functionality related to HTTP processing. But most of the > tests
Also --REDIRECTTEST-- as used in PDO drivers is needed or requires a replacement. johannes