Automatic performance benchmarking: PHP 8.1 is ~30% faster than PHP 7.4

  116323
November 12, 2021 10:18 kocsismate90@gmail.com (=?UTF-8?B?TcOhdMOpIEtvY3Npcw==?=)
Hello Internals,

I'm writing this email because lately, I've been working on an automatic
benchmarking framework for PHP, and I'd like to share some news regarding
it. The initial implementation
was sponsored by Craig Francis, and it was used for the evaluation of the
performance aspects of the is_literal() RFC. Since then, I fixed numerous
issues and implemented many new features, so with the very close advent of
PHP 8.1, the time has come to unveil it.

I'm sure that many of you still remember Intel's automatic benchmarks from
a couple of years ago (e.g. https://externals.io/message/89843#89843). I
loved these emails, so this project served as a great inspiration for me
to start working on something similar. I have to admit
though that I won't be able to replicate their extremely advanced setup.

My main goal was to develop a framework (
https://github.com/kocsismate/php-version-benchmarks) which is:
- fully automatic so that it can be easily run regularly, and the
benchmarks are reproducible
- it's possible to try it out locally via Docker, but it can be run in the
cloud (currently, only AWS is
supported), and the instance type is configurable
- supports different CPU platforms (X86-64 and ARM64), and advanced options
whether
turbo boost/hyper threading/deeper CPU C-states are enabled is configurable
- supports any version of PHP since PHP 7.4, including any git branches or
commits
- supports the most important PHP configurations, including whether
OPcache/JIT/preloading
is enabled
- supports multiple tests: currently, the micro benchmarks bundled with
php-src, as well as the
Symfony and the Laravel demo sites as "real-life" tests are included with
some customizability (number of warmups, iterations number, number of
requests)
- results are measured via using PHP-CGI which is a lightweight web server
with very a small overhead

The most current benchmark is available at
https://github.com/kocsismate/php-version-benchmarks/blob/main/docs/results/2021_11_11_09_20_1_aws_arm64_c6g_4xlarge/result.md
.. I'm happy to share that the results suggest PHP 8.1.0 is around 28-32%
faster than PHP 7.4 in real-life tests, and a few percent faster when it
comes to micro benchmarks. The benchmark was performed on an ARM64 instance
because this platform provided much more stable results than X86-64-based
ones did, mainly due to their fixed CPU frequency.

Unfortunately, the benchmark is not yet suitable for detecting minor
performance changes between commits as the run-to-run variation of the
results is a bit too high; in some cases, it can be up to 1-2%. I hope that
this problem can be mitigated to an acceptable level in the future, but
most probably I'll need some help to achieve this. So any help is
appreciated!

Furthermore, I'd have some questions regarding this project:
- Would you welcome automatic emails on internals (or on a dedicated
mailing list), just like how
Intel did it? I'm not sure about the frequency, but in my opinion,
1-2/month would be a sensible one, given there is interest in it.
- Does anyone know how to get some sponsorship from AWS (or from some other
cloud
provider at last resort)? I already tried to apply for AWS Activate's free
credits, but my application was rejected due to different reasons
(normally, this program is available for startups). I have some hope that
running the benchmark on a dedicated instance would decrease variation of
the results, but enabling this feature is relatively expensive compared to
other costs, so I'd be very happy if I wouldn't have to sponsor all of
this, when I already dedicate a great deal of my time to the cause.
- As always, I'm happy to receive feedback and improvements (ideally along
with an implementation :) ). For example, currently I've started working on
some visualization and chart support, but frontend stuff is not my area of
expertise, so I'm progressing a bit slower than normally. Also, the
statistical foundations could be reviewed, and possibly improved.

Regards:
Máté
  116339
November 13, 2021 11:25 come@chilliet.eu (=?ISO-8859-1?Q?C=F4me_Chilliet?=)
Le 12 novembre 2021 11:18:50 GMT+01:00, "Máté Kocsis" <kocsismate90@gmail.com> a écrit :
> >Furthermore, I'd have some questions regarding this project: >- Would you welcome automatic emails on internals (or on a dedicated >mailing list), just like how >Intel did it? I'm not sure about the frequency, but in my opinion, >1-2/month would be a sensible one, given there is interest in it.
I would favor having this on a website, with an atom/rss feed. But I'm not against having it on the ML, but more like when there is an RC or when some change have significant impact.
>- As always, I'm happy to receive feedback and improvements (ideally >along >with an implementation :) ). For example, currently I've started >working on >some visualization and chart support, but frontend stuff is not my area >of >expertise, so I'm progressing a bit slower than normally. Also, the >statistical foundations could be reviewed, and possibly improved.
The link to the results in the readme points to https://kocsismate.github.io/php-version-benchmarks/index.html which is empty. Maybe this is known. Charts This quite nice! Côme
  116341
November 13, 2021 12:31 craig@craigfrancis.co.uk (Craig Francis)
On Fri, 12 Nov 2021 at 10:19, Máté Kocsis <kocsismate90@gmail.com> wrote:

> Furthermore, I'd have some questions regarding this project: > - Would you welcome automatic emails on internals (or on a dedicated > mailing list), just like how Intel did it? I'm not sure about the > frequency, but in my opinion, 1-2/month would be a sensible one, given > there is interest in it.
I'm sure you know this already, but I'd be interested... I quite like knowing when regressions have happened as soon as possible; and having something which is keeping an eye on things, would hopefully highlight issues sooner. - Does anyone know how to get some sponsorship from AWS... Very funny, Amazon paying :-P - As always, I'm happy to receive feedback and improvements (ideally along
> with an implementation :) ). For example, currently I've started working on > some visualization and chart support, but frontend stuff is not my area of > expertise...
For my sins, I can do some frontend stuff... are we talking CSS, SVG, and maybe a bit of the old JavaScript? If so, feel free to contact me off-list, I probably can't do anything in the next week or two, but I'm optimistically (naively) hoping I can clear some of my todo list :-) Craig
  116343
November 13, 2021 12:40 deleugyn@gmail.com (Deleu)
On Sat, Nov 13, 2021 at 1:31 PM Craig Francis <craig@craigfrancis.co.uk>
wrote:

> On Fri, 12 Nov 2021 at 10:19, Máté Kocsis <kocsismate90@gmail.com> wrote: > > - Does anyone know how to get some sponsorship from AWS... > > Craig >
You may fill out this form https://pages.awscloud.com/AWS-Credits-for-Open-Source-Projects to apply for 12 months credit. If I'm not mistaken it could be a recurring thing by re-applying every 12 months. It might be important to have a dedicated account only for the purpose of the benchmark. You can also read more about it here: https://aws.amazon.com/blogs/opensource/aws-promotional-credits-open-source-projects/ -- Marco Aurélio Deleu
  116471
November 22, 2021 08:41 kocsismate90@gmail.com (=?UTF-8?B?TcOhdMOpIEtvY3Npcw==?=)
Thanks all for the answers!

> AWS is meant to have credits for open source projects: https://aws.amazon.com/blogs/opensource/aws-promotional-credits-open-source-projects/
Nice! An AWS support representative suggested me that I should apply for AWS Activate (https://aws.amazon.com/activate/), which clearly doesn't seem the best program for my purpose, so I'll have a try instead with the promotional credits. Thanks for the idea!
> The link to the results in the readme points to https://kocsismate.github.io/php-version-benchmarks/index.html which is
empty. Maybe this is known. Thanks for the heads up, I forgot that this URL is linked when I posted the benchmark. This is the page where I'd like to display the visualization I mentioned in my previous message. I'll need some time until I finish it, but until then, the current results can be accessed in an markdown format at https://github.com/kocsismate/php-version-benchmarks/blob/main/docs/results/2021_11_11_09_20_1_aws_arm64_c6g_4xlarge/result.md
> For my sins, I can do some frontend stuff... are we talking CSS, SVG, and maybe a bit of the old JavaScript? If so, feel free to contact me off-list,
> I probably can't do anything in the next week or two, but I'm optimistically (naively) hoping I can clear some of my todo list :-)
Thanks for the offer, I'll DM you! :) What I already have locally is a very-very basic chart of the most recent benchmark results. I currently use chart.js (https://www.chartjs.org/) and a fork of semantic-ui (https://fomantic-ui.com/), only because I am familiar with these libraries, so I'm not very attached to them, if there are libraries more suitable for the job.
> Overall against having it on the ML: I set up a filter to ignore those pesky emails by Intel. As much as I appreciate their effort, they were
really noisy, and not really useful to the larger group. I appreciate your feedback! I've just realized that there's probably as many people looking forward to new benchmark results as who are not interested in them at all, so I won't propose sending a regular mail about the results to the internals list. Regards: Máté Deleu <deleugyn@gmail.com> ezt írta (időpont: 2021. nov. 13., Szo, 13:40):
> > > On Sat, Nov 13, 2021 at 1:31 PM Craig Francis <craig@craigfrancis.co.uk> > wrote: > >> On Fri, 12 Nov 2021 at 10:19, Máté Kocsis <kocsismate90@gmail.com> wrote: >> >> - Does anyone know how to get some sponsorship from AWS... >> >> Craig >> > > You may fill out this form > https://pages.awscloud.com/AWS-Credits-for-Open-Source-Projects to apply > for 12 months credit. If I'm not mistaken it could be a recurring thing by > re-applying every 12 months. It might be important to have a dedicated > account only for the purpose of the benchmark. > > You can also read more about it here: > https://aws.amazon.com/blogs/opensource/aws-promotional-credits-open-source-projects/ > > -- > Marco Aurélio Deleu >
  116513
November 25, 2021 18:16 kocsismate90@gmail.com (=?UTF-8?B?TcOhdMOpIEtvY3Npcw==?=)
Sorry Folks, but I have to provide some update about the results:

Unfortunately, due to a silly bug in my calculator code, the % differences
in the benchmark results were slightly off from their real values, so I
have just retroactively adjusted them.
In fact, the Symfony demo app on PHP 8.1 is ~23% faster than on PHP 7.4,
while Laravel runs ~21.5% faster on PHP 8.1 than on PHP 7.4. In my opinion,
this performance increase
is still very remarkable, so I'm not disappointed at all that the
improvement is slightly less dramatic than I previously suggested. In any
case, sorry for the misleading news!

I wish a happy PHP 8.1 release party to everyone:
Máté
  116515
November 25, 2021 18:26 mweierophinney@gmail.com ("Matthew Weier O'Phinney")
On Thu, Nov 25, 2021, 12:17 PM Máté Kocsis <kocsismate90@gmail.com> wrote:

> Sorry Folks, but I have to provide some update about the results: > > Unfortunately, due to a silly bug in my calculator code, the % differences > in the benchmark results were slightly off from their real values, so I > have just retroactively adjusted them. > In fact, the Symfony demo app on PHP 8.1 is ~23% faster than on PHP 7.4, > while Laravel runs ~21.5% faster on PHP 8.1 than on PHP 7.4. In my opinion, > this performance increase > is still very remarkable, so I'm not disappointed at all that the > improvement is slightly less dramatic than I previously suggested.
Honestly, anything over 10% improvement when it comes to performance, particularly in a programming language, is significant. 20% can have a huge impact on production loads! In any
> case, sorry for the misleading news! > > I wish a happy PHP 8.1 release party to everyone: > Máté >
  116555
December 2, 2021 10:59 patrickallaert@php.net (Patrick ALLAERT)
Le jeu. 25 nov. 2021 à 19:17, Máté Kocsis <kocsismate90@gmail.com> a écrit :
> > Sorry Folks, but I have to provide some update about the results: > > Unfortunately, due to a silly bug in my calculator code, the % differences > in the benchmark results were slightly off from their real values, so I > have just retroactively adjusted them. > In fact, the Symfony demo app on PHP 8.1 is ~23% faster than on PHP 7.4, > while Laravel runs ~21.5% faster on PHP 8.1 than on PHP 7.4. In my opinion, > this performance increase > is still very remarkable, so I'm not disappointed at all that the > improvement is slightly less dramatic than I previously suggested. In any > case, sorry for the misleading news! > > I wish a happy PHP 8.1 release party to everyone: > Máté
Hi Máté, Thank you for those stats. Would you have the chance to have some data including PHP 8.0 as well? Because without PHP 8.0 data, we can't deduce anything about what PHP 8.1 brings over PHP 8.0 and or 8.0 over 7.4. Virtually, 8.1 may even be slower than 8.0 if the big increase of performance happened in 8.0. Cheers, Patrick
  116573
December 6, 2021 09:00 kocsismate90@gmail.com (=?UTF-8?B?TcOhdMOpIEtvY3Npcw==?=)
Hi Patrick,


> Thank you for those stats. Would you have the chance to have some data > including PHP 8.0 as well? > Because without PHP 8.0 data, we can't deduce anything about what PHP > 8.1 brings over PHP 8.0 and or 8.0 over 7.4. > Virtually, 8.1 may even be slower than 8.0 if the big increase of > performance happened in 8.0. >
Have you seen the results page? It is available at https://github.com/kocsismate/php-version-benchmarks/blob/main/docs/results/2021_11_25_10_08_1_aws_arm64_c6g_4xlarge/result.md and it lists the exact results for each release. Besides, the PHP 8.1 release page ( https://www.php.net/releases/8.1/en.php#performance_improvements) displays the performance improvement based on PHP 8.0 and also includes some of the optimizations which led to the boost. Cheers, Máté
  116574
December 6, 2021 09:35 patrickallaert@php.net (Patrick ALLAERT)
Hi Máté,

Le lun. 6 déc. 2021 à 10:00, Máté Kocsis <kocsismate90@gmail.com> a écrit :
> > Hi Patrick, > >> >> Thank you for those stats. Would you have the chance to have some data >> including PHP 8.0 as well? >> Because without PHP 8.0 data, we can't deduce anything about what PHP >> 8.1 brings over PHP 8.0 and or 8.0 over 7.4. >> Virtually, 8.1 may even be slower than 8.0 if the big increase of >> performance happened in 8.0. > > > Have you seen the results page? It is available at > https://github.com/kocsismate/php-version-benchmarks/blob/main/docs/results/2021_11_25_10_08_1_aws_arm64_c6g_4xlarge/result.md
No, but now yes, thanks for the link, that's what I missed.
> and it lists the exact results for each release. Besides, the PHP 8.1 release page (https://www.php.net/releases/8.1/en.php#performance_improvements) > displays the performance improvement based on PHP 8.0 and also includes some of the optimizations which led to the boost. > > Cheers, > Máté
Cheers, Patrick
  116342
November 13, 2021 12:37 ocramius@gmail.com (Marco Pivetta)
Hey Máté,

On Fri, Nov 12, 2021 at 11:19 AM Máté Kocsis <kocsismate90@gmail.com> wrote:

> Hello Internals, > > I'm writing this email because lately, I've been working on an automatic > benchmarking framework for PHP, and I'd like to share some news regarding > it. The initial implementation > was sponsored by Craig Francis, and it was used for the evaluation of the > performance aspects of the is_literal() RFC. Since then, I fixed numerous > issues and implemented many new features, so with the very close advent of > PHP 8.1, the time has come to unveil it. > > I'm sure that many of you still remember Intel's automatic benchmarks from > a couple of years ago (e.g. https://externals.io/message/89843#89843). I > loved these emails, so this project served as a great inspiration for me > to start working on something similar. I have to admit > though that I won't be able to replicate their extremely advanced setup. > > My main goal was to develop a framework ( > https://github.com/kocsismate/php-version-benchmarks) which is: > - fully automatic so that it can be easily run regularly, and the > benchmarks are reproducible > - it's possible to try it out locally via Docker, but it can be run in the > cloud (currently, only AWS is > supported), and the instance type is configurable > - supports different CPU platforms (X86-64 and ARM64), and advanced options > whether > turbo boost/hyper threading/deeper CPU C-states are enabled is configurable > - supports any version of PHP since PHP 7.4, including any git branches or > commits > - supports the most important PHP configurations, including whether > OPcache/JIT/preloading > is enabled > - supports multiple tests: currently, the micro benchmarks bundled with > php-src, as well as the > Symfony and the Laravel demo sites as "real-life" tests are included with > some customizability (number of warmups, iterations number, number of > requests) > - results are measured via using PHP-CGI which is a lightweight web server > with very a small overhead > > The most current benchmark is available at > > https://github.com/kocsismate/php-version-benchmarks/blob/main/docs/results/2021_11_11_09_20_1_aws_arm64_c6g_4xlarge/result.md > . I'm happy to share that the results suggest PHP 8.1.0 is around 28-32% > faster than PHP 7.4 in real-life tests, and a few percent faster when it > comes to micro benchmarks. The benchmark was performed on an ARM64 instance > because this platform provided much more stable results than X86-64-based > ones did, mainly due to their fixed CPU frequency. > > Unfortunately, the benchmark is not yet suitable for detecting minor > performance changes between commits as the run-to-run variation of the > results is a bit too high; in some cases, it can be up to 1-2%. I hope that > this problem can be mitigated to an acceptable level in the future, but > most probably I'll need some help to achieve this. So any help is > appreciated! > > Furthermore, I'd have some questions regarding this project: > - Would you welcome automatic emails on internals (or on a dedicated > mailing list), just like how > Intel did it? I'm not sure about the frequency, but in my opinion, > 1-2/month would be a sensible one, given there is interest in it. > - Does anyone know how to get some sponsorship from AWS (or from some other > cloud > provider at last resort)? I already tried to apply for AWS Activate's free > credits, but my application was rejected due to different reasons > (normally, this program is available for startups). I have some hope that > running the benchmark on a dedicated instance would decrease variation of > the results, but enabling this feature is relatively expensive compared to > other costs, so I'd be very happy if I wouldn't have to sponsor all of > this, when I already dedicate a great deal of my time to the cause. > - As always, I'm happy to receive feedback and improvements (ideally along > with an implementation :) ). For example, currently I've started working on > some visualization and chart support, but frontend stuff is not my area of > expertise, so I'm progressing a bit slower than normally. Also, the > statistical foundations could be reviewed, and possibly improved. > > Regards: > Máté >
Nice results, and nice project! Overall against having it on the ML: I set up a filter to ignore those pesky emails by Intel. As much as I appreciate their effort, they were really noisy, and not really useful to the larger group. If your tool could export the results in machine format with something like an attached XSD or JSON-Schema, it would be possible to build UIs like https://arewefastyet.com/ , without the need to resort to a notification model. What's needed is to save them in a branch on the repo (or similar approach) for later consumption. Greets, Marco Pivetta http://twitter.com/Ocramius http://ocramius.github.com/ On Fri, Nov 12, 2021 at 11:19 AM Máté Kocsis <kocsismate90@gmail.com> wrote:
> Hello Internals, > > I'm writing this email because lately, I've been working on an automatic > benchmarking framework for PHP, and I'd like to share some news regarding > it. The initial implementation > was sponsored by Craig Francis, and it was used for the evaluation of the > performance aspects of the is_literal() RFC. Since then, I fixed numerous > issues and implemented many new features, so with the very close advent of > PHP 8.1, the time has come to unveil it. > > I'm sure that many of you still remember Intel's automatic benchmarks from > a couple of years ago (e.g. https://externals.io/message/89843#89843). I > loved these emails, so this project served as a great inspiration for me > to start working on something similar. I have to admit > though that I won't be able to replicate their extremely advanced setup. > > My main goal was to develop a framework ( > https://github.com/kocsismate/php-version-benchmarks) which is: > - fully automatic so that it can be easily run regularly, and the > benchmarks are reproducible > - it's possible to try it out locally via Docker, but it can be run in the > cloud (currently, only AWS is > supported), and the instance type is configurable > - supports different CPU platforms (X86-64 and ARM64), and advanced options > whether > turbo boost/hyper threading/deeper CPU C-states are enabled is configurable > - supports any version of PHP since PHP 7.4, including any git branches or > commits > - supports the most important PHP configurations, including whether > OPcache/JIT/preloading > is enabled > - supports multiple tests: currently, the micro benchmarks bundled with > php-src, as well as the > Symfony and the Laravel demo sites as "real-life" tests are included with > some customizability (number of warmups, iterations number, number of > requests) > - results are measured via using PHP-CGI which is a lightweight web server > with very a small overhead > > The most current benchmark is available at > > https://github.com/kocsismate/php-version-benchmarks/blob/main/docs/results/2021_11_11_09_20_1_aws_arm64_c6g_4xlarge/result.md > . I'm happy to share that the results suggest PHP 8.1.0 is around 28-32% > faster than PHP 7.4 in real-life tests, and a few percent faster when it > comes to micro benchmarks. The benchmark was performed on an ARM64 instance > because this platform provided much more stable results than X86-64-based > ones did, mainly due to their fixed CPU frequency. > > Unfortunately, the benchmark is not yet suitable for detecting minor > performance changes between commits as the run-to-run variation of the > results is a bit too high; in some cases, it can be up to 1-2%. I hope that > this problem can be mitigated to an acceptable level in the future, but > most probably I'll need some help to achieve this. So any help is > appreciated! > > Furthermore, I'd have some questions regarding this project: > - Would you welcome automatic emails on internals (or on a dedicated > mailing list), just like how > Intel did it? I'm not sure about the frequency, but in my opinion, > 1-2/month would be a sensible one, given there is interest in it. > - Does anyone know how to get some sponsorship from AWS (or from some other > cloud > provider at last resort)? I already tried to apply for AWS Activate's free > credits, but my application was rejected due to different reasons > (normally, this program is available for startups). I have some hope that > running the benchmark on a dedicated instance would decrease variation of > the results, but enabling this feature is relatively expensive compared to > other costs, so I'd be very happy if I wouldn't have to sponsor all of > this, when I already dedicate a great deal of my time to the cause. > - As always, I'm happy to receive feedback and improvements (ideally along > with an implementation :) ). For example, currently I've started working on > some visualization and chart support, but frontend stuff is not my area of > expertise, so I'm progressing a bit slower than normally. Also, the > statistical foundations could be reviewed, and possibly improved. > > Regards: > Máté >