PHP 8.1 and PECL ext builds for Windows

  114751
June 6, 2021 13:25 cmbecker69@gmx.de ("Christoph M. Becker")
Hi all,

on Tuesday, PHP 8.1.0alpha1 is supposed to be tagged, and since I don't
have the capacity to do these builds manually (as currently done with
the PHP 8.0 builds), I've set up an automation which does the builds on
GH action runners[1].  This should likely be integrated into php-src or
php-sdk, whereyby the latter needs to be forked into a php organization
repo as soon as possible, since there is a pending commit regarding the
exclusion of yet unsupported PGO training scenarios, and we also should
roll a new SDK release, and update some of the bundled tools.

Anyhow, if these PHP 8.1 builds turn out to be good, I'd also like to do
the PHP 8.0 builds this way; this would free required capacities on the
current build machine (plus save a bit of my time), which is actually
supposed to do the PECL extension builds.  While these builds are
automated, and mostly work well, there are sometimes issues with new
releases and the dependency libraries are rarely updated (a lot of
these[2] are ancient versions).  And although I'm planning to enable
snapshot builds when PHP 8.1 enters the beta release cycle (as usual),
and to do the mass rebuild after PHP 8.1.0 is released, I won't be able
to spend much time to help with resolving issues.  In my opinion, it
would be beneficial to push the burden of providing Windows builds to
the extension maintainers.  There are already AppVeyor integrations for
several PECL extensions, some of them producing binaries which are
basically identical to the PECL builds, and generally Windows CI should
be helpful for package maintainers to detect potential issues before new
releases.  Furthermore, extensions maintainers would be more flexible
regarding the supported PHP version (currently, the PECL builds are done
for PHP 7.3, 7.4 and 8.0 only).

So yes, I'm proposing to stop centralized builds of PECL extensions for
Windows in the long run, and also to stop providing prebuilt dependency
libraries for PECL extensions.  If we do that, we can also sunset the
rmtools[3], which are used to provide snapshot builds of php-src (no
longer the case for PHP 8.0 due to lack of a dedicated build machine;
nightly snapshots are made available by Shivam[4]), and to provide
builds of the PECL extensions.

Thoughts?

[1] <https://github.com/cmb69/php-ftw/>
[2] <https://windows.php.net/downloads/pecl/deps/>
[3] <https://github.com/php/web-rmtools>
[4] <https://github.com/shivammathur/php-builder-windows>

--
Christoph M. Becker
  114757
June 7, 2021 08:02 nikita.ppv@gmail.com (Nikita Popov)
On Sun, Jun 6, 2021 at 3:26 PM Christoph M. Becker <cmbecker69@gmx.de>
wrote:

> Hi all, > > on Tuesday, PHP 8.1.0alpha1 is supposed to be tagged, and since I don't > have the capacity to do these builds manually (as currently done with > the PHP 8.0 builds), I've set up an automation which does the builds on > GH action runners[1].
To clarify, this builds the binary Windows release artifacts? Automating that using GH actions is great.
> This should likely be integrated into php-src or > php-sdk, whereyby the latter needs to be forked into a php organization > repo as soon as possible, since there is a pending commit regarding the > exclusion of yet unsupported PGO training scenarios, and we also should > roll a new SDK release, and update some of the bundled tools. >
Is php-sdk referring to https://github.com/microsoft/php-sdk-binary-tools? Having this in the PHP organization sounds reasonable, though I don't really follow your reasoning. How is the exclusion of PGO training scenarios relevant here? Is the concern here that Microsoft will not be maintaining the SDK for new PHP versions, so the PHP organization should take over doing that going forward? Anyhow, if these PHP 8.1 builds turn out to be good, I'd also like to do
> the PHP 8.0 builds this way; this would free required capacities on the > current build machine (plus save a bit of my time), which is actually > supposed to do the PECL extension builds.
Automating PHP 8.0 builds sounds good as well.
> While these builds are > automated, and mostly work well, there are sometimes issues with new > releases and the dependency libraries are rarely updated (a lot of > these[2] are ancient versions). And although I'm planning to enable > snapshot builds when PHP 8.1 enters the beta release cycle (as usual), > and to do the mass rebuild after PHP 8.1.0 is released, I won't be able > to spend much time to help with resolving issues. In my opinion, it > would be beneficial to push the burden of providing Windows builds to > the extension maintainers. There are already AppVeyor integrations for > several PECL extensions, some of them producing binaries which are > basically identical to the PECL builds, and generally Windows CI should > be helpful for package maintainers to detect potential issues before new > releases. Furthermore, extensions maintainers would be more flexible > regarding the supported PHP version (currently, the PECL builds are done > for PHP 7.3, 7.4 and 8.0 only). >
I see some possible complications here, mainly around storage and accessibility of the produced artifacts. Artifacts produced by AppVeyor are only stored for one month and not easily found. A nice thing about the current system is that the artifacts for all extensions can be found in one central place. The minimum would be to move artifacts from AppVeyor into GitHub release artifacts to make sure they're persistent, which is rather tedious without some automation (there are >16 Windows release artifacts for apcu). Regarding dependencies, does this mean that extensions should also build DLLs for dependency libraries themselves? Are there any concerns about different extensions building different versions of the same library, or similar? Regards, Nikita
  114759
June 7, 2021 08:58 cmbecker69@gmx.de ("Christoph M. Becker")
On 07.06.2021 at 10:02, Nikita Popov wrote:

> On Sun, Jun 6, 2021 at 3:26 PM Christoph M. Becker <cmbecker69@gmx.de> > wrote:
>> on Tuesday, PHP 8.1.0alpha1 is supposed to be tagged, and since I don't >> have the capacity to do these builds manually (as currently done with >> the PHP 8.0 builds), I've set up an automation which does the builds on >> GH action runners[1]. > > To clarify, this builds the binary Windows release artifacts? Automating > that using GH actions is great.
Right, that is about building the relese artifacts, which will then be available from <https://windows.php.net/>.
>> This should likely be integrated into php-src or >> php-sdk, whereyby the latter needs to be forked into a php organization >> repo as soon as possible, since there is a pending commit regarding the >> exclusion of yet unsupported PGO training scenarios, and we also should >> roll a new SDK release, and update some of the bundled tools. > > Is php-sdk referring to https://github.com/microsoft/php-sdk-binary-tools? > Having this in the PHP organization sounds reasonable, though I don't > really follow your reasoning. How is the exclusion of PGO training > scenarios relevant here? Is the concern here that Microsoft will not be > maintaining the SDK for new PHP versions, so the PHP organization should > take over doing that going forward?
Indeed, Microsoft has no intentions to maintain that repo for PHP 8, so we need to fork.
>> While these builds are >> automated, and mostly work well, there are sometimes issues with new >> releases and the dependency libraries are rarely updated (a lot of >> these[2] are ancient versions). And although I'm planning to enable >> snapshot builds when PHP 8.1 enters the beta release cycle (as usual), >> and to do the mass rebuild after PHP 8.1.0 is released, I won't be able >> to spend much time to help with resolving issues. In my opinion, it >> would be beneficial to push the burden of providing Windows builds to >> the extension maintainers. There are already AppVeyor integrations for >> several PECL extensions, some of them producing binaries which are >> basically identical to the PECL builds, and generally Windows CI should >> be helpful for package maintainers to detect potential issues before new >> releases. Furthermore, extensions maintainers would be more flexible >> regarding the supported PHP version (currently, the PECL builds are done >> for PHP 7.3, 7.4 and 8.0 only). > > I see some possible complications here, mainly around storage and > accessibility of the produced artifacts. Artifacts produced by AppVeyor are > only stored for one month and not easily found. A nice thing about the > current system is that the artifacts for all extensions can be found in one > central place. > > The minimum would be to move artifacts from AppVeyor into GitHub release > artifacts to make sure they're persistent, which is rather tedious without > some automation (there are >16 Windows release artifacts for apcu).
AppVeyor (and other CI providers) allows to upload artifacts to arbitrary locations; at least AppVeyor supports uploading of artifacts to GH realeases[1], so this could be automated. I'd be very surprised, if other GH CI providers wouldn't support that as well. [1] <https://www.appveyor.com/docs/deployment/github/>
> Regarding dependencies, does this mean that extensions should also build > DLLs for dependency libraries themselves? Are there any concerns about > different extensions building different versions of the same library, or > similar?
Oh, right, that could be an issue. Maybe we should stick with providing the dependencies from windows.php.net? Not sure how to handle the details, though. This doesn't look super urgent to me, but I would like to see more (Windows) CI integrations of the package repos soon. Reusable, publicly available GH actions might make CI integration for packages super simple even on Windows. Any help welcome! Christoph
  114765
June 7, 2021 11:06 nikita.ppv@gmail.com (Nikita Popov)
On Mon, Jun 7, 2021 at 10:58 AM Christoph M. Becker <cmbecker69@gmx.de>
wrote:

> On 07.06.2021 at 10:02, Nikita Popov wrote: > > > On Sun, Jun 6, 2021 at 3:26 PM Christoph M. Becker <cmbecker69@gmx.de> > > wrote: > > >> on Tuesday, PHP 8.1.0alpha1 is supposed to be tagged, and since I don't > >> have the capacity to do these builds manually (as currently done with > >> the PHP 8.0 builds), I've set up an automation which does the builds on > >> GH action runners[1]. > > > > To clarify, this builds the binary Windows release artifacts? Automating > > that using GH actions is great. > > Right, that is about building the relese artifacts, which will then be > available from <https://windows.php.net/>. > > >> This should likely be integrated into php-src or > >> php-sdk, whereyby the latter needs to be forked into a php organization > >> repo as soon as possible, since there is a pending commit regarding the > >> exclusion of yet unsupported PGO training scenarios, and we also should > >> roll a new SDK release, and update some of the bundled tools. > > > > Is php-sdk referring to > https://github.com/microsoft/php-sdk-binary-tools? > > Having this in the PHP organization sounds reasonable, though I don't > > really follow your reasoning. How is the exclusion of PGO training > > scenarios relevant here? Is the concern here that Microsoft will not be > > maintaining the SDK for new PHP versions, so the PHP organization should > > take over doing that going forward? > > Indeed, Microsoft has no intentions to maintain that repo for PHP 8, so > we need to fork. >
Okay, sounds good to me. Rather than a GitHub fork, I'd suggest pushing a clone of the repo, and indicate the original source in the repo description. GitHub forks have some annoying limitations (like the inability to use search), which are not great for long-term forks.
> >> While these builds are > >> automated, and mostly work well, there are sometimes issues with new > >> releases and the dependency libraries are rarely updated (a lot of > >> these[2] are ancient versions). And although I'm planning to enable > >> snapshot builds when PHP 8.1 enters the beta release cycle (as usual), > >> and to do the mass rebuild after PHP 8.1.0 is released, I won't be able > >> to spend much time to help with resolving issues. In my opinion, it > >> would be beneficial to push the burden of providing Windows builds to > >> the extension maintainers. There are already AppVeyor integrations for > >> several PECL extensions, some of them producing binaries which are > >> basically identical to the PECL builds, and generally Windows CI should > >> be helpful for package maintainers to detect potential issues before new > >> releases. Furthermore, extensions maintainers would be more flexible > >> regarding the supported PHP version (currently, the PECL builds are done > >> for PHP 7.3, 7.4 and 8.0 only). > > > > I see some possible complications here, mainly around storage and > > accessibility of the produced artifacts. Artifacts produced by AppVeyor > are > > only stored for one month and not easily found. A nice thing about the > > current system is that the artifacts for all extensions can be found in > one > > central place. > > > > The minimum would be to move artifacts from AppVeyor into GitHub release > > artifacts to make sure they're persistent, which is rather tedious > without > > some automation (there are >16 Windows release artifacts for apcu). > > AppVeyor (and other CI providers) allows to upload artifacts to > arbitrary locations; at least AppVeyor supports uploading of artifacts > to GH realeases[1], so this could be automated. I'd be very surprised, > if other GH CI providers wouldn't support that as well. > > [1] <https://www.appveyor.com/docs/deployment/github/> >
Okay, that looks nice, and should make producing Windows builds a matter of creating a GitHub tag. Do you know if any PHP ext already uses this, so it could be seen in action?
> Regarding dependencies, does this mean that extensions should also build > > DLLs for dependency libraries themselves? Are there any concerns about > > different extensions building different versions of the same library, or > > similar? > > Oh, right, that could be an issue. Maybe we should stick with providing > the dependencies from windows.php.net? Not sure how to handle the > details, though. This doesn't look super urgent to me, but I would like > to see more (Windows) CI integrations of the package repos soon. > Reusable, publicly available GH actions might make CI integration for > packages super simple even on Windows. Any help welcome! >
Right. I think a problem here is that the AppVeyor configuration for extensions is a bit arcane (at least to me). If one could just pick up a centrally maintained action from the GH marketplace, put in your target PHP versions and let the magic happen, that would make things simpler. Regards, Nikita
  114766
June 7, 2021 11:32 weltling@outlook.de (Anatol Belski)
Hi,

> -----Original Message----- > From: Nikita Popov ppv@gmail.com> > Sent: Monday, June 7, 2021 1:07 PM > To: Christoph M. Becker <cmbecker69@gmx.de> > Cc: PHP internals <internals@lists.php.net> > Subject: Re: [PHP-DEV] PHP 8.1 and PECL ext builds for Windows > > On Mon, Jun 7, 2021 at 10:58 AM Christoph M. Becker > <cmbecker69@gmx.de> > wrote: > > > On 07.06.2021 at 10:02, Nikita Popov wrote: > > > > > On Sun, Jun 6, 2021 at 3:26 PM Christoph M. Becker > <cmbecker69@gmx.de> > > > wrote: > > > > >> on Tuesday, PHP 8.1.0alpha1 is supposed to be tagged, and since I don't > > >> have the capacity to do these builds manually (as currently done with > > >> the PHP 8.0 builds), I've set up an automation which does the builds on > > >> GH action runners[1]. > > > > > > To clarify, this builds the binary Windows release artifacts? Automating > > > that using GH actions is great. > > > > Right, that is about building the relese artifacts, which will then be > > available from <https://windows.php.net/>. > > > > >> This should likely be integrated into php-src or > > >> php-sdk, whereyby the latter needs to be forked into a php > organization > > >> repo as soon as possible, since there is a pending commit regarding the > > >> exclusion of yet unsupported PGO training scenarios, and we also > should > > >> roll a new SDK release, and update some of the bundled tools. > > > > > > Is php-sdk referring to > > https://github.com/microsoft/php-sdk-binary-tools? > > > Having this in the PHP organization sounds reasonable, though I don't > > > really follow your reasoning. How is the exclusion of PGO training > > > scenarios relevant here? Is the concern here that Microsoft will not be > > > maintaining the SDK for new PHP versions, so the PHP organization > should > > > take over doing that going forward? > > > > Indeed, Microsoft has no intentions to maintain that repo for PHP 8, so > > we need to fork. > > > > Okay, sounds good to me. Rather than a GitHub fork, I'd suggest pushing a > clone of the repo, and indicate the original source in the repo > description. GitHub forks have some annoying limitations (like the > inability to use search), which are not great for long-term forks. > > > > >> While these builds are > > >> automated, and mostly work well, there are sometimes issues with new > > >> releases and the dependency libraries are rarely updated (a lot of > > >> these[2] are ancient versions). And although I'm planning to enable > > >> snapshot builds when PHP 8.1 enters the beta release cycle (as usual), > > >> and to do the mass rebuild after PHP 8.1.0 is released, I won't be able > > >> to spend much time to help with resolving issues. In my opinion, it > > >> would be beneficial to push the burden of providing Windows builds to > > >> the extension maintainers. There are already AppVeyor integrations for > > >> several PECL extensions, some of them producing binaries which are > > >> basically identical to the PECL builds, and generally Windows CI should > > >> be helpful for package maintainers to detect potential issues before > new > > >> releases. Furthermore, extensions maintainers would be more flexible > > >> regarding the supported PHP version (currently, the PECL builds are > done > > >> for PHP 7.3, 7.4 and 8.0 only). > > > > > > I see some possible complications here, mainly around storage and > > > accessibility of the produced artifacts. Artifacts produced by AppVeyor > > are > > > only stored for one month and not easily found. A nice thing about the > > > current system is that the artifacts for all extensions can be found in > > one > > > central place. > > > > > > The minimum would be to move artifacts from AppVeyor into GitHub > release > > > artifacts to make sure they're persistent, which is rather tedious > > without > > > some automation (there are >16 Windows release artifacts for apcu). > > > > AppVeyor (and other CI providers) allows to upload artifacts to > > arbitrary locations; at least AppVeyor supports uploading of artifacts > > to GH realeases[1], so this could be automated. I'd be very surprised, > > if other GH CI providers wouldn't support that as well. > > > > [1] <https://www.appveyor.com/docs/deployment/github/> > > > > Okay, that looks nice, and should make producing Windows builds a matter > of > creating a GitHub tag. Do you know if any PHP ext already uses this, so it > could be seen in action? > > > Regarding dependencies, does this mean that extensions should also build > > > DLLs for dependency libraries themselves? Are there any concerns about > > > different extensions building different versions of the same library, or > > > similar? > > > > Oh, right, that could be an issue. Maybe we should stick with providing > > the dependencies from windows.php.net? Not sure how to handle the > > details, though. This doesn't look super urgent to me, but I would like > > to see more (Windows) CI integrations of the package repos soon. > > Reusable, publicly available GH actions might make CI integration for > > packages super simple even on Windows. Any help welcome! > > > > Right. I think a problem here is that the AppVeyor configuration for > extensions is a bit arcane (at least to me). If one could just pick up a > centrally maintained action from the GH marketplace, put in your target PHP > versions and let the magic happen, that would make things simpler. >
Thanks, Christoph, for managing the Windows builds nicely so far! Removing the centralized PECL builder and dependency manager would most likely lead to a huge regression in the support and manageability. Right now there's one place pecl.php.net to go for the non core extension builds and any dependencies are guaranteed to be non conflicting. If this gets decentralized, the effort is moved to the extension maintainers which will most likely mean the chaos in where to get a DLL, DLL hell issues, absent DLL because the configuration is hard. This will steadily lead to the situation that was there before. IMO even keeping the basic version of the centralized approach even having a sporadic chance to fix issues is a far better way to go than dropping the existing achievements. Also in the long run, other approaches like moving to vcpkg for deps, checking on other things like cmake and pickle might be a good way, if there's a community interest. More volunteers on the community side would be great in this sense, too. Thanks anatol
  114774
June 7, 2021 21:42 cmbecker69@gmx.de ("Christoph M. Becker")
On 07.06.2021 at 13:32, Anatol Belski wrote:

> Removing the centralized PECL builder and dependency manager would most likely lead to a huge regression in the support and manageability. Right now there's one place pecl.php.net to go for the non core extension builds and any dependencies are guaranteed to be non conflicting. If this gets decentralized, the effort is moved to the extension maintainers which will most likely mean the chaos in where to get a DLL, DLL hell issues, absent DLL because the configuration is hard. This will steadily lead to the situation that was there before. > > IMO even keeping the basic version of the centralized approach even having a sporadic chance to fix issues is a far better way to go than dropping the existing achievements. Also in the long run, other approaches like moving to vcpkg for deps, checking on other things like cmake and pickle might be a good way, if there's a community interest. More volunteers on the community side would be great in this sense, too.
Good points, thank you for bringing them up! I have to fully agree that we should not drop the central point of distribution (i.e. windows.php.net). I don't think, however, that we can stick with the current PECL build system for long. Maybe the biggest issue is that extension maintainers may see automatic DLL builds as a given, or at least may not be able to fix things, because only few have access to the build machine. And even if that was not an issue, not many more would know where to look at. In other words, the bus factor is very low, and it may happen at some point in time, that no new DLLs would be built for *any* extension. This is why I still think it would be good to shift some of the burden of maintaining Windows builds to extension maintainers is a good thing. It is not about making their job harder, but rather about preventing serious issues, and also to correct expectations; extension maintainers might well assume that their extension is supported on common Linux distros, but they shouldn't *assume* it is supported on Windows as well (let alone that the dependency libraries have fixes for all known relevant security issues). Even if extensions are developed solely on Linux (and most are, as far as I know), they should have some Windows CI (at least doing the actual builds; better to run the test suite as well, of course), and that shouldn't be a real problem – there are several CI providers which are free for OSS projects. We should do our best to provide them with appropriate tools, so Windows CI integration can be set up as easily as for Linux phpize builds. That would not solve the issues regarding dependencies, but appears to be a reasonable first step in the right direction. Thanks, Christoph
  114781
June 8, 2021 10:59 ab@php.net (Anatol Belski)
Hi Christoph,

> -----Original Message----- > From: Christoph M. Becker <cmbecker69@gmx.de> > Sent: Monday, June 7, 2021 11:42 PM > To: Anatol Belski <weltling@outlook.de>; Nikita Popov > ppv@gmail.com> > Cc: PHP internals <internals@lists.php.net> > Subject: Re: [PHP-DEV] PHP 8.1 and PECL ext builds for Windows > > On 07.06.2021 at 13:32, Anatol Belski wrote: > > > Removing the centralized PECL builder and dependency manager would > most likely lead to a huge regression in the support and manageability. Right > now there's one place pecl.php.net to go for the non core extension builds > and any dependencies are guaranteed to be non conflicting. If this gets > decentralized, the effort is moved to the extension maintainers which will > most likely mean the chaos in where to get a DLL, DLL hell issues, absent DLL > because the configuration is hard. This will steadily lead to the situation that > was there before. > > > > IMO even keeping the basic version of the centralized approach even > having a sporadic chance to fix issues is a far better way to go than dropping > the existing achievements. Also in the long run, other approaches like > moving to vcpkg for deps, checking on other things like cmake and pickle > might be a good way, if there's a community interest. More volunteers on > the community side would be great in this sense, too. > > Good points, thank you for bringing them up! I have to fully agree that > we should not drop the central point of distribution (i.e. > windows.php.net). I don't think, however, that we can stick with the > current PECL build system for long. Maybe the biggest issue is that > extension maintainers may see automatic DLL builds as a given, or at > least may not be able to fix things, because only few have access to the > build machine. And even if that was not an issue, not many more would > know where to look at. In other words, the bus factor is very low, and > it may happen at some point in time, that no new DLLs would be built for > *any* extension. > > This is why I still think it would be good to shift some of the burden > of maintaining Windows builds to extension maintainers is a good thing. > It is not about making their job harder, but rather about preventing > serious issues, and also to correct expectations; extension maintainers > might well assume that their extension is supported on common Linux > distros, but they shouldn't *assume* it is supported on Windows as well > (let alone that the dependency libraries have fixes for all known > relevant security issues). > > Even if extensions are developed solely on Linux (and most are, as far > as I know), they should have some Windows CI (at least doing the actual > builds; better to run the test suite as well, of course), and that > shouldn't be a real problem – there are several CI providers which are > free for OSS projects. We should do our best to provide them with > appropriate tools, so Windows CI integration can be set up as easily as > for Linux phpize builds. That would not solve the issues regarding > dependencies, but appears to be a reasonable first step in the right > direction. > This would be nice, but a similar approach didn't work in the past. Please see how Python or node.js deal with this. The suggested way, PECL will be moving away from a good practice, not improve, and likely lead back the chaos it was before. It's not about assumptions, there was none before. As there's no resources on the team to maintain this, it's a clear story, but probably should mention the likehood of the consequence clearly, too. A good way would be first create a new path and ensure it can replace the old one, not just abandon the old one and hope for best.
Regards Anatol
  114768
June 7, 2021 12:05 pierre.php@gmail.com (Pierre Joye)
On Mon, Jun 7, 2021, 3:02 PM Nikita Popov

Regarding dependencies, does this mean that extensions should also build
> DLLs for dependency libraries themselves? Are there any concerns about > different extensions building different versions of the same library, or > similar? >
This the very reason we did it ourselves, way too many issues.
> Regards, > Nikita >