Automatically build extensions available on PECL

[asgrim]

I've seen several requests in the issues backlog for "support extension X, Y, Z" sort of things. I started reading through https://github.com/heroku/heroku-buildpack-php/blob/master/support/build/README.md and realised what a huge document this is and how complex this process is currently...

I feel like this could be solved by, if Heroku finds an extension not supported "out the box" that it tries to install it from PECL, e.g. #371, if an application depends on ext-uuid, and uuid is not available already, it tries pecl install uuid perhaps?

I think this would solve the majority of extension requirements, but I have a feeling it might be easier said than done 😅

[dzuelke]

It's much much much easier said than done; I've tried to prototype it ;)

We use Composer with a custom installer for platform package installation; the packages contain PHP and extensions, and the installer is a plugin; all runs together during an initial composer install in .heroku/php, using a minimal PHP build, and then installs PHP, libraries, extensions and web servers. This is to guarantee that the later composer install step also succeeds, because it uses the same dependency resolution logic.

For installation, all available packages must therefor be known ahead of time, so that Composer can calculate an installable set of dependencies. We can't "try" an installation, not install e.g. ext-uuid, and then try installing that one later with a pecl install - Composer will bail out much earlier than that, because ext-uuid is not there.

My approach was to proxy all package metadata from pecl.php.net, transform the data to a Composer repository in a form that the custom platform installer understands (e.g. it knows it's a PECL package, so it would attempt a pecl install for it), and optionally even figure out caching.

However, the plan already falls apart with this bug: https://bugs.php.net/76719

Assuming that bug were ever fixed (by rewriting the entire codebase to not use string concatenation for XML, so that it'll actually be properly reliable), we could give it another shot :)

[asgrim]

Thanks for the response @dzuelke, these things are always more complicated in practice :)

Edit: that bug though 😁

[dzuelke]

I'm glad you're excited about that bug and are volunteering to fix it @asgrim, for the good of all mankind ;)

[stof]

Another thing is that many Pecl extensions are actually binding for C extensions. And then, supporting them would depend on having the C extension available in the stack image. Otherwise, the compilation will fail.

[asgrim]

Very good point!

[dzuelke]

Although that'd typically be fixable with an APT buildpack.

[dannycoulombe]

I'm trying too to add a simple pecl (or apt) library (zmq). I passed the whole day trying to implement it.. forking and analysing other implementation of this same repo but without any luck.

Where in the documentation (if any) can we understand of to achieve this?

[dzuelke]

All the necessary docs are at https://github.com/heroku/heroku-buildpack-php/blob/master/support/build/README.md (yes, it takes a while to understand; no, it can't be simplified).

The zmq extension is completely unmaintained, hasn't been updated in over four years (since early 2016, and the last tag on GitHub is from 2013), and has no releases that are compatible with PHP 7.2 or later, so there is little point in trying anyway.

[chasovskih-grisha]

yep, adding some extensions from pecl is a PAIN.
Would be great if Heroku will provide some really useful approach how to add such extensions.

[dzuelke]

If anybody wants to help with this, https://bugs.php.net/76719 needs to be fixed in order for this functionality to have any chance of ever being able to see the light of day.

[grisha-monaco]

Guys, please vote on that issue.
More votes -> more attention.
Rate the importance of this bug to you: high

https://bugs.php.net/bug.php?id=76719

Anyway, how we can support fixing? Who to write to and where bug should be fixed?

[JSn1nj4]

Came across this while looking into customizing the PHP buildpack to add protobuf and grpc (using this as a vague go-by). Insane that that bug is as old as 2010. 👀 (the original linked at the bottom of that convo)

But I voted. 😅

In the meantime, gonna see if that Sitepoint guide will help.

[dzuelke]

The bug still exists; if you run this, it'll show you all the PECL categories with broken XML (as in, those who have more than one <?xml … prolog):

for f in $(curl -s https://pecl.php.net/rest/c/categories.xml | grep -o 'c xlink:href="[^"]*' | cut -d'"' -f2 | sed 's/info.xml/packagesinfo.xml/' | sed s#^#https://pecl.php.net#); do { echo -n "$f: "; curl --silent $f | grep '<?xml' | wc -l; } | grep -v 1; done

If anyone wants to help move this forward, the PECL website code has been substantially reworked in the meantime, so it should be a lot easier to fix this.

https://github.com/php/web-pecl/blob/master/src/Rest.php is the code in question. As you can see, it uses string concatenation to generate XML, instead of a DOM or XMLWriter based approach, and that's where the problems come from.

https://github.com/php/web-pecl/blob/master/templates/pages/feeds/feeds.php could/should also be fixed in the same instance; it probably uses properly escaped input, but still.

If you're a little familiar with TAL, https://github.com/phptal/PHPTAL might be an alternative solution for templating purposes (for both parts of the code I linked), since it would allow a template based approach that's guaranteed to produce well-formed XML.

@JSn1nj4 please do not follow that Sitepoint article, the instructions are old, outdated, and will not work. The PHP buildpack supports custom package repositories, you do not have to run a modified version of the buildpack. Please see https://github.com/cachewerk/heroku-php-extensions for an example where @tillkruss maintains a few third-party extensions.

[JSn1nj4]

@dzuelke thanks for heading me off. I'll dig into that other repo you linked.