Improve non-interactive install where multiple packages match

[Thahun]

So far we used PECL and are now trying to replace it with Pie, but having some issues with automating it.

Just as a short background, due to our container structure we're at first building a base image and afterwards using that as a base to build more granular images for different purposes (e.g. app and cron). But they share the same PHP setup and to avoid redundant work and time, PHP is already installed in the base image. For now that was quite simple and we could simply use pecl install + docker-php-ext-enable to install and enable our PHP extensions like e.g. mongodb and redis. They were part of the base image and by that the setup was equal for all subsequent images. Next to that, build dependencies are deleted at the end of the build to keep the images smaller and the final containers are of course running as www-data so only the build has root permissions.
Of course we already use composer to install dependencies as well, so we already have them defined in the composer.json like

 "require": {
"php": "~8.4.0",
"ext-mongodb": "~2.1",
"ext-redis": "~6.1",
[...]
}

Having the setup given, we tried to use Pie in a similar way and hit an issue with this non interactive approach.
Of course we could use Pie with hardcoded packages in the Dockerfile like:

pie install mongodb/mongodb-extension:~2.1
pie install phpredis/phpredis:~6.1
[...]

(Installing multiple extensions at once is not yet possible, but there's already a feature request for it: #466)

But we're more in favor of relying on the composer.json integration to avoid defining the extensions twice, cause for Composer they are anyways already defined. So just calling:

pie install [--allow-non-interactive-project-install]

And here's our issue, the resolution of ext-* to a package name is always triggering a prompt.
Even with --allow-non-interactive-project-install or --no-interaction the prompt remains (which is quite understandable, cause how should Pie be sure about which package is correct, except just using the suggested one automatically)
The only location we could determine where this mapping is persisted is the pie.lock in /root/.pie/php8.4_d2b6f37be78b1b629f281b53efb060b9/pie.lock but of course it's not designed to be modified.

So, the final question, is there any way to use Pie in a fully non interactive way, using the composer.json as a source?
At least I couldn't find anything in the documentation nor the command help.

The only working but unsafe way to achieve it, by automatically accepting option 1 of the prompts, seems to be currently

printf "%s\n" 1 | pie install --allow-non-interactive-project-install

[asgrim]

First a question for you:

For ext-mongodb and ext-redis, there should only be one option for either of those; so those two specific extensions shouldn't prompt when using --allow-non-interactive-project-install - could you check which specific extensions are being asked for in that case? To try to reproduce your issue, I built a Dockerfile with the following:

FROM ubuntu AS test_non_interactive
RUN apt-get update && apt-get install -y php unzip libbz2-dev gcc make autoconf bison re2c pkg-config libtool php-dev
COPY --from=build_pie_phar /app/pie.phar /usr/local/bin/pie
RUN mkdir /foo
WORKDIR /foo
RUN echo '{"require":{"php":"~8.3.0","ext-mongodb":"~2.1","ext-redis":"~6.1"}}' >> /foo/composer.json
RUN echo '{"_readme":["This file locks the dependencies of your project to a known state","Read more about it at https://getcomposer.org/doc/01-basic-usage.md#installing-dependencies","This file is @generated automatically"],"content-hash":"201d95d4413cb1f999fba94555181bb0","packages":[],"packages-dev":[],"aliases":[],"minimum-stability":"stable","stability-flags":{},"prefer-stable":false,"prefer-lowest":false,"platform":{"php":"~8.3.0","ext-mongodb":"~2.1","ext-redis":"~6.1"},"platform-dev":{},"plugin-api-version":"2.6.0"}' >> /foo/composer.lock
RUN pie install --allow-non-interactive-project-install
RUN pie show

And this successfully installs the two missing extensions:

#19 [test_non_interactive 9/9] RUN pie show
#19 0.208 🥧 PHP Installer for Extensions (PIE) 1.3.5@a13873e, from The PHP Foundation
#19 0.322 You are running PHP 8.3.6
#19 0.340 Target PHP installation: 8.3.6 nts, on Linux/OSX/etc x86_64 (from /usr/bin/php8.3)
#19 0.391 Tip: to include extensions in this list that PIE does not manage, use the --all flag.
#19 0.574 #19 0.574 Loaded PIE extensions:
#19 1.146 redis:6.3.0 (from 🥧 phpredis/phpredis:6.3.0 ✅)
#19 1.533 mongodb:2.1.4 (from 🥧 mongodb/mongodb-extension:2.1.4 ✅)
#19 DONE 1.5s

In this particular instance, it --allow-non-interactive-project-install should work just fine for you, I think?

However, yes; if there is more than one extension (maybe there is an extension not given in your example) that provides the ext-<extname> it will fail. In this circumstance, the issue is "how do we define that map" as you rightly point out. Using the pie.lock isn't going to work; as those instructions wouldn't exist in your "project" as such.

This is definitely something I'd like to improve on, but we need to find an effective way for PIE users to say "when I want ext-blah I mean extension foo/blah (as opposed to bar/blah or baz/blah-improved, etc.).

[Thahun]

Sorry for not providing the reproduceable example and wasting your time.

Cause now I discovered the issue that let us think we have an issue.

FROM ubuntu AS test_non_interactive
RUN apt-get update && apt-get install -y php unzip libbz2-dev gcc make autoconf bison re2c pkg-config libtool php-dev
COPY --from=ghcr.io/php/pie:bin /pie /usr/bin/pie
RUN mkdir /foo
WORKDIR /foo
RUN echo '{"require":{"php":"~8.3.0","ext-mongodb":"~2.1","ext-redis":"~6.1"}}' >> /foo/composer.json
RUN echo '{"_readme":["This file locks the dependencies of your project to a known state","Read more about it at https://getcomposer.org/doc/01-basic-usage.md#installing-dependencies","This file is @generated automatically"],"content-hash":"201d95d4413cb1f999fba94555181bb0","packages":[],"packages-dev":[],"aliases":[],"minimum-stability":"stable","stability-flags":{},"prefer-stable":false,"prefer-lowest":false,"platform":{"php":"~8.3.0","ext-mongodb":"~2.1","ext-redis":"~6.1"},"platform-dev":{},"plugin-api-version":"2.6.0"}' >> /foo/composer.lock
WORKDIR /foo

docker build -t pie_test .
docker run -it pie_test

root@a220a1923045:/foo# pie install --allow-non-interactive-project-install
🥧 PHP Installer for Extensions (PIE) 1.3.5, from The PHP Foundation
You are running PHP 8.3.6
Target PHP installation: 8.3.6 nts, on Linux/OSX/etc arm64 (from /usr/bin/php8.3)
Checking extensions for your project __root__ (path: /foo)
requires: ext-mongodb:~2.1 🚫 Missing
The following packages may be suitable, which would you like to install:
[0] None
[1] mongodb/mongodb-extension: MongoDB driver extension
> 0
Okay I won't install anything for mongodb
requires: ext-redis:~6.1 🚫 Missing
The following packages may be suitable, which would you like to install:
[0] None
[1] phpredis/phpredis: A PHP extension for Redis
> 0
Okay I won't install anything for redis
Finished checking extensions.

The whole issue is that the --allow-non-interactive-project-install option doesn't have any effect on an interactive shell, and by that the install command can't be manually tested. Or at least it behaves different and we didn't expect that. So we just thought it's not working as expected while infact it works on a real non-interactive shell, just not on an interactive shell.

Independet of that, yes it would be good to have some option to define a mapping of ext to package, cause pretty sure at some point there'll be extensions matching multiple packages. In best case this could be done directly in composer.json to avoid a second config, but even an additional config yaml and/or option for the install command would be fine from my point of view.

[asgrim]

No worries at all :) FWIW, you could use COMPOSER_NO_INTERACTION=1 pie install --allow-non-interactive-project-install to simulate a non-interactive shell (I think)

But yes, I would like to have some way of defining what extensions are picked, but I was hoping to shelve that problem for now 😆

[Thahun]

Yes, that works, I tried it. We just didn't have that in mind. Of course it's obvious when having the relation between Composer and Pie in mind that Composer environment variables should work as well, but maybe it's worth to add a hint to the documentation of --allow-non-interactive-project-install or even a general one if all Composer environment variables are useable in Pie as well.

Anyways, thanks for the help :)

[Ninos]

Are there already any ideas how to finally solve this? I created an issue for that with a possible solution:
#506

In my opinion there must be some mappings to the specific extensions (not "just* for security reasons...), if you want to implement such non-interactive mode (required for building images)

[asgrim]

For continuity/ease of discussion, the linked issue suggestion was:

Currently it's not possible to autoinstall pie just using composer.json/composer.lock file.

Current behavior

Create a composer.json file, e.g. with following content:

{
"name": "1m9/php-build",
"type": "example",
"require": {
"php": ">=8.0",
"ext-apcu": "*",
"ext-redis": "*"
}
}

After that run composer install --ignore-platform-reqs to generate composer.lock file.

To install pie extensions, you need to run pie install in same folder. Pie is now (happily for security reasons) asking, which extension we want to install, e.g. apcu/apcu for ext-apcu.

Because of those user input requirements it's not possible to autoinstall pie extensions, e.g. in container environment.

Solution

Add an additional section, e.g. extensions, which is loading all relevant meta data for required extensions (not just the package & version, but also the additional pie flags (--with-xyz, ...).

Example

{
"name": "1m9/php-build",
"type": "example",
"extensions": {
"apcu/apcu": "*",
"phpredis/phpredis": {
"version": "*",
"flags": ["--with-xyz", "--without-xyz", "--enable-xyz", "--whatever..."]
}
},
"require": {
"php": ">=8.0",
"ext-apcu": "*",
"ext-redis": "*"
}
}

[asgrim]

@Ninos Yep, something like that could work in theory, but I need to discuss with the Composer team if it is feasible, or if we might need to take a different approach. I'm well aware of this issue; at the moment at least for now it shouldn't be a major issue, but certainly as forks or alternate approaches for extensions become available, this will certainly become a problem.

FWIW, in the interim a workaround if you need a specific extension where more than one match causes issues, or you need to provide specific flags would be something like:

RUN pie install phpredis/phpredis --with-xyz=foo --enable-xyz \
&& pie install --allow-non-interactive-project-install # + other flags if necessary...

(the intention being: install phpredis/phpredis with specific configure flags, then install everything else automatically)

[Ninos]

I see, thx for clarification! Let me know, if there's something I can help with, but I think at least you have better connections to the composer team... :D

Also thx for the workaround, solved it with a custom script by looping through all extensions, think it's a bit easier to handle :-) For interested folks:
https://ego.codes/1m9/library/php-build/-/blob/main/bin/php-build?ref_type=heads#L146-157

Would just be awesome to see such thing in future, makes lots of things easier (no need to define extensions in Containerfile) 👼

[rvanlaak]

For context; Packagist is experiencing downtime on their search due to bot traffic towards Algolia. As result my builds currently fail.

symfony-devs.slack.com/archives/CE5PPJU22/p1775118197646609
phpc.social/@packagist/116334279107558992

Having packages pinned would reduce having to query Packagist, and would allow installing from the package / extension cache in case Docker layer would have a cache miss.

[asgrim]

I think quite a nice solution to this would be to define in the require directly, but we'd have to update Composer to understand that:

{
"name": "asgrim/demo-php-project",
"description": "Example PHP project",
"type": "project",
"require": {
"php": "^8.0",
"ext-curl": "php/curl:*",
"ext-simplexml": "php/simplexml:*",
"ext-pdo": "php/pdo:*",
"ext-example_pie_extension": "asgrim/example-pie-extension:^2.0",
"ext-xdebug": "xdebug/xdebug:*"
}
}

note, that this doesn't validate at the moment (as it is passed directly to VersionParser)

[rvanlaak]

The current version of PIE already allows us to define several of our extensions as composer require directly, and PIE currently resolves them through a Packagist search.

So, I think the Composer change that'd be required for this would be to in the lock file allow pinning an extension to a certain installable that PIE is able to understand:

The most notable here with regards to "what currently works" is the MLML extension, which actually is a Rust package that gets installed through PIE to thereafter drop Rust / Cargo. The result; a 3MB cacheable Docker layer that includes all extensions that were installed through PIE:

COPY --link composer.* symfony.* ./
ENV PATH="/root/.cargo/bin:${PATH}"RUN curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | /bin/sh -s -- -y --no-modify-path \
&& pie install --allow-non-interactive-project-install \
&& rustup self uninstall -y \
&& rm -rf ~/.cargo ~/.rustup

[asgrim]

The issue in question is really; how do we resolve an extension requirement (e.g. ext-foo to the correct package name, e.g. my/foo). If there is one option, that's fine; but if there are multiple, who decides which is correct? The end user SHOULD; so some way for end users of declaring when I require ext-foo, I require foo/foo not bar/foo. Since Composer itself does not handle ext- dependencies at all, beyond checking they're installed (the rest is all done by PIE), it needs to be something defined in composer.json I think.

Or, avoid composer.json directly perhaps, and define a separate map, e.g. .pie-extensions:

ext-foo:foo/foo
ext-xdebug:xdebug/xdebug
ext-example_pie_extension:asgrim/example-pie-extension`

etc.

[pmparviz]

Multiple packages were found for ext-redis:
90.48 phpredis/phpredis
90.48 pie-extensions/redis

Anyone else had this? I have to explicitly install phpredis/phpredis in my Docker image because of this. I am not sure why the second one exists, it has 70 lifetime downloads and seems to be the same extension as the first one unless I am misunderstanding something. Sorry if it's not the right thread for this

[rvanlaak]

Asked them the same question, still not entirely clear to me why it gets preferred to introduce a new mirror repo over contributing to the quite active phpredis repo instead: pie-extensions/core#78

[asgrim]

This functionality will be addressed in issue #624 - thank you for participating!

[pmparviz]

thank you