Statically linked openssl and/or OpenSSL libraries: Enable simple use of static third party provider(s)

[martinschmatz]

There are situations where fully static linking of the openssl application or applications making use of the OpenSSL libraries is a must. This implies that any third-party provider (and its dependencies) used in such applications should also be statically linked.

What is requested is therefore that statically linked providers can easily be added to the openssl application itself and to it's libs for ease of use in other statically linked applications.

I hope I didn't overlook a discussion or issue around this topic, but here's how that can be achieved:

0) Overview:

• In the build configuration options, use build defines to specify name and init function of the provider and options to facilitate static linking,
• Inject the information from the build defines during build in file provider_predefined.c to treat the static provider as "built-in" provider in analogy to the default provider.
• Specify and activate the provider in openssl.cnf

The following assumes that the provider init function is myprovider_static_provider_init and that we (arbitrarily) specified the name of the provider to be myprovider_static.

1) Changes in the file provider_predefined.c:
Add here:

/* Check whether a statically linked external provider should be added*/
#if(defined _STATIC_EXTERNAL_PROVIDER_NAME && defined _STATIC_EXTERNAL_PROVIDER_INITFUNC)
/* Declaration of the external init function*/
extern OSSL_provider_init_fn _STATIC_EXTERNAL_PROVIDER_INITFUNC;
/* Prep to add the provider to the "built-in" provider array 'ossl_predefined_providers' below*/
# define ADD_STATIC_EXTERNAL_PROVIDER_(name, initfunc) { #name, NULL, initfunc, NULL, 1 }
# define ADD_STATIC_EXTERNAL_PROVIDER(name, initfunc) ADD_STATIC_EXTERNAL_PROVIDER_(name, initfunc)
#endif

and insert after this line:

 /* Add an external static provider if needed */
# if(defined _STATIC_EXTERNAL_PROVIDER_NAME && defined _STATIC_EXTERNAL_PROVIDER_INITFUNC)
 ADD_STATIC_EXTERNAL_PROVIDER(_STATIC_EXTERNAL_PROVIDER_NAME, _STATIC_EXTERNAL_PROVIDER_INITFUNC),
# endif

2) Add section in openssl.cnf:

[ openssl_init ]
providers = provider_sect
[ provider_sect ]
default = default_sect
myprovider_static = myprovider_static_sect
[ default_sect ]
activate = 1
[ myprovider_static_sect ]
activate=1

3) Use/add ./Configure options during build:

 no-shared \
 -D_STATIC_EXTERNAL_PROVIDER_NAME=myprovider_static \
 -D_STATIC_EXTERNAL_PROVIDER_INITFUNC=myprovider_static_provider_init \
 --static -L/usr/lib64/ossl-modules -l:myprovider_static.a -L/usr/lib64/ -l:myprovider_static_dependency.a -l:libssl.a -l:libcrypto.a -static \

The above approach was successfully tested by 'integrating' a statically linked QOS provider into a statically linked openssl and then generating Q-safe certificates using it, and by building cURL with static linking of oqsprovider.a, liboqs.a, libssl.a and libcrypto.a and testing it against the OQS Interop Server without changing a line of code on cURL.

Somebody skilled in the art of C pre-processor macro expansion might be able to extend the above to multiple providers.

If there's interest in the above approach, we could start working on a PR.

[nhorman]

Isn't what you're describing enabled by the OSSL_PROVIDER_add_builtin call?

That is to say, if you want a provider that is statically linked in with your application, you can just:

1. Create the provider, and build it as a .a file
2. link it to your application along with -lcrypto and -lssl
3. Have the application call OSSL_PROVIDER_add_builtin(ctx, "provider name", init_fn);

Then it should work like any other builtin provider, no need to modify the build infrastructure of openssl at all.

[martinschmatz]

@nhorman Yes, I understand the available concepts for adding/loading a statically linked provider in an application.

The first application I tried out was openssl itself.
I tried really hard to insert combinations of the triple OSSL_PROVIDER_available, OSSL_PROVIDER_add_builtin and OSSL_PROVIDER_load into the OpenSSL source code to get a static version of the openssl application with OQS provider. With the the concept of a "default library context" in combination with threading used in the openssl application, I wasn't successful. Maybe you can provide hints how/where in the OpenSSL source code to load a statically linked provider for the openssl application? (it is understood that this implies code changes in OpenSSL as well. Btw, using the undocumented feature via environment variable OPENSSL_TEST_LIBCTX does not cut it, neither does adding provider loading to OSSL_LIB_CTX_new).

The next applications were cURL, HAproxy, Nginx. Doable using the available functions - but with code changes in each of these widely used applications.

I then got to the approach outlined above, which - without any further code changes - enables statically linked versions for all of these applications. And as a side benefit, a statically linked openssl itself can also make use of statically linked provider(s). I'd like to emphasize the latter because I would not know how to achieve that otherwise.

One important aspect I didn't mention so far because I thought it might be obvious once a third party provider looks and feels like a built-in provider: There's the option for semi-static linking. This means static linking of provider(s) when building the dynamic OpenSSL libs, but all dynamic linking when building applications - or when using pre-built, packaged applications after replacing the default OpenSSL libraries with their provider enhanced versions (plus provider enablement in openssl.cnf). Verified both via cURL from package and built from source code.

In other words: With a code change in one file of OpenSSL, the above outlined approach enables addition of statically linked provider(s) to every application with makes use of the OpenSSL libraries, irrespective of whether these apps are statically or dynamically linked and without requiring [to add] an "ossl-modules" directory. This in contrast to required code changes in every application.

This is why I argue in favor of OpenSSL enabling transparent (= no app code changes) use of statically linked provider(s), which btw is the equivalent usability level that dynamically loaded providers achieve.

[nhorman]

So, you're approach above, as you've noted only works for a single additional provider. There would have to be some significant work done to enable multiple provider linking, which the existing api already allows. And the only advantage that I see to your approach is that it offer some mild level of convenience in that you only have to link -lcrypto instead of -lcrypto -l, which seems negligible in that the process is automated in most build environments.

You're welcome to create and submit a PR, but I think you'll run into challenges getting it accepted, as I don't see it offering a significant advantage over the programatic approach thats available now.

I'd be interested to know what failed with your approach using OSSL_PROVIDER_add_builtin(). That approach shouldn't require any additional changes to openssl code at all. The application just links in your provider, and during application startup makes a call to OSSL_PROVIDER_add_builtin, after which it should be loadable via OSSL_PROVIDER_load, like any other provider, or via a customized config file. Can you elaborate on what you did there and what went wrong?

[martinschmatz]

The entire feature request here is about avoiding code changes - which I claim nobody likes. As such, and with all due respect, I would not classify the benefits this provides as "some mild level of convenience". For example, there's a plethora of statically linked custom applications out there which must be upgraded to Q-safe TLS operation. The existing API is enforcing code changes for each of those. Which leads to the question of why enforcing API use in the first place, thereby enforcing code changes.

From your comment "shouldn't require any additional changes to openssl code" I conclude that I wasn't successful in making clear that the openssl binary itself was the application to receive static provider functionality.

int OSSL_PROVIDER_add_builtin(OSSL_LIB_CTX *libctx, const char *name, ossl_provider_init_fn *init_fn); requires a library context. Hence, one can't simply add this anywhere in the application code.

The question for the openssl app was therefore not how to use the existing API in an application, but where.

So I tried to add loading the provider here (inside an else statement) to 'sneak it in' whenever a new library context is generated. This was leading to a successful build and I was able to use openssl req ... - but wasn't able to successfully use it with openssl X509 ...

I then realized that openssl is making use of a "default library context", which is replicated to threads. I tried a couple of code places to explicitly load the provider for the default context, but never got openssl to build.

Finally, I tried to sneak it in here (including specifying the environment variable OPENSSL_TEST_LIBCTX), which built but again without success for all openssl sub-commands.

Last-not-least, related to "significant work done to enable multiple provider linking": Rather than changing the file provider_predefined.c explicitly, one most probably could change it implicitly by throwing in some simple sed statements via makefile.

[nhorman]

you were clear, I just don't agree with your assertion. of course, I'm not the final decision maker here, so take that for what its worth.

I understand your argument. Given that a multitude of applications would like to use a 3rd party provider in a statically linked environment, you would like to see openssl add infrastructure to link those providers to libcrypto directly, rather than to each individual application, and additional infrastructure to automatically call that providers initialization function, so that applications can be unaware of the addition of the new provider. My concern with that approach is that it needs to be modified so that an aribtrary number of provider libraries could be linked in, which adds complexity to the library, and I personally don't think the added application convenience is worth the trade off for that complexity.

An application already has several options to solve the need for an additional 3rd party provider

1. Load it at run time dynamically via config file. I understand that doesn't address your need for static builds, thats ok
2. link it statically. We can already do this as described, it just requires that the needed provider be linked to the application in question. This also necessitates calling OSSL_PROVIDER_add_builtin, which I understand you don't want to do, but its the current option
3. You can take a hybrid approach (assuming you support c99 constructors). Modify your third party provider to include a function such as:

static my_provider_init(void) __attribute__((constructor))
{
 OPENSSL_init_crypto(0, NULL);
 OSSL_PROVIDER_add_builtin(NULL, "providername", init_function);
}

Note that you don't actually need a library context here, passing NULL, will cause openssl to use the default context.

You may have to modify that some to suit your needs, but it should work. With that, you should be able to merge libcrypto.a with your 3rd party provider ,link to your application, and the constructor should be called on application start without any additional application modification.

So, given these options available, I don't think you're going to get much traction adding a 4th option into this mix, but you're always welcome to propose a PR. I'm happy to assign this to you as a community effort if you would like.

[levitte]

On the idea front, how about writing a script that takes a series of {provider-name}:{init-fn}:{static-provider-library-path} tuples, and generates two things:

1. A C file that contains a function to add all those providers, like so:

   int add_all_extra_builtins(OSSL_LIB_CTX *libctx)
   {
    int ok = 1;
   
    if (ok) ok &= OPENSSL_PROVIDER_add_builtin(libctx, "{provider-name}", {init-fn});
    /* ... more lines like that, one for each provider ... */
    return ok;
   }

2. The list of static provider libraries, to be used when linking the application. This just has to be a list of file names, the build system used should know what to do with them.

All the build system needs to do is to include the generated C file in the build, along with all the static provider library files.
All the application needs to do is to call add_all_extra_builtins()

Of course, add_all_extra_builtins can be any name you choose.

[levitte]

For OpenSSL specifically, I could imagine Configure having a configuration option --with-static-provider={name}:{initfn}:{static-provider-path}, which works by accumulating the data into an array that's stored in configdata.pm. The {static-provider-path} part could actually be treated just like any -l, but with the understanding that it's a full path, not just a name. Finally, the generator script for OpenSSL would have to be written in Perl, to take advantage of the data in configdata.pm.

Other applications will have to solve the same problem on their own, following the principles given in the previous comment. What works for OpenSSL proper is usually not directly applicable to other applications, because different build systems.

[martinschmatz]

@levitte I like your approach with a sequence of {provider-name}:{init-fn} (or equivalent). I'll see whether I can find the right spot to include something like that (I'm not fluent in Perl). For completeness, I'll also check what can be done with build defines and macro expansion tricks for the case with multiple providers, but that most probably would become ugly.

The issue I see with {static-provider-library-path}: Providers might have (static) dependencies (e.g. oqsprovider -> liboqs). Which is why I included -l:myprovider_static_dependency.a in point (3) of my initial post.
In other words, I'd be fine to specify the link procedure as shown in my initial post and therefore not using {static-provider-library-path}.

[martinschmatz]

@nhorman It is my understanding that __attribute__((constructor)) is for shared libs. For completeness, tried it out nevertheless with the static libs - but without success.

[nhorman]

Constructor attributes are usable with any object file, regardless of final linkage. There is an idiosyncracy when using static libs, in that you have to link with the --whole-archive option to prevent constructors from getting eliminated during the final link. You can strip the binary after it's done iirc to reduce the size