Skip to content

Visibility hidden#995

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
jagerman merged 6 commits into from
Aug 14, 2017
Merged

Visibility hidden #995

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
3879045
Force hidden visibility on pybind code
jagerman Aug 10, 2017
88ae0c1
Use hidden visibility always; set via cmake property rather than comp…
jagerman Aug 10, 2017
368fb7f
Update docs to reflect forced hidden visibility
jagerman Aug 10, 2017
17c332d
Add -fvisibility=hidden to interface targets
jagerman Aug 13, 2017
8b9c6c5
Update docs wrt hidden visibility
jagerman Aug 13, 2017
22eae8a
Remove visibility=hidden build/install separation
jagerman Aug 13, 2017
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CMakeLists.txt
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -91,6 +91,9 @@ if(NOT (CMAKE_VERSION VERSION_LESS 3.0)) # CMake >= 3.0
$<BUILD_INTERFACE:${PYTHON_INCLUDE_DIRS}>
$<INSTALL_INTERFACE:${CMAKE_INSTALL_INCLUDEDIR}>)
target_compile_options(pybind11 INTERFACE $<BUILD_INTERFACE:${PYBIND11_CPP_STANDARD}>)
if(NOT MSVC)
target_compile_options(pybind11 INTERFACE -fvisibility=hidden)
endif()

add_library(module INTERFACE)
add_library(pybind11::module ALIAS module)
Expand Down
20 changes: 7 additions & 13 deletions docs/advanced/misc.rst
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -135,22 +135,16 @@ has been executed:

Naturally, both methods will fail when there are cyclic dependencies.

Note that compiling code which has its default symbol visibility set to
*hidden* (e.g. via the command line flag ``-fvisibility=hidden`` on GCC/Clang) can interfere with the
ability to access types defined in another extension module. Workarounds
include changing the global symbol visibility (not recommended, because it will
lead unnecessarily large binaries) or manually exporting types that are
accessed by multiple extension modules:
Note that pybind11 code compiled with hidden-by-default symbol visibility (e.g.
via the command line flag ``-fvisibility=hidden`` on GCC/Clang), which is
required proper pybind11 functionality, can interfere with the ability to
access types defined in another extension module. Working around this requires
manually exporting types that are accessed by multiple extension modules;
pybind11 provides a macro to do just this:

.. code-block:: cpp

#ifdef _WIN32
# define EXPORT_TYPE __declspec(dllexport)
#else
# define EXPORT_TYPE __attribute__ ((visibility("default")))
#endif

class EXPORT_TYPE Dog : public Animal {
class PYBIND11_EXPORT Dog : public Animal {
...
};

Expand Down
21 changes: 11 additions & 10 deletions docs/compiling.rst
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -74,13 +74,15 @@ removes this target from the default build (see CMake docs for details).

Since pybind11 is a template library, ``pybind11_add_module`` adds compiler
flags to ensure high quality code generation without bloat arising from long
symbol names and duplication of code in different translation units. The
additional flags enable LTO (Link Time Optimization), set default visibility
to *hidden* and strip unneeded symbols. See the :ref:`FAQ entry <faq:symhidden>`
for a more detailed explanation. These optimizations are never applied in
``Debug`` mode. If ``NO_EXTRAS`` is given, they will always be disabled, even
in ``Release`` mode. However, this will result in code bloat and is generally
not recommended.
symbol names and duplication of code in different translation units. It
sets default visibility to *hidden*, which is required for some pybind11
features and functionality when attempting to load multiple pybind11 modules
compiled under different pybind11 versions. It also adds additional flags

@dean0x7ddean0x7dAug 10, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Weird question, but: Why two spaces after a period?

@jagermanjagermanAug 10, 2017•
edited
Loading

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In monospace fonts, it is visually closer to, e.g., a TeX inter-sentence space. It's also consistent with what you get from tools like man when used in a terminal. And while I do think it looks a bit better, it's mainly habit: I do it without even thinking about it.

@jagermanjagermanAug 13, 2017

Copy link
Copy Markdown
MemberAuthor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also (I just noticed) vim does this automatically when rewrapping (e.g. if a line ends with a .). I'd be quite surprised if there isn't an option to control that, but it isn't something I turned on specifically.

enabling LTO (Link Time Optimization) and strip unneeded symbols. See the
:ref:`FAQ entry <faq:symhidden>` for a more detailed explanation. These
latter optimizations are never applied in ``Debug`` mode. If ``NO_EXTRAS`` is
given, they will always be disabled, even in ``Release`` mode. However, this
will result in code bloat and is generally not recommended.

As stated above, LTO is enabled by default. Some newer compilers also support
different flavors of LTO such as `ThinLTO`_. Setting ``THIN_LTO`` will cause
Expand DownExpand Up@@ -181,9 +183,8 @@ to an independently constructed (through ``add_library``, not
flags (i.e. this is up to you).

These include Link Time Optimization (``-flto`` on GCC/Clang/ICPC, ``/GL``
and ``/LTCG`` on Visual Studio). Default-hidden symbols on GCC/Clang/ICPC
(``-fvisibility=hidden``) and .OBJ files with many sections on Visual Studio
(``/bigobj``). The :ref:`FAQ <faq:symhidden>` contains an
and ``/LTCG`` on Visual Studio) and .OBJ files with many sections on Visual
Studio (``/bigobj``). The :ref:`FAQ <faq:symhidden>` contains an
explanation on why these are needed.

Embedding the Python interpreter
Expand Down
40 changes: 35 additions & 5 deletions docs/faq.rst
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -151,6 +151,33 @@ specifying a larger value, e.g. ``-ftemplate-depth=1024`` on GCC/Clang. The
culprit is generally the generation of function signatures at compile time
using C++14 template metaprogramming.

.. _`faq:hidden_visibility`:

"‘SomeClass’ declared with greater visibility than the type of its field ‘SomeClass::member’ [-Wattributes]"
============================================================================================================

This error typically indicates that you are compiling without the required
``-fvisibility`` flag. pybind11 code internally forces hidden visibility on
all internal code, but if non-hidden (and thus *exported*) code attempts to
include a pybind type (for example, ``py::object`` or ``py::list``) you can run
into this warning.

To avoid it, make sure you are specifying ``-fvisibility=hidden`` when
compiling pybind code.

As to why ``-fvisibility=hidden`` is necessary, because pybind modules could
have been compiled under different versions of pybind itself, it is also
important that the symbols defined in one module do not clash with the
potentially-incompatible symbols defined in another. While Python extension
modules are usually loaded with localized symbols (under POSIX systems
typically using ``dlopen`` with the ``RTLD_LOCAL`` flag), this Python default
can be changed, but even if it isn't it is not always enough to guarantee
complete independence of the symbols involved when not using
``-fvisibility=hidden``.

Additionally, ``-fvisiblity=hidden`` can deliver considerably binary size
savings. (See the following section for more details).


.. _`faq:symhidden`:

Expand DownExpand Up@@ -192,11 +219,14 @@ world. So we'll generally only want to export symbols for those functions which
are actually called from the outside.

This can be achieved by specifying the parameter ``-fvisibility=hidden`` to GCC
and Clang, which sets the default symbol visibility to *hidden*. It's best to
do this only for release builds, since the symbol names can be helpful in
debugging sessions. On Visual Studio, symbols are already hidden by default, so
nothing needs to be done there. Needless to say, this has a tremendous impact
on the final binary size of the resulting extension library.
and Clang, which sets the default symbol visibility to *hidden*, which has a
tremendous impact on the final binary size of the resulting extension library.
(On Visual Studio, symbols are already hidden by default, so nothing needs to
be done there.)

In addition to decreasing binary size, ``-fvisibility=hidden`` also avoids
potential serious issues when loading multiple modules and is required for
proper pybind operation. See the previous FAQ entry for more details.

Another aspect that can require a fair bit of code are function signature
descriptions. pybind11 automatically generates human-readable function
Expand Down
4 changes: 2 additions & 2 deletions include/pybind11/attr.h
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -12,7 +12,7 @@

#include "cast.h"

NAMESPACE_BEGIN(pybind11)
NAMESPACE_BEGIN(PYBIND11_NAMESPACE)

@dean0x7ddean0x7dAug 10, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't it be simpler to add the visibility attribute to the NAMESPACE_BEGIN macro instead of introducing a new one?

@wjakobwjakobAug 10, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm against that, for selfish reasons :). I have several other projects that use the NAMESPACE_BEGIN macro, where that behavior may not be intended.

@dean0x7ddean0x7dAug 10, 2017

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, I guess that's why NAMESPACE_BEGIN/END are the only macros not prefixed with PYBIND11 :)


/// \addtogroup annotations
/// @{
Expand DownExpand Up@@ -478,4 +478,4 @@ constexpr bool expected_num_args(size_t nargs, bool has_args, bool has_kwargs) {
}

NAMESPACE_END(detail)
NAMESPACE_END(pybind11)
NAMESPACE_END(PYBIND11_NAMESPACE)
4 changes: 2 additions & 2 deletions include/pybind11/buffer_info.h
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -11,7 +11,7 @@

#include "common.h"

NAMESPACE_BEGIN(pybind11)
NAMESPACE_BEGIN(PYBIND11_NAMESPACE)

/// Information record describing a Python buffer object
struct buffer_info {
Expand DownExpand Up@@ -105,4 +105,4 @@ template <typename T> struct compare_buffer_info<T, detail::enable_if_t<std::is_
};

NAMESPACE_END(detail)
NAMESPACE_END(pybind11)
NAMESPACE_END(PYBIND11_NAMESPACE)
4 changes: 2 additions & 2 deletions include/pybind11/cast.h
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -30,7 +30,7 @@
#include <string_view>
#endif

NAMESPACE_BEGIN(pybind11)
NAMESPACE_BEGIN(PYBIND11_NAMESPACE)
NAMESPACE_BEGIN(detail)
// Forward declarations:
inline PyTypeObject *make_static_property_type();
Expand DownExpand Up@@ -2084,4 +2084,4 @@ NAMESPACE_END(detail)
template<> class type_caster<Type> : public type_caster_base<Type> { }; \
}}

NAMESPACE_END(pybind11)
NAMESPACE_END(PYBIND11_NAMESPACE)
4 changes: 2 additions & 2 deletions include/pybind11/chrono.h
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -27,7 +27,7 @@
#define PyDateTime_DELTA_GET_MICROSECONDS(o) (((PyDateTime_Delta*)o)->microseconds)
#endif

NAMESPACE_BEGIN(pybind11)
NAMESPACE_BEGIN(PYBIND11_NAMESPACE)
NAMESPACE_BEGIN(detail)

template <typename type> class duration_caster {
Expand DownExpand Up@@ -159,4 +159,4 @@ template <typename Rep, typename Period> class type_caster<std::chrono::duration
};

NAMESPACE_END(detail)
NAMESPACE_END(pybind11)
NAMESPACE_END(PYBIND11_NAMESPACE)
4 changes: 2 additions & 2 deletions include/pybind11/class_support.h
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -11,7 +11,7 @@

#include "attr.h"

NAMESPACE_BEGIN(pybind11)
NAMESPACE_BEGIN(PYBIND11_NAMESPACE)
NAMESPACE_BEGIN(detail)

inline PyTypeObject *type_incref(PyTypeObject *type) {
Expand DownExpand Up@@ -599,4 +599,4 @@ inline PyObject* make_new_python_type(const type_record &rec) {
}

NAMESPACE_END(detail)
NAMESPACE_END(pybind11)
NAMESPACE_END(PYBIND11_NAMESPACE)
15 changes: 13 additions & 2 deletions include/pybind11/common.h
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -16,6 +16,17 @@
# define NAMESPACE_END(name) }
#endif

// Robust support for some features and loading modules compiled against different pybind versions
// requires forcing hidden visibility on pybind code, so we enforce this by setting the attribute on
// the main `pybind11` namespace.
#if !defined(PYBIND11_NAMESPACE)
# ifdef __GNUG__
# define PYBIND11_NAMESPACE pybind11 __attribute__((visibility("hidden")))
# else
# define PYBIND11_NAMESPACE pybind11
# endif
#endif

#if !defined(_MSC_VER) && !defined(__INTEL_COMPILER)
# if __cplusplus >= 201402L
# define PYBIND11_CPP14
Expand DownExpand Up@@ -297,7 +308,7 @@ extern "C" {
void pybind11_init_##name(pybind11::module &variable)


NAMESPACE_BEGIN(pybind11)
NAMESPACE_BEGIN(PYBIND11_NAMESPACE)

using ssize_t = Py_ssize_t;
using size_t = std::size_t;
Expand DownExpand Up@@ -877,4 +888,4 @@ NAMESPACE_END(detail)



NAMESPACE_END(pybind11)
NAMESPACE_END(PYBIND11_NAMESPACE)
4 changes: 2 additions & 2 deletions include/pybind11/complex.h
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -17,7 +17,7 @@
# undef I
#endif

NAMESPACE_BEGIN(pybind11)
NAMESPACE_BEGIN(PYBIND11_NAMESPACE)

template <typename T> struct format_descriptor<std::complex<T>, detail::enable_if_t<std::is_floating_point<T>::value>> {
static constexpr const char c = format_descriptor<T>::c;
Expand DownExpand Up@@ -58,4 +58,4 @@ template <typename T> class type_caster<std::complex<T>> {
PYBIND11_TYPE_CASTER(std::complex<T>, _("complex"));
};
NAMESPACE_END(detail)
NAMESPACE_END(pybind11)
NAMESPACE_END(PYBIND11_NAMESPACE)
4 changes: 2 additions & 2 deletions include/pybind11/descr.h
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -12,7 +12,7 @@

#include "common.h"

NAMESPACE_BEGIN(pybind11)
NAMESPACE_BEGIN(PYBIND11_NAMESPACE)
NAMESPACE_BEGIN(detail)

/* Concatenate type signatures at compile time using C++14 */
Expand DownExpand Up@@ -182,4 +182,4 @@ PYBIND11_NOINLINE inline descr type_descr(descr&& d) { return _("{") + std::move
#endif

NAMESPACE_END(detail)
NAMESPACE_END(pybind11)
NAMESPACE_END(PYBIND11_NAMESPACE)
4 changes: 2 additions & 2 deletions include/pybind11/eigen.h
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -35,7 +35,7 @@
// of matrices seems highly undesirable.
static_assert(EIGEN_VERSION_AT_LEAST(3,2,7), "Eigen support in pybind11 requires Eigen >= 3.2.7");

NAMESPACE_BEGIN(pybind11)
NAMESPACE_BEGIN(PYBIND11_NAMESPACE)

// Provide a convenience alias for easier pass-by-ref usage with fully dynamic strides:
using EigenDStride = Eigen::Stride<Eigen::Dynamic, Eigen::Dynamic>;
Expand DownExpand Up@@ -601,7 +601,7 @@ struct type_caster<Type, enable_if_t<is_eigen_sparse<Type>::value>> {
};

NAMESPACE_END(detail)
NAMESPACE_END(pybind11)
NAMESPACE_END(PYBIND11_NAMESPACE)

#if defined(__GNUG__) || defined(__clang__)
# pragma GCC diagnostic pop
Expand Down
4 changes: 2 additions & 2 deletions include/pybind11/embed.h
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -63,7 +63,7 @@
void pybind11_init_##name(pybind11::module &variable)


NAMESPACE_BEGIN(pybind11)
NAMESPACE_BEGIN(PYBIND11_NAMESPACE)
NAMESPACE_BEGIN(detail)

/// Python 2.7/3.x compatible version of `PyImport_AppendInittab` and error checks.
Expand DownExpand Up@@ -190,4 +190,4 @@ class scoped_interpreter {
bool is_valid = true;
};

NAMESPACE_END(pybind11)
NAMESPACE_END(PYBIND11_NAMESPACE)
4 changes: 2 additions & 2 deletions include/pybind11/eval.h
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -13,7 +13,7 @@

#include "pybind11.h"

NAMESPACE_BEGIN(pybind11)
NAMESPACE_BEGIN(PYBIND11_NAMESPACE)

enum eval_mode {
/// Evaluate a string containing an isolated expression
Expand DownExpand Up@@ -114,4 +114,4 @@ object eval_file(str fname, object global = globals(), object local = object())
return reinterpret_steal<object>(result);
}

NAMESPACE_END(pybind11)
NAMESPACE_END(PYBIND11_NAMESPACE)
4 changes: 2 additions & 2 deletions include/pybind11/functional.h
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -12,7 +12,7 @@
#include "pybind11.h"
#include <functional>

NAMESPACE_BEGIN(pybind11)
NAMESPACE_BEGIN(PYBIND11_NAMESPACE)
NAMESPACE_BEGIN(detail)

template <typename Return, typename... Args>
Expand DownExpand Up@@ -82,4 +82,4 @@ struct type_caster<std::function<Return(Args...)>> {
};

NAMESPACE_END(detail)
NAMESPACE_END(pybind11)
NAMESPACE_END(PYBIND11_NAMESPACE)
4 changes: 2 additions & 2 deletions include/pybind11/numpy.h
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -34,7 +34,7 @@
upon the library user. */
static_assert(sizeof(ssize_t) == sizeof(Py_intptr_t), "ssize_t != Py_intptr_t");

NAMESPACE_BEGIN(pybind11)
NAMESPACE_BEGIN(PYBIND11_NAMESPACE)

class array; // Forward declaration

Expand DownExpand Up@@ -1591,7 +1591,7 @@ Helper vectorize(Return (Class::*f)(Args...) const) {
return Helper(std::mem_fn(f));
}

NAMESPACE_END(pybind11)
NAMESPACE_END(PYBIND11_NAMESPACE)

#if defined(_MSC_VER)
#pragma warning(pop)
Expand Down
4 changes: 2 additions & 2 deletions include/pybind11/operators.h
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -18,7 +18,7 @@
# pragma warning(disable: 4127) // warning C4127: Conditional expression is constant
#endif

NAMESPACE_BEGIN(pybind11)
NAMESPACE_BEGIN(PYBIND11_NAMESPACE)
NAMESPACE_BEGIN(detail)

/// Enumeration with all supported operator types
Expand DownExpand Up@@ -160,7 +160,7 @@ NAMESPACE_END(detail)

using detail::self;

NAMESPACE_END(pybind11)
NAMESPACE_END(PYBIND11_NAMESPACE)

#if defined(_MSC_VER)
# pragma warning(pop)
Expand Down
4 changes: 2 additions & 2 deletions include/pybind11/options.h
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -11,7 +11,7 @@

#include "common.h"

NAMESPACE_BEGIN(pybind11)
NAMESPACE_BEGIN(PYBIND11_NAMESPACE)

class options {
public:
Expand DownExpand Up@@ -62,4 +62,4 @@ class options {
state previous_state;
};

NAMESPACE_END(pybind11)
NAMESPACE_END(PYBIND11_NAMESPACE)
4 changes: 2 additions & 2 deletions include/pybind11/pybind11.h
View file
Open in desktop
Original file line numberDiff line numberDiff line change
Expand Up@@ -44,7 +44,7 @@
#include "options.h"
#include "class_support.h"

NAMESPACE_BEGIN(pybind11)
NAMESPACE_BEGIN(PYBIND11_NAMESPACE)

/// Wraps an arbitrary C++ function/method/lambda function/.. into a callable Python object
class cpp_function : public function {
Expand DownExpand Up@@ -1862,7 +1862,7 @@ template <class T> function get_overload(const T *this_ptr, const char *name) {
#define PYBIND11_OVERLOAD_PURE(ret_type, cname, fn, ...) \
PYBIND11_OVERLOAD_PURE_NAME(ret_type, cname, #fn, fn, __VA_ARGS__)

NAMESPACE_END(pybind11)
NAMESPACE_END(PYBIND11_NAMESPACE)

#if defined(_MSC_VER)
# pragma warning(pop)
Expand Down
Loading