We are building docker package that contains our software package and we use conda (miniconda) to manage most of its dependencies. Recently, all of our docker builds started failing with what seems to be problems with dynamic library loading.
Core of the issue is that the base miniconda image comes with old libstdc++.so.6 while pandas package relies on a newer one. As expected, conda install these binary dependencies in the environment (/home/catalyst/env), but for some reason, when running python program that imports pandas, linker somehow falls back to system-wide library and breaks:
$ conda activate /home/catalyst/env
$ echo “import pandas” | python -
echo “import pandas” | python -
Traceback (most recent call last):
File “”, line 1, in
File “/home/catalyst/env/lib/python3.11/site-packages/pandas/init.py”, line 46, in
from pandas.core.api import (
File “/home/catalyst/env/lib/python3.11/site-packages/pandas/core/api.py”, line 47, in
from pandas.core.groupby import (
File “/home/catalyst/env/lib/python3.11/site-packages/pandas/core/groupby/init.py”, line 1, in
from pandas.core.groupby.generic import (
File “/home/catalyst/env/lib/python3.11/site-packages/pandas/core/groupby/generic.py”, line 67, in
from pandas.core.frame import DataFrame
File “/home/catalyst/env/lib/python3.11/site-packages/pandas/core/frame.py”, line 142, in
from pandas.core.generic import (
File “/home/catalyst/env/lib/python3.11/site-packages/pandas/core/generic.py”, line 187, in
from pandas.core.window import (
File “/home/catalyst/env/lib/python3.11/site-packages/pandas/core/window/init.py”, line 1, in
from pandas.core.window.ewm import (
File “/home/catalyst/env/lib/python3.11/site-packages/pandas/core/window/ewm.py”, line 11, in
import pandas._libs.window.aggregations as window_aggregations
ImportError: /lib/x86_64-linux-gnu/libstdc++.so.6: version `GLIBCXX_3.4.29’ not found (required by /home/catalyst/env/lib/python3.11/site-packages/pandas/_libs/window/aggregations.cpython-311-x86_64-linux-gnu.so)
However, ldd seems to find the right libstdc++.so.6:
ldd /home/catalyst/env/lib/python3.11/site-packages/pandas/_libs/window/aggregations.cpython-311-x86_64-linux-gnu.so
libstdc++.so.6 => /home/catalyst/env/lib/python3.11/site-packages/pandas/_libs/window/…/…/…/…/…/libstdc++.so.6 (0x00007fffffd93000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007fffffc41000)
libgcc_s.so.1 => /home/catalyst/env/lib/python3.11/site-packages/pandas/_libs/window/…/…/…/…/…/libgcc_s.so.1 (0x00007fffffc26000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fffffa34000)
librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007fffffa2a000)
/lib64/ld-linux-x86-64.so.2 (0x00007ffffffd0000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fffffa05000)
I’ve tried to research this problem a bit more, but I don’t quite understand:
- what could have changed to trigger this problem.
- what could explain the different behavior between python and ldd
I have worked-around this issue temporarily by setting LD_LIBRARY_PATH=/home/catalyst/env/lib but this seems to be discouraged and I would really like to understand what is the root cause of this issue.