Ffi.dll missing after conda upgrade

rswitt · August 24, 2023, 12:45am

New Fall Semester, new upgrade problems as usual!

I downloaded and did a fresh install of Anaconda 2023.07 to a clean Windows 11 system. Everything worked basically fine except for the extra debug messages while using conda (solved by upgrading conda-build from 3.26.0 to 3.26.1).

Knowing I was asking for trouble I decided to try upgrading Spyder from 5.4.3 to the latest 5.4.4 version. Since 5.4.3 is not available in the default repo, I pointed it to conda-forge:

conda update spyder -c conda-forge

That returned an enormous list of packages to update, replace, remove, etc, etc. Way bigger than I though it should be. Given that this wasn’t a critical install, I told it to go ahead.

After it completed, several errors were returned:

Executing transaction: done
Traceback (most recent call last):
  File "C:\Users\me\AppData\Local\anaconda3\Lib\site-packages\conda\exception_handler.py", line 17, in __call__
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\me\AppData\Local\anaconda3\Lib\site-packages\conda\cli\main.py", line 76, in main_sourced
    from ..base.context import context
  File "C:\Users\me\AppData\Local\anaconda3\Lib\site-packages\conda\base\context.py", line 33, in <module>
    from ..common._os.linux import linux_get_libc_version
  File "C:\Users\me\AppData\Local\anaconda3\Lib\site-packages\conda\common\_os\__init__.py", line 8, in <module>
    from .windows import get_free_space_on_windows as get_free_space
  File "C:\Users\me\AppData\Local\anaconda3\Lib\site-packages\conda\common\_os\windows.py", line 11, in <module>
    from ctypes import (
  File "C:\Users\me\AppData\Local\anaconda3\Lib\ctypes\__init__.py", line 8, in <module>
    from _ctypes import Union, Structure, Array
ImportError: DLL load failed while importing _ctypes: The specified module could not be found.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\me\AppData\Local\anaconda3\Scripts\conda-script.py", line 12, in <module>
    sys.exit(main())
             ^^^^^^
  File "C:\Users\me\AppData\Local\anaconda3\Lib\site-packages\conda\cli\main.py", line 109, in main
    return conda_exception_handler(main, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\me\AppData\Local\anaconda3\Lib\site-packages\conda\exception_handler.py", line 389, in conda_exception_handler
    return_value = exception_handler(func, *args, **kwargs)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\me\AppData\Local\anaconda3\Lib\site-packages\conda\exception_handler.py", line 20, in __call__
    return self.handle_exception(exc_val, exc_tb)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\me\AppData\Local\anaconda3\Lib\site-packages\conda\exception_handler.py", line 52, in handle_exception
    from .exceptions import (
  File "C:\Users\me\AppData\Local\anaconda3\Lib\site-packages\conda\exceptions.py", line 31, in <module>
    from .models.channel import Channel
  File "C:\Users\me\AppData\Local\anaconda3\Lib\site-packages\conda\models\channel.py", line 25, in <module>
    from ..base.context import Context, context
  File "C:\Users\me\AppData\Local\anaconda3\Lib\site-packages\conda\base\context.py", line 33, in <module>
    from ..common._os.linux import linux_get_libc_version
  File "C:\Users\me\AppData\Local\anaconda3\Lib\site-packages\conda\common\_os\__init__.py", line 8, in <module>
    from .windows import get_free_space_on_windows as get_free_space
  File "C:\Users\me\AppData\Local\anaconda3\Lib\site-packages\conda\common\_os\windows.py", line 11, in <module>
    from ctypes import (
  File "C:\Users\me\AppData\Local\anaconda3\Lib\ctypes\__init__.py", line 8, in <module>
    from _ctypes import Union, Structure, Array
ImportError: DLL load failed while importing _ctypes: The specified module could not be found.

After that conda and most other things l would fail with the importing _ctypes error.

After some research I found some pointers that this could be caused by some missing libraries. I noticed that my /library/bin directory did NOT contain ffi.dll (it DID have ffi-8.dll). I copied that file from another computer, and everything immediately worked.

I’m pretty sure I followed all best practices here, so something in that upgrade seems to remove the ffi.dll without replacing it. Is this a bug?

sweller · August 28, 2023, 7:55pm

Hello,

This is actually NOT the recommended approach, because you are mixing packages from conda-forge with packages from Anaconda default - that will not work.

You shouid have first created a new conda environment, activated it and then installed Spyder into that new conda environment.

You can revert the applied changes in your conda base environment as follows(this should get you back to a working state):

conda list -revisions
conda install -revision 0

rswitt · August 28, 2023, 8:17pm

Reasonable, I guess. I was fortunately able to get to working state by copying the ffi.dll file back (I don’t think upgrading should have caused the ffi.dll file to be deleted?

Follow-up question then, when creating a new conda environtment, what should be in the new environment by default? I’ve seen 3 different things occur in different installations:

the new environment is a copy of the default Anaconda base environment
the new environment is mostly empty with only conda and maybe python
the new environment is completely empty without even conda installed (not sure how to add conda in that state?)

The first 2 work ok, I can add packages as needed. The last is problematic, I’ve had to remove it and install a new environment specifying that conda be included.

sweller · August 28, 2023, 8:25pm

What OS are you working on(Windows, Linux or Mac-OSX)?

I am asking because conda works by updating your system PATH settings. In general you shouldn’t have any problems working from a empty conda environment, because your PATH settings will point conda to what it needs in the base environment. You might want to check your system path once you activate the new empty conda environment and make sure that the anaconda directory is part of your PATH.

rswitt · August 29, 2023, 12:20am

I am working in Windows (mainly).

I am specifically NOT selecting the “Add Anaconda to my Path environment variable” at installation (and which is specifically not recommended) because I know that can lead to problems.

Looking at my path variable, I do see that several Anaconda paths are included despite that:

C:\Users\me\anaconda3;
C:\Users\me\anaconda3\Library\mingw-w64\bin;
C:\Users\me\anaconda3\Library\usr\bin;
C:\Users\me\anaconda3\Library\bin;
C:\Users\me\anaconda3\Scripts;

Funny thing is that I believe I’ve noticed some installations ended up in C:\users<user>\anaconda3, and others ended up in c:\users<user>\Appdata\Local\anaconda3. Was there a change at some point, or what else may cause that?

sweller · August 29, 2023, 4:48pm

Hello,

There is a good reason for the two different installation directories - the ‘user specific’ install is the one that installs everything under ‘C:\users\anaconda3’ foider and is the one that most users should use. The second install option is a ‘system-wide’ install and on Windows that will put the main install under ‘C:\Program Files’ directory. The files that get put under ‘C:\users\Appdata\Local\anaconda3’ are temporary files used by the program.

rswitt · August 29, 2023, 5:15pm

Ahh, that’s super helpful! I’m teaching a class and saw the Local directory on some students’ computers when troubleshooting. Clearly they hadn’t followed directions!