After each new environment I need to run conda clean, why?

Thomas_Haverkamp · June 21, 2023, 11:15am

Hi,
I recently installed miniconda on our HPC cluster for all users in our project group.
see also this issue: Permissions issues with conda clean?

I experience that after each run to install packages in a random environment, that with the next installation, I have conda failing like this:

conda install -c bioconda -c defaults ariba_2.14.6
Collecting package metadata (current_repodata.json): failed

CondaError: An error occurred when loading cached repodata.  Executing
`conda clean --index-cache` will remove cached repodata files
so they can be downloaded again.

Which means that after each installation it is necessary to run conda clean -i. Which is not very userfriendly in a multi-user installation.
Is this new for the latest conda versions, or is there something not right with our installation.

Anybody an idea where the issue might be?

sweller · June 21, 2023, 4:31pm

Hello,

Are you running this install in the base conda environment?

Please create a new conda environment and try it again:

conda create -n biocondaenv
conda activate biocondaenv
conda install -c bioconda -c defaults ariba_2.14.6 --dry-run --vvv. (verbose debug output)

Thomas_Haverkamp · June 22, 2023, 8:11am

No I don’t run the install in the base environment.conda

I have now done this, following your instructions

conda create -n biocondaenv
conda activate biocondaenv
conda install -c bioconda -c defaults ariba_2.14.6 --dry-run -vvv

That failed (see output 1 below), I than ran conda clean -i

conda clean -i 
conda install -c bioconda -c defaults ariba_2.14.6 --dry-run -vvv

that finished, with really a long output. There seems to be an error at the end. I have put the output 2 below

The output 1:

DEBUG conda.gateways.logging:set_verbosity(217): verbosity set to 3
DEBUG conda.core.solve:solve_final_state(296): solving prefix /cluster/projects/nn9305k/src/miniconda/envs/biocondaenv
  specs_to_remove: frozenset()
  specs_to_add: frozenset({MatchSpec("ariba_2.14.6")})
  prune: Null
Collecting package metadata (current_repodata.json): ...working... TRACE conda.gateways.disk.test:file_path_is_writable(21): checking path is writable /cluster/projects/nn9305k/src/miniconda/pkgs/urls.txt
DEBUG conda.core.package_cache_data:_check_writable(312): package cache directory '/cluster/projects/nn9305k/src/miniconda/pkgs' writable: True
DEBUG conda.gateways.repodata:fetch_latest(858): Using cached repodata for https://conda.anaconda.org/bioconda/linux-64/current_repodata.json at /cluster/projects/nn9305k/src/miniconda/pkgs/cache/a0b50a38.json. Timeout in 1250 sec
DEBUG conda.gateways.repodata:read_cache(940): Loading raw json for https://conda.anaconda.org/bioconda/linux-64/current_repodata.json at /cluster/projects/nn9305k/src/miniconda/pkgs/cache/a0b50a38.json
DEBUG conda.gateways.repodata:fetch_latest(858): Using cached repodata for https://conda.anaconda.org/bioconda/noarch/current_repodata.json at /cluster/projects/nn9305k/src/miniconda/pkgs/cache/e599ec5e.json. Timeout in 1251 sec
DEBUG conda.gateways.repodata:fetch_latest(867): Local cache timed out for https://repo.anaconda.com/pkgs/main/linux-64/current_repodata.json at /cluster/projects/nn9305k/src/miniconda/pkgs/cache/9e0f62c3.json
DEBUG conda.gateways.repodata:read_cache(940): Loading raw json for https://conda.anaconda.org/bioconda/noarch/current_repodata.json at /cluster/projects/nn9305k/src/miniconda/pkgs/cache/e599ec5e.json
DEBUG conda.gateways.repodata:__init__(122): Using CondaRepoInterface
DEBUG conda.gateways.repodata:fetch_latest(867): Local cache timed out for https://repo.anaconda.com/pkgs/r/linux-64/current_repodata.json at /cluster/projects/nn9305k/src/miniconda/pkgs/cache/f1f1f480.json
DEBUG conda.gateways.repodata:__init__(122): Using CondaRepoInterface
DEBUG conda.gateways.repodata:fetch_latest(858): Using cached repodata for https://conda.anaconda.org/conda-forge/linux-64/current_repodata.json at /cluster/projects/nn9305k/src/miniconda/pkgs/cache/18414ddb.json. Timeout in 1241 sec
DEBUG conda.gateways.repodata:fetch_latest(867): Local cache timed out for https://repo.anaconda.com/pkgs/main/noarch/current_repodata.json at /cluster/projects/nn9305k/src/miniconda/pkgs/cache/b35683e8.json
DEBUG conda.gateways.repodata:read_cache(940): Loading raw json for https://conda.anaconda.org/conda-forge/linux-64/current_repodata.json at /cluster/projects/nn9305k/src/miniconda/pkgs/cache/18414ddb.json
DEBUG conda.gateways.repodata:fetch_latest(858): Using cached repodata for https://conda.anaconda.org/conda-forge/noarch/current_repodata.json at /cluster/projects/nn9305k/src/miniconda/pkgs/cache/62c98390.json. Timeout in 1247 sec
DEBUG conda.gateways.repodata:__init__(122): Using CondaRepoInterface
DEBUG conda.gateways.repodata:fetch_latest(867): Local cache timed out for https://repo.anaconda.com/pkgs/r/noarch/current_repodata.json at /cluster/projects/nn9305k/src/miniconda/pkgs/cache/227b7582.json
DEBUG conda.gateways.repodata:read_cache(940): Loading raw json for https://conda.anaconda.org/conda-forge/noarch/current_repodata.json at /cluster/projects/nn9305k/src/miniconda/pkgs/cache/62c98390.json
DEBUG conda.gateways.repodata:__init__(122): Using CondaRepoInterface
DEBUG urllib3.connectionpool:_new_conn(1048): Starting new HTTPS connection (1): repo.anaconda.com:443
DEBUG urllib3.connectionpool:_new_conn(1048): Starting new HTTPS connection (1): repo.anaconda.com:443
DEBUG urllib3.connectionpool:_new_conn(1048): Starting new HTTPS connection (1): repo.anaconda.com:443
DEBUG urllib3.connectionpool:_new_conn(1048): Starting new HTTPS connection (1): repo.anaconda.com:443
DEBUG urllib3.connectionpool:_make_request(546): https://repo.anaconda.com:443 "GET /pkgs/main/noarch/current_repodata.json HTTP/1.1" 304 0
DEBUG conda.gateways.repodata:repodata(152):
>>GET /pkgs/main/noarch/current_repodata.json HTTPS
> User-Agent: conda/23.5.0 requests/2.31.0 CPython/3.9.16 Linux/5.14.0-162.23.1.el9_1.x86_64 rocky/9.1 glibc/2.34
> Accept: */*
> Accept-Encoding: gzip, deflate, zstd
> Connection: keep-alive
> If-Modified-Since: Fri, 16 Jun 2023 17:09:30 GMT
> If-None-Match: W/"73bdc2dfb68aad52e9ce656564fc1e73"

<<HTTPS 304 Not Modified
< Age: 485152
< Cache-Control: public, max-age=30
< CF-Cache-Status: HIT
< CF-RAY: 7db2f2b02faeb51e-OSL
< Content-Security-Policy: upgrade-insecure-requests; frame-ancestors 'self' https://content.anaconda.com/
< Date: Thu, 22 Jun 2023 07:58:45 GMT
< ETag: "73bdc2dfb68aad52e9ce656564fc1e73"
< Expires: Thu, 22 Jun 2023 07:59:15 GMT
< Last-Modified: Fri, 16 Jun 2023 17:09:30 GMT
< Server: cloudflare
< Set-Cookie: __cf_bm=n10XVel4it2r1v_q1vAMyBO.49O0JLTxG_gX0d19Uio-1687420725-0-AdEKHh2EKJwe3AUABNjWjh9hFxbUIg3SLw8lRWi1yy2/iUK8Slsow14+T96IQNpkrPP8HccU+JsBX/Mxg/TEsko=; path=/; expires=Thu, 22-Jun-23 08:28:45 GMT; domain=.anaconda.com; HttpOnly; Secure; SameSite=None
< Vary: Accept-Encoding
< x-amz-id-2: GfA2I9nBc4laFD6KqKL6egHCYLOLO97OoPkQYmQslQPvoE7cHrfdOKqWVKk4LPQuar1rOhAYPKw=
< x-amz-request-id: ZZ9ZAV5EYV5RQPY5
< x-amz-version-id: _DuG6RAv7Eo1yjauHahCcPsh7zvwjWuD
< Connection: keep-alive
< Elapsed: 00:00.205704


DEBUG conda.gateways.repodata:fetch_latest(890): 304 NOT MODIFIED for 'https://repo.anaconda.com/pkgs/main/noarch/current_repodata.json'. Updating mtime and loading from disk
DEBUG urllib3.connectionpool:_make_request(546): https://repo.anaconda.com:443 "GET /pkgs/main/linux-64/current_repodata.json HTTP/1.1" 304 0
DEBUG conda.gateways.repodata:repodata(152):
>>GET /pkgs/main/linux-64/current_repodata.json HTTPS
> User-Agent: conda/23.5.0 requests/2.31.0 CPython/3.9.16 Linux/5.14.0-162.23.1.el9_1.x86_64 rocky/9.1 glibc/2.34
> Accept: */*
> Accept-Encoding: gzip, deflate, zstd
> Connection: keep-alive
> If-Modified-Since: Thu, 22 Jun 2023 07:41:01 GMT
> If-None-Match: W/"d8073fe7969d103fb9146cacbbb8a8b0"

<<HTTPS 304 Not Modified
< Age: 584
< Cache-Control: public, max-age=30
< CF-Cache-Status: HIT
< CF-RAY: 7db2f2b05c620b59-OSL
< Content-Security-Policy: upgrade-insecure-requests; frame-ancestors 'self' https://content.anaconda.com/
< Date: Thu, 22 Jun 2023 07:58:45 GMT
< ETag: "d8073fe7969d103fb9146cacbbb8a8b0"
< Expires: Thu, 22 Jun 2023 07:59:15 GMT
< Last-Modified: Thu, 22 Jun 2023 07:41:01 GMT
< Server: cloudflare
< Set-Cookie: __cf_bm=1erKjY0fdWwXZcEv3.2TTYjJqJxOrs.lsAnBHOvJktQ-1687420725-0-AWAcNejILUKzPM+dUAeRFmkS/CqWBNCiVc2KjTditNr+4SC9Ch9sE0O3a6WRFuI7HODi6QaduIoD7BX4I+vdNGE=; path=/; expires=Thu, 22-Jun-23 08:28:45 GMT; domain=.anaconda.com; HttpOnly; Secure; SameSite=None
< Vary: Accept-Encoding
< x-amz-id-2: iZMZVQ6Sn3eJ4C6MSFr7466qtbEcIFkYpLMt6ziqJGFfpcw4tdwRFO1IVOnCjnt0jncow3+O3KA=
< x-amz-request-id: WPF8KVH02HG8NTY5
< x-amz-version-id: tT78b28H4fcJUHSM.g77N_39rCn5uXYU
< Connection: keep-alive
< Elapsed: 00:00.229817

DEBUG conda.gateways.repodata:fetch_latest(890): 304 NOT MODIFIED for 'https://repo.anaconda.com/pkgs/main/linux-64/current_repodata.json'. Updating mtime and loading from disk
DEBUG urllib3.connectionpool:_make_request(546): https://repo.anaconda.com:443 "GET /pkgs/r/linux-64/current_repodata.json HTTP/1.1" 304 0
DEBUG conda.gateways.repodata:repodata(152):
>>GET /pkgs/r/linux-64/current_repodata.json HTTPS
> User-Agent: conda/23.5.0 requests/2.31.0 CPython/3.9.16 Linux/5.14.0-162.23.1.el9_1.x86_64 rocky/9.1 glibc/2.34
> Accept: */*
> Accept-Encoding: gzip, deflate, zstd
> Connection: keep-alive
> If-Modified-Since: Thu, 25 May 2023 20:49:30 GMT
> If-None-Match: W/"209378d48d53c6ef7fbb4a8bd091e1d3"

<<HTTPS 304 Not Modified
< Age: 2372396
< Cache-Control: public, max-age=30
< CF-Cache-Status: HIT
< CF-RAY: 7db2f2b08c34b4ed-OSL
< Content-Security-Policy: upgrade-insecure-requests; frame-ancestors 'self' https://content.anaconda.com/
< Date: Thu, 22 Jun 2023 07:58:45 GMT
< ETag: "209378d48d53c6ef7fbb4a8bd091e1d3"
< Expires: Thu, 22 Jun 2023 07:59:15 GMT
< Last-Modified: Thu, 25 May 2023 20:49:30 GMT
< Server: cloudflare
< Set-Cookie: __cf_bm=5BD9vO2gkjE3OAywcc3kk95vJALTNfxvQWzlALSPVTs-1687420725-0-AWPsQJZWmxSJBYPl43sptcbPwQmF9W4NJEVEGt9rwmUyg2zShnc4RpTHTLmMI9e+Q34+T5SEdLx5Zob5Q6tSX0c=; path=/; expires=Thu, 22-Jun-23 08:28:45 GMT; domain=.anaconda.com; HttpOnly; Secure; SameSite=None
< Vary: Accept-Encoding
< x-amz-id-2: kfYuyMzIhzj3pXrU5FdK4g4GgELkHh7n/jQbbl8rYdG22AF47ZVyBP9S5KmON6KgF2AT17N2Uo0=
< x-amz-request-id: 5JF7ST47M410S1H5
< x-amz-version-id: 4J4jX0Xb.XR0J3NH0I4D6wU1YKolZJlW
< Connection: keep-alive
< Elapsed: 00:00.242665


DEBUG conda.gateways.repodata:fetch_latest(890): 304 NOT MODIFIED for 'https://repo.anaconda.com/pkgs/r/linux-64/current_repodata.json'. Updating mtime and loading from disk
DEBUG urllib3.connectionpool:_make_request(546): https://repo.anaconda.com:443 "GET /pkgs/r/noarch/current_repodata.json HTTP/1.1" 304 0
DEBUG conda.gateways.repodata:repodata(152):
>>GET /pkgs/r/noarch/current_repodata.json HTTPS
> User-Agent: conda/23.5.0 requests/2.31.0 CPython/3.9.16 Linux/5.14.0-162.23.1.el9_1.x86_64 rocky/9.1 glibc/2.34
> Accept: */*
> Accept-Encoding: gzip, deflate, zstd
> Connection: keep-alive
> If-Modified-Since: Fri, 28 Oct 2022 15:33:23 GMT
> If-None-Match: W/"324371c02bbeaf564a94b5641af402a7"

<<HTTPS 304 Not Modified
< Age: 1095664
< Cache-Control: public, max-age=30
< CF-Cache-Status: HIT
< CF-RAY: 7db2f2b08c6ab4fd-OSL
< Content-Security-Policy: upgrade-insecure-requests; frame-ancestors 'self' https://content.anaconda.com/
< Date: Thu, 22 Jun 2023 07:58:45 GMT
< ETag: "324371c02bbeaf564a94b5641af402a7"
< Expires: Thu, 22 Jun 2023 07:59:15 GMT
< Last-Modified: Fri, 28 Oct 2022 15:33:23 GMT
< Server: cloudflare
< Set-Cookie: __cf_bm=vOUKMs3gXd0xJ14rZPJEjeHhjUNBGhtGw2LN7CFu2Pg-1687420725-0-ASxEaqLSYpIWxFnj9jIIiIf+a67lWGkdfEpTLder+QA2eI0KVVna/M5+JDzRnswn1J74F16Z6Hh3/WndcVOTH40=; path=/; expires=Thu, 22-Jun-23 08:28:45 GMT; domain=.anaconda.com; HttpOnly; Secure; SameSite=None
< Vary: Accept-Encoding
< x-amz-id-2: b2NuocL9sRH6IZ8449b6x/G2fKXqfWGFRmbU2vwT23xUOSGsw9h+JMd4WxAPy196+sFKJLx481Q=
< x-amz-request-id: PRT1YH5C0D0QXYJ3
< x-amz-version-id: NxKNlZiG2kOcPe0atE2gXOvvxPYnQaXy
< Connection: keep-alive
< Elapsed: 00:00.268674


DEBUG conda.gateways.repodata:fetch_latest(890): 304 NOT MODIFIED for 'https://repo.anaconda.com/pkgs/r/noarch/current_repodata.json'. Updating mtime and loading from disk
DEBUG conda.gateways.repodata:read_cache(940): Loading raw json for https://repo.anaconda.com/pkgs/main/noarch/current_repodata.json at /cluster/projects/nn9305k/src/miniconda/pkgs/cache/b35683e8.json
DEBUG conda.gateways.repodata:read_cache(940): Loading raw json for https://repo.anaconda.com/pkgs/main/linux-64/current_repodata.json at /cluster/projects/nn9305k/src/miniconda/pkgs/cache/9e0f62c3.json
DEBUG conda.gateways.repodata:read_cache(954): Error for cache path: '/cluster/projects/nn9305k/src/miniconda/pkgs/cache/b35683e8.json'
JSONDecodeError('Extra data: line 9 column 2 (char 283)')
DEBUG conda.gateways.repodata:read_cache(954): Error for cache path: '/cluster/projects/nn9305k/src/miniconda/pkgs/cache/9e0f62c3.json'
JSONDecodeError('Extra data: line 9 column 2 (char 285)')
DEBUG conda.gateways.repodata:read_cache(940): Loading raw json for https://repo.anaconda.com/pkgs/r/noarch/current_repodata.json at /cluster/projects/nn9305k/src/miniconda/pkgs/cache/227b7582.json
DEBUG conda.gateways.repodata:read_cache(940): Loading raw json for https://repo.anaconda.com/pkgs/r/linux-64/current_repodata.json at /cluster/projects/nn9305k/src/miniconda/pkgs/cache/f1f1f480.json
DEBUG conda.gateways.repodata:read_cache(954): Error for cache path: '/cluster/projects/nn9305k/src/miniconda/pkgs/cache/227b7582.json'
JSONDecodeError('Extra data: line 9 column 2 (char 280)')
DEBUG conda.gateways.repodata:read_cache(954): Error for cache path: '/cluster/projects/nn9305k/src/miniconda/pkgs/cache/f1f1f480.json'
JSONDecodeError('Extra data: line 9 column 2 (char 282)')
failed
Traceback (most recent call last):
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/gateways/repodata/__init__.py", line 879, in fetch_latest
    raw_repodata = repo.repodata(cache.state)  # type: ignore
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/gateways/repodata/__init__.py", line 159, in repodata
    raise Response304ContentUnchanged()
conda.gateways.repodata.Response304ContentUnchanged

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/gateways/repodata/__init__.py", line 949, in read_cache
    raw_repodata_str = cache.load()
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/gateways/repodata/__init__.py", line 575, in load
    state = json.loads(state_file.read())
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 9 column 2 (char 285)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/exception_handler.py", line 16, in __call__
    return func(*args, **kwargs)
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/cli/main.py", line 84, in main_subshell
    exit_code = do_call(args, p)
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/cli/conda_argparse.py", line 126, in do_call
    return getattr(module, func_name)(args, parser)
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/notices/core.py", line 123, in wrapper
    return func(*args, **kwargs)
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/cli/main_install.py", line 22, in execute
    install(args, parser, "install")
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/cli/install.py", line 309, in install
    unlink_link_transaction = solver.solve_for_transaction(
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/core/solve.py", line 153, in solve_for_transaction
    unlink_precs, link_precs = self.solve_for_diff(
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/core/solve.py", line 214, in solve_for_diff
    final_precs = self.solve_final_state(
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/core/solve.py", line 357, in solve_final_state
    ssc = self._collect_all_metadata(ssc)
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/common/io.py", line 83, in decorated
    return f(*args, **kwds)
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/core/solve.py", line 571, in _collect_all_metadata
    index, r = self._prepare(prepared_specs)
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/core/solve.py", line 1285, in _prepare
    reduced_index = get_reduced_index(
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/core/index.py", line 286, in get_reduced_index
    new_records = SubdirData.query_all(
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/core/subdir_data.py", line 157, in query_all
    result = tuple(
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/concurrent/futures/_base.py", line 609, in result_iterator
    yield fs.pop().result()
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/concurrent/futures/_base.py", line 446, in result

    return self.__get_result()
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
    raise self._exception
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/core/subdir_data.py", line 142, in subdir_query
    return tuple(
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/core/subdir_data.py", line 164, in query
    self.load()
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/core/subdir_data.py", line 264, in load
    _internal_state = self._load()
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/core/subdir_data.py", line 321, in _load
    repodata, state = fetcher.fetch_latest_parsed()
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/gateways/repodata/__init__.py", line 757, in fetch_latest_parsed
    parsed, state = self.fetch_latest()
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/gateways/repodata/__init__.py", line 897, in fetch_latest
    _internal_state = self.read_cache()
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/gateways/repodata/__init__.py", line 958, in read_cache
    raise CondaError(message)
conda.CondaError: An error occurred when loading cached repodata.  Executing
`conda clean --index-cache` will remove cached repodata files
so they can be downloaded again.

The output 2

DEBUG conda.resolve:__init__(128): restricting to unmanageable packages: __archspec
DEBUG conda.resolve:__init__(128): restricting to unmanageable packages: __glibc
DEBUG conda.resolve:__init__(128): restricting to unmanageable packages: __linux
DEBUG conda.resolve:__init__(128): restricting to unmanageable packages: __unix
done
Solving environment: ...working... DEBUG conda.resolve:get_reduced_index(669): Retrieving packages for:
  - ariba_2.14.6
unsuccessful initial attempt using frozen solve. Retrying with flexible solve.
Traceback (most recent call last):
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/cli/install.py", line 309, in install
    unlink_link_transaction = solver.solve_for_transaction(
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/core/solve.py", line 153, in solve_for_transaction
    unlink_precs, link_precs = self.solve_for_diff(
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/core/solve.py", line 214, in solve_for_diff
    final_precs = self.solve_final_state(
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/core/solve.py", line 379, in solve_final_state
    ssc = self._add_specs(ssc)
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/core/solve.py", line 732, in _add_specs
    explicit_pool = ssc.r._get_package_pool(self.specs_to_add)
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/resolve.py", line 649, in _get_package_pool
    pool = self.get_reduced_index(specs)
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/common/io.py", line 83, in decorated
    return f(*args, **kwds)
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/resolve.py", line 674, in get_reduced_index
    explicit_specs, features = self.verify_specs(explicit_specs)
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/resolve.py", line 329, in verify_specs
    raise ResolvePackageNotFound(bad_deps)
conda.exceptions.ResolvePackageNotFound:
  - ariba_2.14.6

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/exception_handler.py", line 16, in __call__
    return func(*args, **kwargs)
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/cli/main.py", line 84, in main_subshell
    exit_code = do_call(args, p)
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/cli/conda_argparse.py", line 126, in do_call
    return getattr(module, func_name)(args, parser)
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/notices/core.py", line 123, in wrapper
    return func(*args, **kwargs)
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/cli/main_install.py", line 22, in execute
    install(args, parser, "install")
  File "/cluster/projects/nn9305k/src/miniconda/lib/python3.9/site-packages/conda/cli/install.py", line 340, in install
    raise PackagesNotFoundError(e._formatted_chains, channels_urls)
conda.exceptions.PackagesNotFoundError: The following packages are not available from current channels:

  - ariba_2.14.6

Current channels:

  - https://conda.anaconda.org/bioconda/linux-64
  - https://conda.anaconda.org/bioconda/noarch
  - https://repo.anaconda.com/pkgs/main/linux-64
  - https://repo.anaconda.com/pkgs/main/noarch
  - https://repo.anaconda.com/pkgs/r/linux-64
  - https://repo.anaconda.com/pkgs/r/noarch
  - https://conda.anaconda.org/conda-forge/linux-64
  - https://conda.anaconda.org/conda-forge/noarch

To search for alternate channels that may provide the conda package you're
looking for, navigate to

    https://anaconda.org

and use the search bar at the top of the page.

sweller · June 22, 2023, 7:34pm

Can you tell me whether you are installing on a Linux machine or on a Mac?
If it is a Linux machine, what version of Linux(Redhat, Ubuntu, etc.)

Thomas_Haverkamp · June 23, 2023, 7:24am

I am installing on an linux cluster running Rocky Linux 9.1.

It is supposed to be a multi-user installation.

sweller · June 27, 2023, 6:11pm

Hello Thomas,

Can you please try the installation again as follows:

conda create -n biocondaenv
conda activate biocondaenv
conda install -c bioconda -c defaults ariba

I think the reason why it is failing for you is that you are pinning the version of ariba to 2.14.6. I don’t think this version is compatible with other packages that need to be installed from the bioconda repository. Try that and let me know if it works for you… It worked for me on a Redhat Linux instance.

Thomas_Haverkamp · June 28, 2023, 7:53am

Okay, I will document here all my steps, with the output to show my problem with conda clean.

conda create -n biocondaenv

output:

Retrieving notices: ...working... done
Collecting package metadata (current_repodata.json): done
Solving environment: done

CondaError: An error occurred when loading cached repodata.  Executing
`conda clean --index-cache` will remove cached repodata files
so they can be downloaded again.

running:

conda clean --index-cache

output:

Will remove 1 index cache(s).
Proceed ([y]/n)? y

running:

 conda create -n biocondaenv

output:

Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /cluster/projects/nn9305k/src/miniconda/envs/biocondaenv



Proceed ([y]/n)? y

Preparing transaction: done
Verifying transaction: done
Executing transaction: done
#
# To activate this environment, use
#
#     $ conda activate biocondaenv
#
# To deactivate an active environment, use
#
#     $ conda deactivate

activating environment

conda activate biocondaenv

Running:

conda install -c bioconda -c defaults ariba

output:

Collecting package metadata (current_repodata.json): failed

CondaError: An error occurred when loading cached repodata.  Executing
`conda clean --index-cache` will remove cached repodata files
so they can be downloaded again.

Running:

 conda clean -i
Will remove 1 index cache(s).
Proceed ([y]/n)? y

then I try again the installation:

conda install -c bioconda -c defaults ariba

output:

Collecting package metadata (current_repodata.json): done
Solving environment: done

## Package Plan ##

  environment location: /cluster/projects/nn9305k/src/miniconda/envs/biocondaenv

  added / updated specs:
    - ariba


The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    ariba-2.14.6               |   py37h675a0cb_4         2.6 MB  bioconda
    bcftools-1.9               |       ha228f0b_4         807 KB  bioconda
    beautifulsoup4-4.10.0      |     pyh06a4308_0          85 KB
    biopython-1.78             |   py37h7f8727e_0         2.1 MB
    bowtie2-2.2.5              |   py37h22450f8_8        14.9 MB  bioconda
    bzip2-1.0.8                |       h7b6447c_0          78 KB
    c-ares-1.19.0              |       h5eee18b_0         118 KB
    ca-certificates-2023.05.30 |       h06a4308_0         120 KB
    cd-hit-4.8.1               |       h43eeafb_9         232 KB  bioconda
    curl-7.88.1                |       h5eee18b_0          88 KB
    cycler-0.11.0              |     pyhd3eb1b0_0          12 KB
    dendropy-4.6.1             |     pyhdfd78af_0         325 KB  bioconda
    freetype-2.12.1            |       h4a9f257_0         626 KB
    gdbm-1.18                  |       hd4cb3f1_4         194 KB
    icu-58.2                   |       he6710b0_3        10.5 MB
    kiwisolver-1.4.4           |   py37h6a678d5_0          74 KB
    krb5-1.19.4                |       h568e23c_0         1.3 MB
    ld_impl_linux-64-2.38      |       h1181459_1         654 KB
    libblas-3.9.0              |16_linux64_openblas          13 KB  conda-forge
    libcblas-3.9.0             |16_linux64_openblas          13 KB  conda-forge
    libcurl-7.88.1             |       h91b91d3_0         383 KB
    libdeflate-1.0             |       h14c3975_1          43 KB  bioconda
    libedit-3.1.20221030       |       h5eee18b_0         181 KB
    libev-4.33                 |       h7f8727e_1         111 KB
    libffi-3.4.4               |       h6a678d5_0         142 KB
    libgfortran-ng-11.2.0      |       h00389a5_1          20 KB
    libgfortran5-11.2.0        |       h1234567_1         2.0 MB
    libidn2-2.3.4              |       h5eee18b_0         146 KB
    liblapack-3.9.0            |16_linux64_openblas          13 KB  conda-forge
    libnghttp2-1.52.0          |       ha637b67_1         671 KB
    libnsl-2.0.0               |       h5eee18b_0          31 KB
    libopenblas-0.3.21         |       h043d6bf_0         5.4 MB
    libpng-1.6.39              |       h5eee18b_0         304 KB
    libssh2-1.10.0             |       h37d81fd_2         292 KB
    libunistring-0.9.10        |       h27cfd23_0         536 KB
    matplotlib-base-3.1.3      |   py37hef1b27d_0         5.0 MB
    mummer-3.23                |                4         3.4 MB  bioconda
    ncurses-6.4                |       h6a678d5_0         914 KB
    numpy-1.21.6               |   py37h976b520_0         6.1 MB  conda-forge
    openssl-1.1.1u             |       h7f8727e_0         3.7 MB
    perl-5.34.0                |       h5eee18b_2        12.4 MB
    perl-threaded-5.32.1       |       hdfd78af_1           5 KB  bioconda
    pyfastaq-3.17.0            |             py_2          30 KB  bioconda
    pymummer-0.11.0            |     pyhdfd78af_1          14 KB  bioconda
    pyparsing-3.0.9            |   py37h06a4308_0         150 KB
    pysam-0.15.3               |   py37hda2845c_1         2.5 MB  bioconda
    python-3.7.12              |hb7a2778_100_cpython        57.3 MB  conda-forge
    python-dateutil-2.8.2      |     pyhd3eb1b0_0         233 KB
    python_abi-3.7             |          3_cp37m           6 KB  conda-forge
    readline-8.2               |       h5eee18b_0         357 KB
    samtools-1.6               |       hcd7b337_9         495 KB  bioconda
    six-1.16.0                 |     pyhd3eb1b0_1          18 KB
    soupsieve-2.3.1            |     pyhd3eb1b0_0          34 KB
    spades-3.13.0              |                0        13.7 MB  bioconda
    sqlite-3.41.2              |       h5eee18b_0         1.2 MB
    tk-8.6.12                  |       h1ccaba5_0         3.0 MB
    tornado-6.2                |   py37h5eee18b_0         584 KB
    typing_extensions-4.1.1    |     pyh06a4308_0          28 KB
    wget-1.21.4                |       h91b91d3_1         805 KB
    wheel-0.38.4               |   py37h06a4308_0          63 KB
    ------------------------------------------------------------
                                           Total:       157.0 MB

The following NEW packages will be INSTALLED:

  _libgcc_mutex      conda-forge/linux-64::_libgcc_mutex-0.1-conda_forge
  _openmp_mutex      conda-forge/linux-64::_openmp_mutex-4.5-2_gnu
  ariba              bioconda/linux-64::ariba-2.14.6-py37h675a0cb_4
  bcftools           bioconda/linux-64::bcftools-1.9-ha228f0b_4
  beautifulsoup4     pkgs/main/noarch::beautifulsoup4-4.10.0-pyh06a4308_0
  biopython          pkgs/main/linux-64::biopython-1.78-py37h7f8727e_0
  bowtie2            bioconda/linux-64::bowtie2-2.2.5-py37h22450f8_8
  bzip2              pkgs/main/linux-64::bzip2-1.0.8-h7b6447c_0
  c-ares             pkgs/main/linux-64::c-ares-1.19.0-h5eee18b_0
  ca-certificates    pkgs/main/linux-64::ca-certificates-2023.05.30-h06a4308_0
  cd-hit             bioconda/linux-64::cd-hit-4.8.1-h43eeafb_9
  curl               pkgs/main/linux-64::curl-7.88.1-h5eee18b_0
  cycler             pkgs/main/noarch::cycler-0.11.0-pyhd3eb1b0_0
  dendropy           bioconda/noarch::dendropy-4.6.1-pyhdfd78af_0
  freetype           pkgs/main/linux-64::freetype-2.12.1-h4a9f257_0
  gdbm               pkgs/main/linux-64::gdbm-1.18-hd4cb3f1_4
  icu                pkgs/main/linux-64::icu-58.2-he6710b0_3
  kiwisolver         pkgs/main/linux-64::kiwisolver-1.4.4-py37h6a678d5_0
  krb5               pkgs/main/linux-64::krb5-1.19.4-h568e23c_0
  ld_impl_linux-64   pkgs/main/linux-64::ld_impl_linux-64-2.38-h1181459_1
  libblas            conda-forge/linux-64::libblas-3.9.0-16_linux64_openblas
  libcblas           conda-forge/linux-64::libcblas-3.9.0-16_linux64_openblas
  libcurl            pkgs/main/linux-64::libcurl-7.88.1-h91b91d3_0
  libdeflate         bioconda/linux-64::libdeflate-1.0-h14c3975_1
  libedit            pkgs/main/linux-64::libedit-3.1.20221030-h5eee18b_0
  libev              pkgs/main/linux-64::libev-4.33-h7f8727e_1
  libffi             pkgs/main/linux-64::libffi-3.4.4-h6a678d5_0
  libgcc-ng          conda-forge/linux-64::libgcc-ng-13.1.0-he5830b7_0
  libgfortran-ng     pkgs/main/linux-64::libgfortran-ng-11.2.0-h00389a5_1
  libgfortran5       pkgs/main/linux-64::libgfortran5-11.2.0-h1234567_1
  libgomp            conda-forge/linux-64::libgomp-13.1.0-he5830b7_0
  libidn2            pkgs/main/linux-64::libidn2-2.3.4-h5eee18b_0
  liblapack          conda-forge/linux-64::liblapack-3.9.0-16_linux64_openblas
  libnghttp2         pkgs/main/linux-64::libnghttp2-1.52.0-ha637b67_1
  libnsl             pkgs/main/linux-64::libnsl-2.0.0-h5eee18b_0
  libopenblas        pkgs/main/linux-64::libopenblas-0.3.21-h043d6bf_0
  libpng             pkgs/main/linux-64::libpng-1.6.39-h5eee18b_0
  libssh2            pkgs/main/linux-64::libssh2-1.10.0-h37d81fd_2
  libstdcxx-ng       conda-forge/linux-64::libstdcxx-ng-13.1.0-hfd8a6a1_0
  libunistring       pkgs/main/linux-64::libunistring-0.9.10-h27cfd23_0
  libzlib            conda-forge/linux-64::libzlib-1.2.13-hd590300_5
  matplotlib-base    pkgs/main/linux-64::matplotlib-base-3.1.3-py37hef1b27d_0
  mummer             bioconda/linux-64::mummer-3.23-4
  ncurses            pkgs/main/linux-64::ncurses-6.4-h6a678d5_0
  numpy              conda-forge/linux-64::numpy-1.21.6-py37h976b520_0
  openssl            pkgs/main/linux-64::openssl-1.1.1u-h7f8727e_0
  perl               pkgs/main/linux-64::perl-5.34.0-h5eee18b_2
  perl-threaded      bioconda/noarch::perl-threaded-5.32.1-hdfd78af_1
  pip                conda-forge/noarch::pip-23.1.2-pyhd8ed1ab_0
  pyfastaq           bioconda/noarch::pyfastaq-3.17.0-py_2
  pymummer           bioconda/noarch::pymummer-0.11.0-pyhdfd78af_1
  pyparsing          pkgs/main/linux-64::pyparsing-3.0.9-py37h06a4308_0
  pysam              bioconda/linux-64::pysam-0.15.3-py37hda2845c_1
  python             conda-forge/linux-64::python-3.7.12-hb7a2778_100_cpython
  python-dateutil    pkgs/main/noarch::python-dateutil-2.8.2-pyhd3eb1b0_0
  python_abi         conda-forge/linux-64::python_abi-3.7-3_cp37m
  readline           pkgs/main/linux-64::readline-8.2-h5eee18b_0
  samtools           bioconda/linux-64::samtools-1.6-hcd7b337_9
  setuptools         conda-forge/noarch::setuptools-68.0.0-pyhd8ed1ab_0
  six                pkgs/main/noarch::six-1.16.0-pyhd3eb1b0_1
  soupsieve          pkgs/main/noarch::soupsieve-2.3.1-pyhd3eb1b0_0
  spades             bioconda/linux-64::spades-3.13.0-0
  sqlite             pkgs/main/linux-64::sqlite-3.41.2-h5eee18b_0
  tk                 pkgs/main/linux-64::tk-8.6.12-h1ccaba5_0
  tornado            pkgs/main/linux-64::tornado-6.2-py37h5eee18b_0
  typing_extensions  pkgs/main/noarch::typing_extensions-4.1.1-pyh06a4308_0
  wget               pkgs/main/linux-64::wget-1.21.4-h91b91d3_1
  wheel              pkgs/main/linux-64::wheel-0.38.4-py37h06a4308_0
  xz                 conda-forge/linux-64::xz-5.2.6-h166bdaf_0
  zlib               conda-forge/linux-64::zlib-1.2.13-hd590300_5


Proceed ([y]/n)? y

Downloading and Extracting Packages
Preparing transaction: done
Verifying transaction: done
Executing transaction: done

that installed and now I deactivate the environment.

conda deactivate

Next I try to set-up a new environment

conda create -n AnotherEnv

output

Collecting package metadata (current_repodata.json): done
Solving environment: done

CondaError: An error occurred when loading cached repodata.  Executing
`conda clean --index-cache` will remove cached repodata files
so they can be downloaded again.

So this is my problem, I have to run conda clean after almost every step. I do not understand why.

Thomas_Haverkamp · June 28, 2023, 7:55am

For instance if I go to the directory with the cache, I see this:

thhaverk@SAGA.login-4 cache$ ls -lath
total 115M
drwxrwsr-x 317 thhaverk nn9305k  510 Jun 28 09:31 ../
drwxrwsr-x   2 thhaverk nn9305k   16 Jun 28 09:28 ./
-rw-rw-r--   1 thhaverk nn9305k  564 Jun 28 09:46 f1f1f480.info.json
-rw-rw-r--   1 thhaverk nn9305k  560 Jun 28 09:46 227b7582.info.json
-rw-rw-r--   1 thhaverk nn9305k  570 Jun 28 09:46 9e0f62c3.info.json
-rw-rw-r--   1 thhaverk nn9305k  566 Jun 28 09:46 b35683e8.info.json
-rw-rw-r--   1 thhaverk nn9305k  291 Jun 28 09:28 18414ddb.info.json
-rw-rw-r--   1 thhaverk nn9305k  58M Jun 28 09:28 18414ddb.json
-rw-rw-r--   1 thhaverk nn9305k  289 Jun 28 09:28 62c98390.info.json
-rw-rw-r--   1 thhaverk nn9305k  20M Jun 28 09:28 62c98390.json
-rw-rw-r--   1 thhaverk nn9305k  290 Jun 28 09:28 a0b50a38.info.json
-rw-rw-r--   1 thhaverk nn9305k  14M Jun 28 09:28 a0b50a38.json
-rw-rw-r--   1 thhaverk nn9305k  288 Jun 28 09:28 e599ec5e.info.json
-rw-rw-r--   1 thhaverk nn9305k  10M Jun 28 09:28 e599ec5e.json
-rw-rw-r--   1 thhaverk nn9305k 4.6M Jun 28 09:28 227b7582.json
-rw-rw-r--   1 thhaverk nn9305k 6.4M Jun 28 09:28 9e0f62c3.json
-rw-rw-r--   1 thhaverk nn9305k 1.2M Jun 28 09:28 b35683e8.json
-rw-rw-r--   1 thhaverk nn9305k 2.6M Jun 28 09:28 f1f1f480.json

Next I clean it:

thhaverk@SAGA.login-4 cache$ conda clean -i
Will remove 1 index cache(s).
Proceed ([y]/n)? y

WARNING: cannot remove, file permissions: /cluster/projects/nn9305k/src/miniconda/pkgs/cache

Which is odd. since when I look now at the folder, I get this list of files

thhaverk@SAGA.login-4 cache$ ls -lath
total 1.0K
drwxrwsr-x   2 thhaverk nn9305k  16 Jun 28 09:52 ./
drwxrwsr-x 317 thhaverk nn9305k 510 Jun 28 09:31 ../

Everything was removed. So my feeling is that it has to do with permissions on our system, but I do not understand it.