Severe server slowdowns

I’ve had issues in the last 24 hours with conda timing out when trying to fetch repodata.json for a repository hosted on conda.anaconda.org. Specifically the conda-forge repo.

I first ran into the issue about 16 hours ago, across a number of machines, but it seemed to resolve itself in about an hour. However it started happening again about 1h30 ago.

I can currently replicate the issue outside of conda and across a number of different ISPs. For example I can run curl https://conda.anaconda.org/conda-forge/linux-64/repodata.json -O and it will reliably download up to about 60% of the json at the speeds I’d expect, and then stalls out and the transfer speed plummets to mere bits/sec.

Here is an example of this issue where the connection has been active for over 20 minutes:

I’m guessing that this has something to do with the fact that conda-forge is probably one of the largest (if not the largest repo) hosted on anaconda.org. However this is a severe degradation in performance that is not typical despite conda-forge’s size (in case that’s not obvious).

Does anyone have more info on what is causing this? Let me know if I’m barking up the wrong tree by posting here.

Than you for the report. I’ve forwarded this along to our infrastructure team to ask if they can check it out. Would it be possible for you to also share a verbose curl log?

Let me echo Matt’s thank you! We treat conda-forge specially precisely because of its size, so this is definitely a surprising find. That said, I do not find this same slowdown from my location—which is not within Anaconda’s internal network, mind you, so I am not being treated specially! So if you can indeed respond to Matt’s request for verbose log, we’d definitely appreciate it. This will help us determine exactly where within our server & CDN structure the slowdown seems to be appearing.

To give you a sense of what we’d be looking at, here is my verbose log of this curl command. Note in particular the handoff to CloudFlare.

 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 104.17.92.24...
* TCP_NODELAY set
* Connected to conda.anaconda.org (104.17.92.24) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
} [232 bytes data]
* TLSv1.2 (IN), TLS handshake, Server hello (2):
{ [100 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2319 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [114 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
{ [1 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=anaconda.org
*  start date: Jun  5 00:00:00 2021 GMT
*  expire date: Jun  4 23:59:59 2022 GMT
*  subjectAltName: host "conda.anaconda.org" matched cert's "*.anaconda.org"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7faa3a80f200)
> GET /conda-forge/linux-64/repodata.json HTTP/2
> Host: conda.anaconda.org
> User-Agent: curl/7.64.1
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
< HTTP/2 200 
< date: Fri, 07 Jan 2022 14:39:54 GMT
< content-type: application/json
< content-length: 165252103
< cf-ray: 6c9df02cdad5d2a6-DFW
< accept-ranges: bytes
< age: 3973
< cache-control: public, max-age=1200
< etag: "920537c2276f96fe135feeb9174923bc-20"
< expires: Fri, 07 Jan 2022 14:59:54 GMT
< last-modified: Fri, 07 Jan 2022 13:22:21 GMT
< vary: Accept-Encoding
< cf-cache-status: HIT
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< x-amz-id-2: xWV1tgHLA6Fq4gN6Uu63jCLX4hEOZ0I25rwWDIRgCz7KGetyBI4xdNbINY39c+A8K4Y2bVTaSr4=
< x-amz-request-id: CKZG3KJW9YAAG0DK
< x-amz-version-id: null
< set-cookie: __cf_bm=GQheCekKN6kOitXdHmZW5upW_2E2kqfRxwuMMrEqefo-1641566394-0-Ab1ZAl8fh1i4kXcmg4Two4X6QOlJHeMJ/TqzzOLOTeB+8Mdq2PMdpjyjXg8a/bOJwHhr6OHTKne55pCbk05IncWlFbFdCaT5/Adb5cPAza/C; path=/; expires=Fri, 07-Jan-22 15:09:54 GMT; domain=.anaconda.org; HttpOnly; Secure; SameSite=None
< server: cloudflare
< 
{ [731 bytes data]
100  157M  100  157M    0     0  11.0M      0  0:00:14  0:00:14 --:--:-- 11.1M
* Connection #0 to host conda.anaconda.org left intact
* Closing connection 0

For sure! The slowdown does seem to be correlated with which edge server we end up with as all the ISPs I previously tested were in the same geographic region. Specifically the cloudflare PoP in SLC:

$ curl https://conda.anaconda.org/conda-forge/linux-64/repodata.json -O -vvvv
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 104.17.92.24:443...
* Connected to conda.anaconda.org (104.17.92.24) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [104 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2319 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [116 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=anaconda.org
*  start date: Jun  5 00:00:00 2021 GMT
*  expire date: Jun  4 23:59:59 2022 GMT
*  subjectAltName: host "conda.anaconda.org" matched cert's "*.anaconda.org"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x5562ed6bac30)
} [5 bytes data]
> GET /conda-forge/linux-64/repodata.json HTTP/2
> Host: conda.anaconda.org
> user-agent: curl/7.76.1
> accept: */*
>
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
} [5 bytes data]
< HTTP/2 200
< date: Fri, 07 Jan 2022 15:11:04 GMT
< content-type: application/json
< content-length: 165302347
< cf-ray: 6c9e1dd78c9c27d4-SLC
< accept-ranges: bytes
< age: 151
< cache-control: public, max-age=1200
< etag: "2e4ff240bcbd4d88b936a1bf39d2411c-20"
< expires: Fri, 07 Jan 2022 15:31:04 GMT
< last-modified: Fri, 07 Jan 2022 14:52:57 GMT
< vary: Accept-Encoding
< cf-cache-status: HIT
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< x-amz-id-2: FoRWz6qqMi84y7I9PhkoP0g8Ta9m/h8Fq3htyKs6lLe+KhobtgNB7X6/phbyEVUUe5Hh+Zk/Qq0=
< x-amz-request-id: 9AP7SK7KC0PM418Y
< x-amz-version-id: null
< set-cookie: __cf_bm=VgFbpHjVW_hnk92S31kEZuGv0bpKuSXLT7vjTDqYjOc-1641568264-0-Adt/D7pt5VLt8KqOm9MVca+L23tjqDQ5Dp0uKmphzc4e3Ydn56S9hZnYbJ6Z46CQKq2Hz35NoYVx+fpvUJ7Ie6k/0WXqZOSlnrx/+6+2zAse; path=/; expires=Fri, 07-Jan-22 15:41:04 GMT; domain=.anaconda.org; HttpOnly; Secure; SameSite=None
< server: cloudflare
<
{ [731 bytes data]
   27  157M   27 43.8M    0     0  24671      0  1:51:40  0:31:04  1:20:36 18888

The issue is currently no longer manifesting on the linux-64 repo; however it has recently started again on the conda-forge/osx-64 repo.

Here is the relevent curl log:

$ curl  https://conda.anaconda.org/conda-forge/osx-64/repodata.json -O -vvvv
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 104.17.93.24:443...
* Connected to conda.anaconda.org (104.17.93.24) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [104 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2319 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [116 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=anaconda.org
*  start date: Jun  5 00:00:00 2021 GMT
*  expire date: Jun  4 23:59:59 2022 GMT
*  subjectAltName: host "conda.anaconda.org" matched cert's "*.anaconda.org"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x558ba7c0dc30)
} [5 bytes data]
> GET /conda-forge/osx-64/repodata.json HTTP/2
> Host: conda.anaconda.org
> user-agent: curl/7.76.1
> accept: */*
>
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
} [5 bytes data]
< HTTP/2 200
< date: Sat, 08 Jan 2022 00:36:22 GMT
< content-type: application/json
< content-length: 147453649
< cf-ray: 6ca159e91e492806-SLC
< accept-ranges: bytes
< age: 3055
< cache-control: public, max-age=1200
< etag: "6cdd75e592b834d0a2ac149f177b99a7-18"
< expires: Sat, 08 Jan 2022 00:56:22 GMT
< last-modified: Fri, 07 Jan 2022 23:35:52 GMT
< vary: Accept-Encoding
< cf-cache-status: HIT
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< x-amz-id-2: y4k/+GpirIUHtcAKwePizFAPD/W9QIQqvkqxOhrDcQVmwDzWpWzTwCAoJZEHYL9xxM6yJBeZA1E=
< x-amz-request-id: 9BFZSC12AHJGQNNP
< x-amz-version-id: null
< set-cookie: __cf_bm=pJswpli0Du4lEDqsKXSwfCQFKwjkLkl6Pv.hfMLNw4g-1641602182-0-ARg+9kE1WaDJsGvMHpXQsJaxAp0VHDeQJckTcD+5AewSt0fTDCZe3nqN546pdeK/223CIzSmR/+XIj9F4EyFLLauoz04W+cZfCaduiHQErQo; path=/; expires=Sat, 08-Jan-22 01:06:22 GMT; domain=.anaconda.org; HttpOnly; Secure; SameSite=None
< server: cloudflare
<
{ [727 bytes data]
 76  140M   76  107M    0     0   285k      0  0:08:24  0:06:24  0:02:00 21319

Hi, Hadn’t seen this issue and have reported a perhaps related issue with slow downloads here, in case it helps: Package server unresponsive - #3 by thomasb