Package server unresponsive

Hi, is there currently a problem with your servers by any chance? It seems downloads of certain packages have crawled to a halt almost. E.g. the download of this pytorch version: https://anaconda.org/pytorch/pytorch/1.9.0/download/linux-64/pytorch-1.9.0-py3.8_cuda11.1_cudnn8.0.5_0.tar.bz2 can take anywhere between 30min and hours, downloading at kb/s while other packages download at megabit speed.

Any help would be very welcome.

Cheers,
Thomas.

Hi Thomas,

Another user reported a similar issue earlier this week. I’m not sure whether it’s the same one, but would it be possible for you to review the request for a verbose curl log here to help with diagnosing the issue?

Thank you for reporting!

Sure, here is the first part of the verbose log:

 curl -v -L -o tmp.bz2 https://anaconda.org/pytorch/pytorch/1.9.0/download/linux-64/pytorch-1.9.0-py3.8_cuda11.1_cudnn8.0.5_0.tar.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 104.17.92.24:443...
* TCP_NODELAY set
* Connected to anaconda.org (104.17.92.24) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [104 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2319 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [116 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=anaconda.org
*  start date: Jun  5 00:00:00 2021 GMT
*  expire date: Jun  4 23:59:59 2022 GMT
*  subjectAltName: host "anaconda.org" matched cert's "anaconda.org"
*  issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x55a8fd009e30)
} [5 bytes data]
> GET /pytorch/pytorch/1.9.0/download/linux-64/pytorch-1.9.0-py3.8_cuda11.1_cudnn8.0.5_0.tar.bz2 HTTP/2
> Host: anaconda.org
> user-agent: curl/7.68.0
> accept: */*
> 
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
} [5 bytes data]
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0< HTTP/2 302 
< date: Tue, 11 Jan 2022 15:58:54 GMT
< content-type: text/html; charset=utf-8
< location: https://binstar-cio-packages-prod.s3.amazonaws.com/5e740d85fe6b2a019fe05e2c/60c3ff0e69aed84d147cb45a?response-content-disposition=attachment%3B%20filename%3D%22pytorch-1.9.0-py3.8_cuda11.1_cudnn8.0.5_0.tar.bz2%22%3B%20filename%2A%3DUTF-8%27%27pytorch-1.9.0-py3.8_cuda11.1_cudnn8.0.5_0.tar.bz2&response-content-type=application%2Fx-tar&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=60&X-Amz-Date=20220111T155854Z&X-Amz-SignedHeaders=host&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEBgaCXVzLWVhc3QtMSJGMEQCIDinowkK%2FbwipoNDp4NpBTBLYyVio8%2F9DiyAvE6DIrVxAiBoaH6cC5A5bRUouHbYtQUryGZfa2ABDN4UysE68l6VOir6AwgxEAAaDDQ1NTg2NDA5ODM3OCIMryHB4Kwku4xyOrP%2FKtcDV04taxltON28N6802GyrRjy%2BaUjPTXD8UARv69ZZmjr7F82qUHP4U1W1t09rgnq8gklcI1pZLH1igI54sQlDrs4GWa%2B9AmJn4w63uFuHwxe1%2BnYL7MPZdqipkQwFgJdci65DcOSjHrpm9nn5uPxexNvuh4u%2FLGJPf7urPYdcdhcJ0zvRD1VmMahXSpBzXjR0svybTZERVYTZjoPMkRe5N5kGqDQynEWPR5jmeyxR%2B2qHUAGEm8NnbEarYd0A88ZVH0SUq1AzRyZC9chC88vDwUdHhMfT1Nn6XT1Sf2jnSK9iX%2BnsldEQAGudTwRkyiBXmsHM6H1MenlFGloIJMGDLmNOcu%2FEWWUdLi8njm%2FGLAH2tKi9VVMl59Kct3jkShYrj5AZTU8A6RTUhGiCYMICv%2Bg5ZIj7jHQbnNO%2BZ2G4nE3IeoN4u58V1tfLkmRaK3zAx0w5NzGDGw6X0NAmrzzZSpcZEd%2FegPDXXhPV1KbZGuwnEdUhBBuAHEXPR%2B%2FkNEFdqRYZSINwlddqvfPjd8R98hCdu6b5SWZQfi8cBIdO%2FBoEVD9oWXU1ooUPdXe8zyxlsb1xvRQcxvkgU2h7nFKk2Zt%2BPsNTWt75TBIqbKcylOFjgmV1tBD%2BMMPB9o4GOqYBCIaDwICSZ%2FPpfF84VRkNf5pHTudxTnxkhqNqVWCr%2BblCTCEcLyNL4sUhjE6sWVxax6s%2Br1EQc9aK7ECK7hNlshCinUumZEotPmpHxUoW2Ijv43A%2BECL8RXCUxeLYDwOHYc5Hot0i1j3gDjux2gTBMiqwroQ5suSF8oJQ7e8zJ%2BaUvxEHGCABNCISNcv7yt7gjA7Qk7jeMEh4pAoKmAY8WIwSNJdtag%3D%3D&X-Amz-Credential=ASIAWUI46DZFOTD7HZ7D%2F20220111%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=1694c05be2e12188a211d7aea818eed36f0f9a413f46e29b037cbcccbbe30165
< strict-transport-security: max-age=31536000
< cf-cache-status: DYNAMIC
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< set-cookie: __cf_bm=s68c1sLrQE0pbM_r8q81F_PCZnwLUGAGAjzRVJpUNhU-1641916734-0-AfXJ1kiTHetQkmrmshKeb5na1k4QDrAP+xhVLD/Ra8MZA9hfImwY6r1/TIu8iYDPgaraLfwbLkoEuN6/ITY+BdbKD/43Y5jnCkznGmBrIfut; path=/; expires=Tue, 11-Jan-22 16:28:54 GMT; domain=.anaconda.org; HttpOnly; Secure; SameSite=None
< server: cloudflare
< cf-ray: 6cbf5962284869f4-MAD
< 
* Ignoring the response-body
{ [990 bytes data]
100  3705    0  3705    0     0   4608      0 --:--:-- --:--:-- --:--:--  4608
* Connection #0 to host anaconda.org left intact
* Issue another request to this URL: 'https://binstar-cio-packages-prod.s3.amazonaws.com/5e740d85fe6b2a019fe05e2c/60c3ff0e69aed84d147cb45a?response-content-disposition=attachment%3B%20filename%3D%22pytorch-1.9.0-py3.8_cuda11.1_cudnn8.0.5_0.tar.bz2%22%3B%20filename%2A%3DUTF-8%27%27pytorch-1.9.0-py3.8_cuda11.1_cudnn8.0.5_0.tar.bz2&response-content-type=application%2Fx-tar&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=60&X-Amz-Date=20220111T155854Z&X-Amz-SignedHeaders=host&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEBgaCXVzLWVhc3QtMSJGMEQCIDinowkK%2FbwipoNDp4NpBTBLYyVio8%2F9DiyAvE6DIrVxAiBoaH6cC5A5bRUouHbYtQUryGZfa2ABDN4UysE68l6VOir6AwgxEAAaDDQ1NTg2NDA5ODM3OCIMryHB4Kwku4xyOrP%2FKtcDV04taxltON28N6802GyrRjy%2BaUjPTXD8UARv69ZZmjr7F82qUHP4U1W1t09rgnq8gklcI1pZLH1igI54sQlDrs4GWa%2B9AmJn4w63uFuHwxe1%2BnYL7MPZdqipkQwFgJdci65DcOSjHrpm9nn5uPxexNvuh4u%2FLGJPf7urPYdcdhcJ0zvRD1VmMahXSpBzXjR0svybTZERVYTZjoPMkRe5N5kGqDQynEWPR5jmeyxR%2B2qHUAGEm8NnbEarYd0A88ZVH0SUq1AzRyZC9chC88vDwUdHhMfT1Nn6XT1Sf2jnSK9iX%2BnsldEQAGudTwRkyiBXmsHM6H1MenlFGloIJMGDLmNOcu%2FEWWUdLi8njm%2FGLAH2tKi9VVMl59Kct3jkShYrj5AZTU8A6RTUhGiCYMICv%2Bg5ZIj7jHQbnNO%2BZ2G4nE3IeoN4u58V1tfLkmRaK3zAx0w5NzGDGw6X0NAmrzzZSpcZEd%2FegPDXXhPV1KbZGuwnEdUhBBuAHEXPR%2B%2FkNEFdqRYZSINwlddqvfPjd8R98hCdu6b5SWZQfi8cBIdO%2FBoEVD9oWXU1ooUPdXe8zyxlsb1xvRQcxvkgU2h7nFKk2Zt%2BPsNTWt75TBIqbKcylOFjgmV1tBD%2BMMPB9o4GOqYBCIaDwICSZ%2FPpfF84VRkNf5pHTudxTnxkhqNqVWCr%2BblCTCEcLyNL4sUhjE6sWVxax6s%2Br1EQc9aK7ECK7hNlshCinUumZEotPmpHxUoW2Ijv43A%2BECL8RXCUxeLYDwOHYc5Hot0i1j3gDjux2gTBMiqwroQ5suSF8oJQ7e8zJ%2BaUvxEHGCABNCISNcv7yt7gjA7Qk7jeMEh4pAoKmAY8WIwSNJdtag%3D%3D&X-Amz-Credential=ASIAWUI46DZFOTD7HZ7D%2F20220111%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=1694c05be2e12188a211d7aea818eed36f0f9a413f46e29b037cbcccbbe30165'
*   Trying 54.231.139.17:443...
* TCP_NODELAY set
* Connected to binstar-cio-packages-prod.s3.amazonaws.com (54.231.139.17) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [91 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2777 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=Washington; L=Seattle; O=Amazon.com, Inc.; CN=*.s3.amazonaws.com
*  start date: Jan 11 00:00:00 2021 GMT
*  expire date: Feb 11 23:59:59 2022 GMT
*  subjectAltName: host "binstar-cio-packages-prod.s3.amazonaws.com" matched cert's "*.s3.amazonaws.com"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert Baltimore CA-2 G2
*  SSL certificate verify ok.
} [5 bytes data]
> GET /5e740d85fe6b2a019fe05e2c/60c3ff0e69aed84d147cb45a?response-content-disposition=attachment%3B%20filename%3D%22pytorch-1.9.0-py3.8_cuda11.1_cudnn8.0.5_0.tar.bz2%22%3B%20filename%2A%3DUTF-8%27%27pytorch-1.9.0-py3.8_cuda11.1_cudnn8.0.5_0.tar.bz2&response-content-type=application%2Fx-tar&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=60&X-Amz-Date=20220111T155854Z&X-Amz-SignedHeaders=host&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEBgaCXVzLWVhc3QtMSJGMEQCIDinowkK%2FbwipoNDp4NpBTBLYyVio8%2F9DiyAvE6DIrVxAiBoaH6cC5A5bRUouHbYtQUryGZfa2ABDN4UysE68l6VOir6AwgxEAAaDDQ1NTg2NDA5ODM3OCIMryHB4Kwku4xyOrP%2FKtcDV04taxltON28N6802GyrRjy%2BaUjPTXD8UARv69ZZmjr7F82qUHP4U1W1t09rgnq8gklcI1pZLH1igI54sQlDrs4GWa%2B9AmJn4w63uFuHwxe1%2BnYL7MPZdqipkQwFgJdci65DcOSjHrpm9nn5uPxexNvuh4u%2FLGJPf7urPYdcdhcJ0zvRD1VmMahXSpBzXjR0svybTZERVYTZjoPMkRe5N5kGqDQynEWPR5jmeyxR%2B2qHUAGEm8NnbEarYd0A88ZVH0SUq1AzRyZC9chC88vDwUdHhMfT1Nn6XT1Sf2jnSK9iX%2BnsldEQAGudTwRkyiBXmsHM6H1MenlFGloIJMGDLmNOcu%2FEWWUdLi8njm%2FGLAH2tKi9VVMl59Kct3jkShYrj5AZTU8A6RTUhGiCYMICv%2Bg5ZIj7jHQbnNO%2BZ2G4nE3IeoN4u58V1tfLkmRaK3zAx0w5NzGDGw6X0NAmrzzZSpcZEd%2FegPDXXhPV1KbZGuwnEdUhBBuAHEXPR%2B%2FkNEFdqRYZSINwlddqvfPjd8R98hCdu6b5SWZQfi8cBIdO%2FBoEVD9oWXU1ooUPdXe8zyxlsb1xvRQcxvkgU2h7nFKk2Zt%2BPsNTWt75TBIqbKcylOFjgmV1tBD%2BMMPB9o4GOqYBCIaDwICSZ%2FPpfF84VRkNf5pHTudxTnxkhqNqVWCr%2BblCTCEcLyNL4sUhjE6sWVxax6s%2Br1EQc9aK7ECK7hNlshCinUumZEotPmpHxUoW2Ijv43A%2BECL8RXCUxeLYDwOHYc5Hot0i1j3gDjux2gTBMiqwroQ5suSF8oJQ7e8zJ%2BaUvxEHGCABNCISNcv7yt7gjA7Qk7jeMEh4pAoKmAY8WIwSNJdtag%3D%3D&X-Amz-Credential=ASIAWUI46DZFOTD7HZ7D%2F20220111%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=1694c05be2e12188a211d7aea818eed36f0f9a413f46e29b037cbcccbbe30165 HTTP/1.1
> Host: binstar-cio-packages-prod.s3.amazonaws.com
> User-Agent: curl/7.68.0
> Accept: */*
> 
{ [5 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< x-amz-id-2: WURN2yC0oMKD6vCipgA+t3lRJ+bTyNp54SdpHxuxyOAiDyAvOWs28Nqhy4ZVmp9at4uJkBgD0NY=
< x-amz-request-id: WTQ08HD0H574T6YM
< Date: Tue, 11 Jan 2022 15:58:55 GMT
< Last-Modified: Sat, 12 Jun 2021 00:25:54 GMT
< ETag: "ecab10144540d49a514f4d7863637380"
< Content-Disposition: attachment; filename="pytorch-1.9.0-py3.8_cuda11.1_cudnn8.0.5_0.tar.bz2"; filename*=UTF-8''pytorch-1.9.0-py3.8_cuda11.1_cudnn8.0.5_0.tar.bz2
< Accept-Ranges: bytes
< Content-Type: application/x-tar
< Server: AmazonS3
< Content-Length: 1542100237
< 
{ [5 bytes data]

And here a screenshot of the progress: