Hi, is there currently a problem with your servers by any chance? It seems downloads of certain packages have crawled to a halt almost. E.g. the download of this pytorch version: https://anaconda.org/pytorch/pytorch/1.9.0/download/linux-64/pytorch-1.9.0-py3.8_cuda11.1_cudnn8.0.5_0.tar.bz2 can take anywhere between 30min and hours, downloading at kb/s while other packages download at megabit speed.
Any help would be very welcome.
Cheers,
Thomas.
Hi Thomas,
Another user reported a similar issue earlier this week. I’m not sure whether it’s the same one, but would it be possible for you to review the request for a verbose curl log here to help with diagnosing the issue?
Thank you for reporting!
Sure, here is the first part of the verbose log:
curl -v -L -o tmp.bz2 https://anaconda.org/pytorch/pytorch/1.9.0/download/linux-64/pytorch-1.9.0-py3.8_cuda11.1_cudnn8.0.5_0.tar.bz2
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* Trying 104.17.92.24:443...
* TCP_NODELAY set
* Connected to anaconda.org (104.17.92.24) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [104 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2319 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [116 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [37 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-ECDSA-CHACHA20-POLY1305
* ALPN, server accepted to use h2
* Server certificate:
* subject: C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=anaconda.org
* start date: Jun 5 00:00:00 2021 GMT
* expire date: Jun 4 23:59:59 2022 GMT
* subjectAltName: host "anaconda.org" matched cert's "anaconda.org"
* issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
} [5 bytes data]
* Using Stream ID: 1 (easy handle 0x55a8fd009e30)
} [5 bytes data]
> GET /pytorch/pytorch/1.9.0/download/linux-64/pytorch-1.9.0-py3.8_cuda11.1_cudnn8.0.5_0.tar.bz2 HTTP/2
> Host: anaconda.org
> user-agent: curl/7.68.0
> accept: */*
>
{ [5 bytes data]
* Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
} [5 bytes data]
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0< HTTP/2 302
< date: Tue, 11 Jan 2022 15:58:54 GMT
< content-type: text/html; charset=utf-8
< location: https://binstar-cio-packages-prod.s3.amazonaws.com/5e740d85fe6b2a019fe05e2c/60c3ff0e69aed84d147cb45a?response-content-disposition=attachment%3B%20filename%3D%22pytorch-1.9.0-py3.8_cuda11.1_cudnn8.0.5_0.tar.bz2%22%3B%20filename%2A%3DUTF-8%27%27pytorch-1.9.0-py3.8_cuda11.1_cudnn8.0.5_0.tar.bz2&response-content-type=application%2Fx-tar&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=60&X-Amz-Date=20220111T155854Z&X-Amz-SignedHeaders=host&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEBgaCXVzLWVhc3QtMSJGMEQCIDinowkK%2FbwipoNDp4NpBTBLYyVio8%2F9DiyAvE6DIrVxAiBoaH6cC5A5bRUouHbYtQUryGZfa2ABDN4UysE68l6VOir6AwgxEAAaDDQ1NTg2NDA5ODM3OCIMryHB4Kwku4xyOrP%2FKtcDV04taxltON28N6802GyrRjy%2BaUjPTXD8UARv69ZZmjr7F82qUHP4U1W1t09rgnq8gklcI1pZLH1igI54sQlDrs4GWa%2B9AmJn4w63uFuHwxe1%2BnYL7MPZdqipkQwFgJdci65DcOSjHrpm9nn5uPxexNvuh4u%2FLGJPf7urPYdcdhcJ0zvRD1VmMahXSpBzXjR0svybTZERVYTZjoPMkRe5N5kGqDQynEWPR5jmeyxR%2B2qHUAGEm8NnbEarYd0A88ZVH0SUq1AzRyZC9chC88vDwUdHhMfT1Nn6XT1Sf2jnSK9iX%2BnsldEQAGudTwRkyiBXmsHM6H1MenlFGloIJMGDLmNOcu%2FEWWUdLi8njm%2FGLAH2tKi9VVMl59Kct3jkShYrj5AZTU8A6RTUhGiCYMICv%2Bg5ZIj7jHQbnNO%2BZ2G4nE3IeoN4u58V1tfLkmRaK3zAx0w5NzGDGw6X0NAmrzzZSpcZEd%2FegPDXXhPV1KbZGuwnEdUhBBuAHEXPR%2B%2FkNEFdqRYZSINwlddqvfPjd8R98hCdu6b5SWZQfi8cBIdO%2FBoEVD9oWXU1ooUPdXe8zyxlsb1xvRQcxvkgU2h7nFKk2Zt%2BPsNTWt75TBIqbKcylOFjgmV1tBD%2BMMPB9o4GOqYBCIaDwICSZ%2FPpfF84VRkNf5pHTudxTnxkhqNqVWCr%2BblCTCEcLyNL4sUhjE6sWVxax6s%2Br1EQc9aK7ECK7hNlshCinUumZEotPmpHxUoW2Ijv43A%2BECL8RXCUxeLYDwOHYc5Hot0i1j3gDjux2gTBMiqwroQ5suSF8oJQ7e8zJ%2BaUvxEHGCABNCISNcv7yt7gjA7Qk7jeMEh4pAoKmAY8WIwSNJdtag%3D%3D&X-Amz-Credential=ASIAWUI46DZFOTD7HZ7D%2F20220111%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=1694c05be2e12188a211d7aea818eed36f0f9a413f46e29b037cbcccbbe30165
< strict-transport-security: max-age=31536000
< cf-cache-status: DYNAMIC
< expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< set-cookie: __cf_bm=s68c1sLrQE0pbM_r8q81F_PCZnwLUGAGAjzRVJpUNhU-1641916734-0-AfXJ1kiTHetQkmrmshKeb5na1k4QDrAP+xhVLD/Ra8MZA9hfImwY6r1/TIu8iYDPgaraLfwbLkoEuN6/ITY+BdbKD/43Y5jnCkznGmBrIfut; path=/; expires=Tue, 11-Jan-22 16:28:54 GMT; domain=.anaconda.org; HttpOnly; Secure; SameSite=None
< server: cloudflare
< cf-ray: 6cbf5962284869f4-MAD
<
* Ignoring the response-body
{ [990 bytes data]
100 3705 0 3705 0 0 4608 0 --:--:-- --:--:-- --:--:-- 4608
* Connection #0 to host anaconda.org left intact
* Issue another request to this URL: 'https://binstar-cio-packages-prod.s3.amazonaws.com/5e740d85fe6b2a019fe05e2c/60c3ff0e69aed84d147cb45a?response-content-disposition=attachment%3B%20filename%3D%22pytorch-1.9.0-py3.8_cuda11.1_cudnn8.0.5_0.tar.bz2%22%3B%20filename%2A%3DUTF-8%27%27pytorch-1.9.0-py3.8_cuda11.1_cudnn8.0.5_0.tar.bz2&response-content-type=application%2Fx-tar&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=60&X-Amz-Date=20220111T155854Z&X-Amz-SignedHeaders=host&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEBgaCXVzLWVhc3QtMSJGMEQCIDinowkK%2FbwipoNDp4NpBTBLYyVio8%2F9DiyAvE6DIrVxAiBoaH6cC5A5bRUouHbYtQUryGZfa2ABDN4UysE68l6VOir6AwgxEAAaDDQ1NTg2NDA5ODM3OCIMryHB4Kwku4xyOrP%2FKtcDV04taxltON28N6802GyrRjy%2BaUjPTXD8UARv69ZZmjr7F82qUHP4U1W1t09rgnq8gklcI1pZLH1igI54sQlDrs4GWa%2B9AmJn4w63uFuHwxe1%2BnYL7MPZdqipkQwFgJdci65DcOSjHrpm9nn5uPxexNvuh4u%2FLGJPf7urPYdcdhcJ0zvRD1VmMahXSpBzXjR0svybTZERVYTZjoPMkRe5N5kGqDQynEWPR5jmeyxR%2B2qHUAGEm8NnbEarYd0A88ZVH0SUq1AzRyZC9chC88vDwUdHhMfT1Nn6XT1Sf2jnSK9iX%2BnsldEQAGudTwRkyiBXmsHM6H1MenlFGloIJMGDLmNOcu%2FEWWUdLi8njm%2FGLAH2tKi9VVMl59Kct3jkShYrj5AZTU8A6RTUhGiCYMICv%2Bg5ZIj7jHQbnNO%2BZ2G4nE3IeoN4u58V1tfLkmRaK3zAx0w5NzGDGw6X0NAmrzzZSpcZEd%2FegPDXXhPV1KbZGuwnEdUhBBuAHEXPR%2B%2FkNEFdqRYZSINwlddqvfPjd8R98hCdu6b5SWZQfi8cBIdO%2FBoEVD9oWXU1ooUPdXe8zyxlsb1xvRQcxvkgU2h7nFKk2Zt%2BPsNTWt75TBIqbKcylOFjgmV1tBD%2BMMPB9o4GOqYBCIaDwICSZ%2FPpfF84VRkNf5pHTudxTnxkhqNqVWCr%2BblCTCEcLyNL4sUhjE6sWVxax6s%2Br1EQc9aK7ECK7hNlshCinUumZEotPmpHxUoW2Ijv43A%2BECL8RXCUxeLYDwOHYc5Hot0i1j3gDjux2gTBMiqwroQ5suSF8oJQ7e8zJ%2BaUvxEHGCABNCISNcv7yt7gjA7Qk7jeMEh4pAoKmAY8WIwSNJdtag%3D%3D&X-Amz-Credential=ASIAWUI46DZFOTD7HZ7D%2F20220111%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=1694c05be2e12188a211d7aea818eed36f0f9a413f46e29b037cbcccbbe30165'
* Trying 54.231.139.17:443...
* TCP_NODELAY set
* Connected to binstar-cio-packages-prod.s3.amazonaws.com (54.231.139.17) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
} [5 bytes data]
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
} [512 bytes data]
* TLSv1.3 (IN), TLS handshake, Server hello (2):
{ [91 bytes data]
* TLSv1.2 (IN), TLS handshake, Certificate (11):
{ [2777 bytes data]
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
{ [333 bytes data]
* TLSv1.2 (IN), TLS handshake, Server finished (14):
{ [4 bytes data]
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
} [70 bytes data]
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
} [1 bytes data]
* TLSv1.2 (OUT), TLS handshake, Finished (20):
} [16 bytes data]
* TLSv1.2 (IN), TLS handshake, Finished (20):
{ [16 bytes data]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
* subject: C=US; ST=Washington; L=Seattle; O=Amazon.com, Inc.; CN=*.s3.amazonaws.com
* start date: Jan 11 00:00:00 2021 GMT
* expire date: Feb 11 23:59:59 2022 GMT
* subjectAltName: host "binstar-cio-packages-prod.s3.amazonaws.com" matched cert's "*.s3.amazonaws.com"
* issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert Baltimore CA-2 G2
* SSL certificate verify ok.
} [5 bytes data]
> GET /5e740d85fe6b2a019fe05e2c/60c3ff0e69aed84d147cb45a?response-content-disposition=attachment%3B%20filename%3D%22pytorch-1.9.0-py3.8_cuda11.1_cudnn8.0.5_0.tar.bz2%22%3B%20filename%2A%3DUTF-8%27%27pytorch-1.9.0-py3.8_cuda11.1_cudnn8.0.5_0.tar.bz2&response-content-type=application%2Fx-tar&X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Expires=60&X-Amz-Date=20220111T155854Z&X-Amz-SignedHeaders=host&X-Amz-Security-Token=IQoJb3JpZ2luX2VjEBgaCXVzLWVhc3QtMSJGMEQCIDinowkK%2FbwipoNDp4NpBTBLYyVio8%2F9DiyAvE6DIrVxAiBoaH6cC5A5bRUouHbYtQUryGZfa2ABDN4UysE68l6VOir6AwgxEAAaDDQ1NTg2NDA5ODM3OCIMryHB4Kwku4xyOrP%2FKtcDV04taxltON28N6802GyrRjy%2BaUjPTXD8UARv69ZZmjr7F82qUHP4U1W1t09rgnq8gklcI1pZLH1igI54sQlDrs4GWa%2B9AmJn4w63uFuHwxe1%2BnYL7MPZdqipkQwFgJdci65DcOSjHrpm9nn5uPxexNvuh4u%2FLGJPf7urPYdcdhcJ0zvRD1VmMahXSpBzXjR0svybTZERVYTZjoPMkRe5N5kGqDQynEWPR5jmeyxR%2B2qHUAGEm8NnbEarYd0A88ZVH0SUq1AzRyZC9chC88vDwUdHhMfT1Nn6XT1Sf2jnSK9iX%2BnsldEQAGudTwRkyiBXmsHM6H1MenlFGloIJMGDLmNOcu%2FEWWUdLi8njm%2FGLAH2tKi9VVMl59Kct3jkShYrj5AZTU8A6RTUhGiCYMICv%2Bg5ZIj7jHQbnNO%2BZ2G4nE3IeoN4u58V1tfLkmRaK3zAx0w5NzGDGw6X0NAmrzzZSpcZEd%2FegPDXXhPV1KbZGuwnEdUhBBuAHEXPR%2B%2FkNEFdqRYZSINwlddqvfPjd8R98hCdu6b5SWZQfi8cBIdO%2FBoEVD9oWXU1ooUPdXe8zyxlsb1xvRQcxvkgU2h7nFKk2Zt%2BPsNTWt75TBIqbKcylOFjgmV1tBD%2BMMPB9o4GOqYBCIaDwICSZ%2FPpfF84VRkNf5pHTudxTnxkhqNqVWCr%2BblCTCEcLyNL4sUhjE6sWVxax6s%2Br1EQc9aK7ECK7hNlshCinUumZEotPmpHxUoW2Ijv43A%2BECL8RXCUxeLYDwOHYc5Hot0i1j3gDjux2gTBMiqwroQ5suSF8oJQ7e8zJ%2BaUvxEHGCABNCISNcv7yt7gjA7Qk7jeMEh4pAoKmAY8WIwSNJdtag%3D%3D&X-Amz-Credential=ASIAWUI46DZFOTD7HZ7D%2F20220111%2Fus-east-1%2Fs3%2Faws4_request&X-Amz-Signature=1694c05be2e12188a211d7aea818eed36f0f9a413f46e29b037cbcccbbe30165 HTTP/1.1
> Host: binstar-cio-packages-prod.s3.amazonaws.com
> User-Agent: curl/7.68.0
> Accept: */*
>
{ [5 bytes data]
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< x-amz-id-2: WURN2yC0oMKD6vCipgA+t3lRJ+bTyNp54SdpHxuxyOAiDyAvOWs28Nqhy4ZVmp9at4uJkBgD0NY=
< x-amz-request-id: WTQ08HD0H574T6YM
< Date: Tue, 11 Jan 2022 15:58:55 GMT
< Last-Modified: Sat, 12 Jun 2021 00:25:54 GMT
< ETag: "ecab10144540d49a514f4d7863637380"
< Content-Disposition: attachment; filename="pytorch-1.9.0-py3.8_cuda11.1_cudnn8.0.5_0.tar.bz2"; filename*=UTF-8''pytorch-1.9.0-py3.8_cuda11.1_cudnn8.0.5_0.tar.bz2
< Accept-Ranges: bytes
< Content-Type: application/x-tar
< Server: AmazonS3
< Content-Length: 1542100237
<
{ [5 bytes data]
Hi Thomas. Very sorry for falling off the map on this one. I’ve raised this issue through our internal escalation process. Have you still been seeing the issue? Is it consistent or intermittent?
