The "requests" module

Technical Topics PyScript
1

Anyone else facing this: ModuleNotFoundError: No module named ‘requests’ ?
I’m building an webpage that shows informations about postal code here in Brazil. But it uses an API. Cant we use this library yet?

2 Likes
2

requests would be impossible to implement in a browser given the way WASM and requests currently work. I believe you must use pyfetch.

1 Like
3

I found this workaround via a requests shim that uses XMLHttpRequest under the hood:
https://bartbroere.eu/2022/04/25/pyodide-requests-binary-works-update/

1 Like
4

I’m also having this problem. I made some progress…

<html>
  <link rel="stylesheet" href="https://pyscript.net/alpha/pyscript.css" />
  <script defer src="https://pyscript.net/alpha/pyscript.js"></script>
  <py-env>
    - requests
  </py-env>
  <body>
    Hello, world!
    <div id="outputDiv" class="font-mono"></div>
    <py-script>
import requests
r = requests.get('https://myexternalip.com/raw')
pyscript.write("outputDiv", r.text)
    </py-script>
  </body>
</html>
  • Anything with http:// produces a “Protocol not available” error.
  • Anything with https:// produces a “Can’t connect to HTTPS URL because the SSL module is not available.” error.

I tried loading/importing PyOpenSSL in hopes to fix the https error, but it can’t install cryptography via micropip because it doesn’t have a pure python 3 wheel. It seems that Pyodide has added cryptography, libssl, and _ssl packages, so I’m unsure why there’s an error: Add cryptography, openssl, and _ssl packages by hoodmane · Pull Request #2263 · pyodide/pyodide · GitHub

You can use XMLHttpRequest as @jim.kring mentioned above, but this isn’t ideal imo

<html>
  <link rel="stylesheet" href="https://pyscript.net/alpha/pyscript.css" />
  <script defer src="https://pyscript.net/alpha/pyscript.js"></script>
  <body>
    Hello, world!
    <div id="outputDiv" class="font-mono"></div>
    <py-script>
from js import XMLHttpRequest

req = XMLHttpRequest.new()
req.open("GET", "https://myexternalip.com/raw", False)
req.send(None)

pyscript.write("outputDiv", str(req.response))
    </py-script>
  </body>
</html>

Keep in mind that if you use XMLHttpRequest in this manner that you need to make a web request that has some actual raw output / json. If not it’s going to pop a CORS error, you can disable this tho in your browser if you want.

5

Easiest way for synchronous text file fetch is using http.open_url from pyodide python api: Python API — Version 0.20.0

3 Likes
6

The requests module uses the operating system’s TCP socket API. Web browsers do not provide the socket API. This is a security restriction enforced by the browser and is not a limitation of PyScript.

You will need to rewrite your application to use the browser’s Fetch API. PyScript includes a wrapper function for fetch() called PyFetch(). I prefer to directly call fetch() because PyFetch() does not return the HTTP response headers (at this time).

2 Likes
7

Here is the link to the documentation, showing how to use js.fetch() correctly and access url content.


import json
from pyodide import to_js
from js import Object

request_params = {
  "method": "POST",
  "body": json.dumps({ "some" : "json" }),
  "credentials": "same-origin",
  "headers": { "Content-Type": "application/json" }
}

resp = await js.fetch(
    'example.com/some_api', 
    to_js(request_params), 
    dict_converter=Object.fromEntries),
)

Also see how pyodide.js.pyfetch can be used to download contents from a url.

1 Like
8

There is a problem with your js.fetch() example. The usage of dict_converter is wrong.

2 Likes
9

Problems with your example:

  1. dict_converter is parameter of to_js
  2. js is not imported (btw: there is same error in pyodide doc)
  3. you made only request, you probably need also response?```
  4. <py-script> and <py-repl> elements is not asynchronous so will fail with ‘await’ outside function error.
  5. as I write above pyodide.open_url is easiest way how to get request/response for PyScript,

This working javascript fetch example inside python code running with pyodide.runPythonAsync:

import json
from pyodide import to_js
import js
from js import Object

request_params = {
    "method": "POST",
    "body": json.dumps({ "some" : "json" }),
    "credentials": "same-origin",
    "headers": { "Content-Type": "application/json" }
}
req = await js.fetch('example.com/some_api', 
    to_js(request_params, dict_converter=js.Object.fromEntries)
)
res = await req.json()
js.JSON.stringify(res)
2 Likes
11

Does anyone know how to pass proxies in js.fetch or in pyodide.http.pyfetch… like we could do with python requests

proxies = {
  "http": "http://10.10.1.10:3128",
  "https": "https://10.10.1.10:1080",
}

requests.get("http://example.org", proxies=proxies)
12

In the browser, proxies are setup and managed by the browser and not in your code. Your example has no equivalent in the browser.

1 Like
13

Thanks for your answer, john ! By the way, i’m a great fan of your blog (PyScript part mostly :wink:)

1 Like
14

I am desperately trying to retrieve json data from the link below… I have tried all the solutions I have found, but nothing works for me. :face_with_spiral_eyes: :tired_face:
Any idea? It works with requests…

Élément

Amundi funds list

1 Like
15

Start with showing your code. Then add a description of the problem and the error messages from the browser debugger.

For coding problems, a good place is Stack Overflow.