Very large data csv file of 18Gb

Getting Started How do I...?
1

Please how do i load a very large dataset of 18Gb csv into jupyter without crashing

2

This is the typical use case for what is called ‘data-chunking’, where you read the data file in chunks into memory, ‘n’ lines at a time using essentially a loop. You can use the ‘chunksize’ argument to read say 100000 lines at a time.

See this post for an example:

1 Like
3

You will most likely also need to increase the memory available to your jupyter notebook:

4

Thank you, i will try it out.