TELEMAC-MASCARET Forum :: Topic: run_telfile.py merge with huge file eats all RAM and crash (1/1)

Welcome, Guest

TOPIC: run_telfile.py merge with huge file eats all RAM and crash

run_telfile.py merge with huge file eats all RAM and crash 1 month 6 days ago #44463

nicogodet
OFFLINE
Senior Boarder
Posts: 127
Thank you received: 31

Hi,

I tried to merge 2 files :: 11Go and 8,5Go with run_telfile.py merge.

$ run_telfile.py merge --kind=time reestuarisation_part_p1.res reestuarisation_part_p2.res test.res


Interpreting command line options
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  ~> Merging reestuarisation_part_p1.res, reestuarisation_part_p2.res into test.res kind=time
Skipping duplicate time 6566400.0 from reestuarisation_part_p2.res
Killed

I ran this on a machine with 32Go of RAM, it crashes.

I retried on 128Go RAM machine and it uses at maximum 75Go of RAM.
This is huge.

I think run_telfile.py merge could be improved by reading/writing by chunk (few timestep at a time) instead of first reading all files, store them in RAM and finally write to disk.

The administrator has disabled public write access.

run_telfile.py merge with huge file eats all RAM and crash 4 days 8 hours ago #44638

borisb
OFFLINE
Admin
Posts: 123
Thank you received: 58

Hi Nico,

This is indeed a problem because TelemacFile is not currently designed to handle direct writing to disk without going through a memory buffer, at least for the mesh part. You're not the first to point out this problem and I'm going to try to change this by proposing two file usage modes in the constructor, one enabling direct I/O to disk, the other going through a read-write memory buffer.

The administrator has disabled public write access.

The following user(s) said Thank You: nicogodet