speed up GEDCOM import (advice from Tamura Jones)
Jul 17, 2024 15:42:09 GMT -8
Post by Uncle Buddy on Jul 17, 2024 15:42:09 GMT -8
Here's something I found in my do list:
# Import pandas and read the file in chunks, in RAM, and concatenate back together to get the resulting file, but keep it in RAM. Don't keep saving it over and over by different names, just do that once at the end. This is Tamura Jones' prescription to me: "Programs that manipulate GEDCOM in memory are the programs that import GEDCOM quickly. Writing to disc is what slows everything down." www.youtube.com/watch?v=97t9zmXeyD0
# Import pandas and read the file in chunks, in RAM, and concatenate back together to get the resulting file, but keep it in RAM. Don't keep saving it over and over by different names, just do that once at the end. This is Tamura Jones' prescription to me: "Programs that manipulate GEDCOM in memory are the programs that import GEDCOM quickly. Writing to disc is what slows everything down." www.youtube.com/watch?v=97t9zmXeyD0