Yeah, it’s pretty memory intensive for the schema migration, modernc more so than native. I ran a few perf tests and created a couple of very primitive plots. This is a test setup with one fairly large folder; 20 358 files, 14 455 737 blocks.
This is initial migration (i.e., 1.x to 2.0) using either stock 2.0.5 or 2.0.5 compiled without CGO, time on the X axis (seconds) and maximum resident set size in bytes on the Y axis. The native C version finished in about half the time compared to the modernc version and used slightly less memory, ~400 vs ~450 MB.
Then I just edited the schema version back from 4 to 3 so the migration would run and restarted. The migration takes roughly the same amount of time for both versions, but the modernc variant has almost twice the peak memory usage, 3.9 GB vs 2.1 GB. Both results are higher than I’d expected/hoped, for sure.
I tried a third variant where I removed our cache size setting and order by in the migration query (blue line), but this made quite little difference. That was with the modernc variant because it’s easier to cross build, but I expect the results would be similar for the native version.
Finally I disabled just the VACUUM, and the result is slightly hard to see in the plot below but it’s the orange line that ends before the final rise in memory usage, at about 1.7 GB. So the second, larger, memory usage peak is all for the vacuum…
Finally I realised I didn’t properly remove our temp_store = MEMORY setting, and doing that brings the modernc variant down to below 450 MB:
I think that should be the shipped default, if we think it matters for performance we’d need to add it back as a tunable…