Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

> P.S. if you need to sort a huge file, just literally use the UNIX/Linux `sort` command. No, it does not load it all into RAM. It knows how to do chunked sorts, dump temp files into /tmp, and then merge them. Old school UNIX tools are smarter than you think.

This so much.

I’ve worked with many devs and admins that don’t understand the tools that they have at their disposal on their systems. They end up trying to reinvent the wheel and their solutions usually don’t consider all the edge cases



Note that this does take a while; I let it go for about two hours before killing sort.


     time gsort --parallel=2 -o $HOME/pwned-pass-sorted.txt pwned-passwords-sha1-ordered-by-count-v5.txt
         2890.06 real      1400.86 user       165.54 sys

48mins

`gsort` is GNU sort on coreutils (not the one included on macOS).

This is on a Mac Mini 2011 (5,1) with the 2.3GHz i5.

They really have a lot of things built into these tools :)




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: