Thanks for the write-up! Two notes from my experience: pgf output works well with latex as well (although will slowdown compilation), and I recommend not using the pyplot submodule, especially if you'll be running things remotely over ssh and don't have a display
I had problems using pyplot over ssh because it can assume there's a display and fail when it couldn't find one.
Maybe this has changed. I use the OO interface. For example https://matplotlib.org/gallery/api/agg_oo_sgskip.html
Indeed, I remember having to switch to Agg for this reason but can't remember why I switched back. Maybe some rendering issues I had with Agg, not sure...