I've tried that, but in this case i would just NFS/sshfs share the directory and work with vim locally.

Vim can also read and write files across networks (while running locally): vim scp://user@myserver[:port]//path/to/file.txt should work.

If you want to run builds and such on a terminal, just have a tmux window with an active ssh connection.

There's a solution to most issues out there.