I think using a non-standard port is a good layer of security, among other layers.

My personal suggestion though is to use 1022 because it's below 1024. This means only root is allowed to bind to it. Preventing possible connection jacking attacks if an attacker is able to crash your own server and run theirs to harvest your passwords.