Restarting the service and redeploying it should be absolutely the last resort and aren't really sound advice, mainly, because you are losing the invaluable crashed state of the system, that may be vital (sometimes logs are not enough) to discover _why_ the system crashed in the first place and then delivering a fix for that particular issue. Once that's done, you incorporate this into your infrastructure automation (having which goes without saying) be it Ansible, Terraform, Kubernetes or whatever else.
Otherwise you allow the problem to persist, pile up with other issues (also fixed by restarts, I assume) and implementing automated restarts in that manner reduces not only your uptime in uncontrollable manner, but also your code/infrastructure quality, increasing your tech debt beyond the point of recovery.
Friends don't let friends fixing things by restarting them ;)
Restarting the service and redeploying it should be absolutely the last resort and aren't really sound advice, mainly, because you are losing the invaluable crashed
I’m speaking in terms of AWS translate to your chosen infrastructure.
At the bare minimum you should have two redundant servers behind an autoscaling group with a min/max of two with health checks.
When you need to get something up now and you want to keep the crash state, configure the crash instance to be taken out the autoscaling group but not terminate and start up a new instance. You can then troubleshoot.
> Restarting the service and redeploying it should be absolutely the last resort and aren't really sound advice, mainly, because you are losing the invaluable crashed state of the system, that may be vital
I assume you have a separated logging mechanism, where all logs are collected, independently from the restarted service. Don't forget to log the state of your system as much as possible for post-mortem analysis
- add health check mechanisms
- if health check is broken => restart service
- if restart service doesn't help after X retry => redeploy previous state (if any available)
Try to use Kubernetes or Docker Swarm if possible, combined with Terraform