I'm also using a Nexus 5X on Nougat and it seems to me that the restarts are caused by the RAM filling up so the OS has to kill processes to be able to keep the active app(s) running. With only Pokémon Go running I'm using 1.6GB out of the 1.8GB total usable memory. The last ~200MB will probably not be enough to run another app so a process has to be killed to make space and since Pokémon Go is the only other running app it gets killed, requiring a restart when next opened.
The only ways around this are, either never exit Pokémon Go or get more RAM which means a new device.