Tableau Server Nodes Stuck in Error State after TSM Restart

Issue

After restarting a Tableau Server cluster, nodes may be stuck in an error state and TSM cannot be started with the following TabsvcHandler error logged:

+ \logs.zip\logs\tabsvc\tabsvc_node2-0.log:

2018-12-15 09:09:54.780 -0600  Thread-1 : INFO  com.tableausoftware.tabsvc.TabSvcHandler - Resurrection data file path set to: F:\Tableau Server\data\tabsvc\tabsvc\0\tabsvc-service-resurrection-data_tabsvc-clusterid.json
2018-12-15 09:09:54.780 -0600  Thread-1 : INFO  com.tableausoftware.tabsvc.TabSvcHandler - Found resurrection data file F:\Tableau Server\data\tabsvc\tabsvc\0\tabsvc-service-resurrection-data_tabsvc-clusterid.json. Will now try to resurrect the services.
2018-12-15 09:09:54.785 -0600  Thread-1 : ERROR com.tableausoftware.tabsvc.TabSvcHandler - Failed to load resurrection data file: F:\Tableau Server\data\tabsvc\tabsvc\0\tabsvc-service-resurrection-data_tabsvc-clusterid.json
        java.lang.RuntimeException: com.fasterxml.jackson.databind.JsonMappingException: No content to map due to end-of-input

Environment

Windows Server 2012 and 2016
Tableau Server with TSM (all current versions)

Resolution

Remove the failed nodes from the cluster, reinstall Tableau Server on the nodes, and add the nodes to the cluster again.

Remove the nodes in error state using the TSM UI, or using the command TSM topology remove-nodes --node-names "node2". This leaves the rest of the cluster in a good state.
Optional: Create a backup of your data before removing Tableau, and save the backup file to a safe location on a computer that is not part of your Tableau installation. Note: By default these are created in C:\ProgramData\Tableau\Tableau Server with a name similar to "uninstall-2018.1.tsbak". Saving backup files on a separate computer will allow you to reinstall the version of Tableau Server that created the backup, and have the data contained in the backup, even after running the tableau-server-obliterate script.
After verifying all Tableau data is backed up from the node, run the Tableau-Server-Obliterate script on each failed node in the cluster, the default location being: C:\Program Files\Tableau\Tableau Server\packages\scripts.<version_code>\tableau-server-obliterate.cmd -y -y –y
Reinstall Tableau Server on the desired nodes using the TSM web interface or TSM CLI.

The general steps for installing Tableau Server on an additional node are:

a. Confirm prerequisites
b. Generate the node bootstrap file 
c. Install an additional node
d. Configure an additional node
e. Install drivers

Note: You should add and configure additional nodes only when you can fully complete the process by applying pending changes. Adding a node without finishing by applying pending changes can result in users being unable to log into Tableau Server.

Apply Pending Changes and restart Tableau Server.
Verify the nodes are added to the cluster and successful, by >viewing the TSM Status page or web UI Server Status page

Cause

The tabsvc resurrection file (tabsvc-service-resurrection-data-*.json) was corrupted and this file is responsible for instantiating the running state of tabsvc in TSM. This can happen when disk space is exhausted.