After Running Out Of Hard Disk Space A Few Processes Are Not Able To Start

Issue

After running out of disk space then freeing up space or expanding more disk space, Tableau Server continues to show as DEGRADED and a few processes (like Gateway, Data Engine, Elastic Server, etc) are in an error state.

In servicestate.json, the "currentState" of those error state processes may display as "INSTALL_FAILED". By default, the servicestate.json can be found in /var/opt/tableau/tableau_server/data/tabsvc/tabadminagent/0/servicestate.json

"hyper_0.20202.20.0425.1206" : {
"currentState" : "INSTALL_FAILED",
"requestedState" : "ENABLED"
},

"elasticserver_0.20202.20.0425.1206" : {
"currentState" : "INSTALL_FAILED",
"requestedState" : "ENABLED"
},

"gateway_0.20202.20.0425.1206" : {
"currentState" : "INSTALL_FAILED",
"requestedState" : "ENABLED"
},

Environment

Tableau Server 2020.2

Resolution

Option 1:

Try to modify the current state of error processes in servicestate.json

Stop Tableau Server (If the error processes were not able to stop successfully ignore it, we will perform an OS reboot)
- tsm stop
Stop TSM services by script
- Linux: sudo /opt/tableau/tableau_server/packages/scripts.<Build number>/stop-administrative-services
- Windows: C:\Program Files\Tableau\Tableau Server\packages\scripts.<Build number>\stop-administrative-services.cmd
Edit the servicestate.json manually, make sure the current state of all processes have "currentState" : "ENABLED". Same goes for requestedState. By default, servicestate.json located in the following directory:
- Linux: /var/opt/tableau/tableau_server/data/tabsvc/tabadminagent/0/servicestate.json
- Windows: C:\ProgramData\Tableau\Tableau Server\data\tabsvc\tabadminagent\0\servicestate.json
Reboot OS
Check if TSM services started automatically
- tsm status -v
If TSM services didn't start automatically, Start TSM services by script.
- Linux: sudo /opt/tableau/tableau_server/packages/scripts.<Build number>/start-administrative-services
- Windows: C:\Program Files\Tableau\Tableau Server\packages\scripts.<Build number>\start-administrative-services.cmd
Start Tableau Server
- tsm start
Check if the status of Tableau Server is "RUNNING"
- tsm status -v
If a few processes still in error state, try to restart Tableau Server one or two additional times
- tsm restart

Option 2:

Perform a clean install then restore the data in the same machine

Creating the backup and gathering any assets that require manual actions. See Backing up Tableau Server for recovery
Copy the backup files (.tsbak, .json) to a non-Tableau Server related directory
Completely remove Tableau Server without removing server licensing. See To completely remove Tableau Server without removing server licensing
Install the latest maintenance release of the same version. Find the latest maintenance release from Product Downloads and Release Notes
Restoring core Tableau Server functionality and Restore other functionality. See Perform a Full Backup and Restore of Tableau Server

Cause

After running out of disk space and some of the processes turn to degraded status, in some cases, Tableau Server may identify the status of processes incorrectly even after the disk issue resolved and generating the wrong servicestate.json file.

Additional Information

For preventing this issue, please monitor free disk space and enable email notifications about low disk space. For more information, see Configure Server Event Notification