Error "An error occurred on the server generating the backup" Backing Up or Upgrade Fails at 61%

Issue

When creating a backup of the Tableau Server Repository using tsm maintenance backup -po -f <filename> on a multi-node cluster with no File Store or Repository on the Tabadmincontroller node, the job returns the below error message, and no .tsbak is created:

"An error occurred on the server generating the backup."

Alternatively, when upgrading a Tableau Server from v2018.3 to v2019.2 with this configuration, the upgrade will not complete, with the following output:

61% - Uninstalling backup services.
Repository backup failed.
An error occurred during installation.
An error occurred during repository backup.

However, creating a full backup using tsm maintenance backup -f <filename> still works as expected.

Environment

Tableau Server 2018.3 - 2019.3
Windows Server

Resolution

Option 1:

Ensure that File Store and the Administration Controller ("tabadmincontroller.exe") are located on the same node.

Option 2:

As an alternative, create a full backup of Tableau Server, without using the -po argument. Note that this option is a workaround for backups created using TSM commands, and will not resolve the issue in a way that allows an upgrade to 2019.2.x to proceed.

Cause

The problem of not being able to complete a -po backup when File Store and Administration Controller are not located on the same node is related to a known issue with Id 947032 which has been fixed in releases 2018.2.14, 2018.3.11, 2019.1.8, 2019.2.4. The fix will also ship in 2019.3.1.

Additional Information

It is recommended in Distributed Installation Recommendations that the File Store and Repository processes are both co-located on the Administration Controller ("tabadmincontroller.exe") node.

For this issue, the Active Repository will specifically be on a node that was added later than the first node where there is currently a File Store. Example configuration:

node1 has the Administration Controller, but does not have File Store or Repository.
node2 has File Store but no Active Repository.
node3 has (either the Active or the only) Repository.