KNOWLEDGE BASE

Tableau Server 2022.3.0 is Slow/Inaccessible After Upgrade


Published: 17 Oct 2022
Last Modified Date: 21 Nov 2022

Issue

After upgrading, or after restoring a backup to a new install to Tableau Server 2022.3.0, the server may be slow or inaccessible, with users unable to log in. 

Environment

  • Tableau Server 2022.3.0
  • Newly upgraded from earlier version of Tableau Server (both in-place upgrade and blue/green)

Resolution

Option 1 (preferred)
 

To avoid or mitigate this issue, upgrade to Tableau Server 2022.3.1 or later.

Option 2:

If an upgrade is not immediately possible, Tableau Development has provided a script to update the DeleteDuplicateSiteUsers job and prevent it from running, which will mitigate the issue.

On environments with an Active and a Passive Repository, perform these steps only on the Active Repository host machine:

1. Download the attached disableDuplicateUserRemovalJob script to the Tableau Server host.

Windows: disableDuplicateUserRemovalJob.cmd 
Linux: disableDuplicateUserRemovalJob.sh

2. Review the script contents in a text editor for instructions in line 13 on what to comment out or change based on whether the Active Repository is co-located with the TSM Controller. Save changes to the file.
3. Ensure that you are logged in as a local administrator on the system.
4. Linux only: grant the shell script executable permission: chmod +x disableDuplicateUserRemovalJob.sh
5. Run the script: 

Windows:c:\path_to_script\disableDuplicateUserRemovalJob.cmd
Linux: /path_to_script/disableDuplicateUserRemovalJob.sh

6. Confirm via the script output that the run_next_at value for the schedule is now set to 100 years from current time. Example output (from a Linux host, with key data to check for in red):

[devuser@hostname install]$ ./disableDuplicateUserRemovalJob.sh

pg_terminate_backend
----------------------
(0 rows)
UPDATE 1
run_next_at     | id  |                 name                 | active | priority | schedule_type | day_of_week_mask | day_of_month_mask | start_at_minute | minute_interval | end_at_minute | end_schedule_at |     run_next_at     |         created_at         |       updated_at        | hidden | serial_collection_id | lock_version | scheduled_action |                 luid                 | defined_by | timezoneid | actionable | linked_task_enabled
---------------------+-----+--------------------------------------+--------+----------+---------------+------------------+-------------------+-----------------+-----------------+---------------+-----------------+---------------------+----------------------------+-------------------------+--------+----------------------+--------------+------------------+--------------------------------------+------------+------------+------------+---------------------
2522-10-19 01:00:00 | 124 | Delete duplicate site users Schedule | t      |       10 |             0 |                0 |                 0 |              60 |            1440 |             0 |           | 2522-10-19 01:00:00 | 2022-10-18 05:48:16.402212 | 2022-10-18 05:58:37.879 | t      |                      |            1 |                0 | f6b7c7a4-cbef-4e5e-baf5-6aed4c369b54 |    0 |            | t          | f
(1 row)

Cause

A Backgrounder job responsible for detecting and removing duplicate users in the Repository may create a lock on (Repository) the users table in Postgres and cause delays on other activities. This behavior is resolved in 2022.3.1 under ID 1468293.

Additional Information

The DeleteDuplicateSiteUsers job, introduced in Tableau Server version 2022.3.0, will run as soon as the server is started after the upgrade or restore, on the first available Backgrounder, and then hourly afterward. In large, complex, or busy environments, this job can create an exclusive lock on the users table in the Tableau Server (Postgres) Repository that may cause delays in other activities on the server, most commonly:
  • very slow view loads 
  • inability for end users to log in 
  • slowness on actions for Site/Server Admins managing users & permissions. 

Identifying a match for this issue:

  • Confirm the full version of Tableau Server: 2022.3.0
  • Check the Backgrounder logs for the following message to confirm that the job is running when the issue is occurring:
"Start duplicate site users removal job."
 
The location of this log on a default Windows Tableau Server install is:

C:\ProgramData\Tableau\Tableau Server\data\tabsvc\logs\backgrounder\backgrounder_node<n>-<n>.log

On a Linux Tableau Server, the default location is:

/var/opt/tableau/tableau_server/data/tabsvc/logs/backgrounder/backgrounder_node<n>-<n>.log

  • In the logs for the Tableau Server (Postgres) Repository, there will be a message during the time of the issue showing that the rails@workgroup user has a lock with a wait queue. Note that the Backgrounder logs will have an offset for the local timezone compared to UTC, but that Postgres logs are in UTC. In the following example, the number - Process holding the lock: 5536 - is the pid of the process holding the lock, and the numbers in the wait queue are pids of other Tableau Server processes waiting for this lock to go away so they can access the Repository:

2022-10-05 01:00:43 UTC:10.202.54.172(56288):rails@workgroup:[14180]:DETAIL: Process holding the lock: 5536. Wait queue: 13482, 14176, 10593, 14180, 1975, 15812, 15818, 29241, 12850, 8358, 14860, 15998, 16000, 16002, 16003, 16004, 16006, 16008, 16009, 10676, 13616, 8401, 16011.

Did this article resolve the issue?