KNOWLEDGE BASE

High memory usage when publishing a large hyper files to Tableau Server


Published: 19 Nov 2022
Last Modified Date: 25 Nov 2022

Issue

When publishing a large hyper file to Tableau Server from the outside of Tableau Server, such as using Tableau Desktop, tabcmd, or Hyper API, a high memory usage was observed.
For example, when publishing a 30GB hyper file to Tableau Server, a more than 50GB memory usage can be observed.
It causes that the other processes (such as vizportal, backgrounder, vizsqlserver, etc) are terminated by Server Resource Manager for releasing the memory, if the free memory is less than 50GB before the 30GB hyper being published.

As additional information, the following messages can be seen in hyper_0_xxxx.txt log during the hyper file being published toTableau Server.

{"ts":"2022-10-14T06:37:11.571","pid":29240,"tid":"623c","sev":"info","req":"3","sess":"AAAAA2BbBBBBBCC3DDDDEE","ctx":{"sess":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX-1:0","req":"YYYYYYYYYYYYYYYYYYYYYYYYYYY","site":"sitename","user":"username","host":"servername","process":"vizportal","workload":"interactive"},"k":"query-begin","v":{"spooling":false,"query-settings-active":false,"query-hash":"abcdefghijklmnopqrstuvwx","query":"VALIDATE DATABASE \"C:/ProgramData/Tableau/Tableau Server/data/tabsvc/temp/vizportal_0.xxxxxxxx/uploadExternalFiles1234567890/extract/hyper_filename.hyper\""}}

 

Environment

  • Tableau Server 2021.1 and later

Resolution

Option 1:
Use extract refresh to update hyper files on Tableau Server side instead of publishing hyper files from the outside of Tableau Server.

Option 2:
Reduce the size of the extracts (.hyper files).

Option 3:
Add the memory of Tableau Server machine.

Option 4:
Add a dedicated Data Engine (Hyper) node.

Cause

This is a function by design.
On Tableau Server 2021.1 and later, Tableau Server starts to validate uploaded extracts (.hyper files) during publishing to see if they are corrupted because they are created on the outside of Tableau Server.
This validating process makes Hyper load the uploaded extract (.hyper file) into the memory for consistency checking, and usually this needs more memory size which is more than the uploaded extract (.hyper file) size, which depends on the compression ratio of the hyper file and the related hyper database files loading.




 

Additional Information

This validating process does not run to a hyper file which is update by extract refresh on Tableau Server side.
Did this article resolve the issue?