KNOWLEDGE BASE

Extract Refresh Error - "Failed to Upload Extract Folder to File Store: No More Writable Hosts"


Published: 26 Apr 2019
Last Modified Date: 26 May 2022

Issue

When refreshing an extract on Tableau Server, an error might occur:

File Store: no more writeable hosts.

Environment

  • Tableau Server 2020.1 and newer

Resolution

Option 1

Add a File Store process to the backgrounder node. Review Disk I/O Analysis from Logshark as well as deployment topology. If high disk queue length observed, Tableau Server will need additional File Store process in the environment. 

An example: 
User-added image

Option 2

Work with your IT team to ensure that Tableau Server processes are not being blocked by folder permission settings, firewalls, or scanning software.

Option 3

Follow these steps to update the port being used:
1. Verify if the new port is already being used (Linux example):
     
      sudo netstat -anp | grep 8999

2. If the port is not used, update the File Store port:
     
    tsm topology set-ports --node-name node1 --port-name filestore:primary --port-value 8999


 

Cause

Limited resources and security policies are preventing the processes on the worker node from writing to the folders on the primary node.

A third cause is the backgrounder is attempting to connect to the File Store using a different port than what the File Store is using.

Additional Information

To help identify if the port needs an adjustment, review the following example:
From the Backgrounder log:

2021-04-15 19:13:08.166 +0530 (Default,,,FF27DA817E92483A83261683FD7646AD-1:1,545240,:refresh_extracts,18b6ca0c-cffd-44d5-9311-349faf55bb9b) pool-12-thread-1 backgrounder: ERROR com.tableausoftware.tdfs.client.FileStoreServiceProvider - Failed connection attempt to localhost
org.apache.thrift.transport.TTransportException: Could not connect to the required filestore.
    at com.tableausoftware.tdfs.client.FileStoreServiceProvider.connect(FileStoreServiceProvider.java:224)
    at com.tableausoftware.tdfs.client.FileStoreServiceProvider.lambda$connectWithRetry$1(FileStoreServiceProvider.java:179)
...
Caused by: org.apache.thrift.transport.TTransportException: Could not get a resource from the pool, hostAndPort=localhost:8691
    at com.tableausoftware.tdfs.client.FileStoreClientPool.getResource(FileStoreClientPool.java:60)
    at com.tableausoftware.tdfs.client.FileStoreServiceProvider.connectToHost(FileStoreServiceProvider.java:444)
    at com.tableausoftware.tdfs.client.FileStoreServiceProvider.connect(FileStoreServiceProvider.java:222)
    ... 65 more
Caused by: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused (Connection refused)
    at org.apache.thrift.transport.TSocket.open(TSocket.java:226)
    at com.tableausoftware.service.thrift.TSocketWithRetry.open(TSocketWithRetry.java:106)
    at com.tableausoftware.tdfs.client.FileStoreClientFactory.makeObject(FileStoreClientFactory.java:42)
    at com.tableausoftware.tdfs.client.FileStoreClientFactory.makeObject(FileStoreClientFactory.java:32)
    at org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:1041)
    at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:357)
    at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:279)
    at com.tableausoftware.tdfs.client.FileStoreClientPool.getResource(FileStoreClientPool.java:57)
    ... 67 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
    at java.net.PlainSocketImpl.socketConnect(Native Method)
    at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
    at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:607)
    at org.apache.thrift.transport.TSocket.open(TSocket.java:221)
    ... 74 more

2021-04-15 19:13:08.370 +0530 (Default,,,,545240,:refresh_extracts,18b6ca0c-cffd-44d5-9311-349faf55bb9b) pool-12-thread-1 backgrounder: WARN  com.tableausoftware.model.workgroup.workers.RefreshExtractsWorker - |status=ExtractTimingFailure|jobId=545240|jobLuid=18b6ca0c-cffd-44d5-9311-349faf55bb9b|siteName="Default"|workbookName="Digital Showroom Live Data Tracking"|refreshedAt="2021-04-15T11:33:54.702Z"|sessionId=FF27DA817E92483A83261683FD7646AD-1:1|scheduleName="DS Live Data Schedule"|scheduleType="FullRefresh"|jobName="Refresh Extracts"|jobType="RefreshExtracts"|totalTimeSeconds=778|runTimeSeconds=774|queuedTime="2021-04-15T13:30:09.940Z"|startedTime="2021-04-15T13:30:13.541Z"|endTime="2021-04-15T13:43:08.368Z"|correlationId=127|priority=1|serialId=716557653|extractsSizeBytes=0|jobNotes="java.lang.Exception: Failed to upload extract folder '/var/opt/tableau/tableau_server/data/tabsvc/temp/backgrounder_1.20201.20.0220.1252/tdfs7208069647451346869/{FC59A67E-6548-4122-8569-AF424449AF6B}' to File Store: no more writable hosts"
java.lang.RuntimeException: java.lang.Exception: Failed to upload extract folder 


From the Filestore workgroup.yml port 8478 is used. This can also be verified through tsm topology list-ports | findstr filestore
filestore.port: 8478
Did this article resolve the issue?