SQLServerWiki

“The Only Thing That Is Constant Is Change”

The fulltext filter daemon host (FDHost.exe) process has stopped abnormally.

Posted by database-wiki on November 23, 2013

From SQL Server Error Logs here’s what we have:-

spid56s The fulltext filter daemon host (FDHost) process has stopped abnormally. This can occur if an incorrectly configured or malfunctioning linguistic component, such as a wordbreaker, stemmer or filter has caused an irrecoverable error during full-text indexing or query processing. The process will be restarted automatically.
spid42s Failed Virtual Allocate Bytes: FAIL_VIRTUAL_RESERVE 65536

Memory Manager KB
—————————————- ———-
VM Reserved 8451272
VM Committed 253496
Locked Pages Allocated 0
Reserved Memory 1024
Reserved Memory In Use 0

MEMORYCLERK_SQLBUFFERPOOL (node 0) KB
—————————————- ———-
VM Reserved 8396800
VM Committed 200960
Locked Pages Allocated 0
SM Reserved 0
SM Committed 0
SinglePage Allocator 0
MultiPage Allocator 400

Memory node Id = 0 KB
—————————————- ———-
VM Reserved 8447624
VM Committed 249960
Locked Pages Allocated 0
MultiPage Allocator 26488
SinglePage Allocator 85696
2011-10-07 13:09:34.65 spid42s
Memory node Id = 64 KB
—————————————- ———-
VM Reserved 2560
VM Committed 2504
Locked Pages Allocated 0
MultiPage Allocator 2416
SinglePage Allocator 85696

=> It is quite evident that none of the Clerks were consuming memory, perfmon proves that as well.

Perfmon
=======

Counter Name Avg Max Min
————————————- ——————– ——————– ——————–
Memory\Available MBytes 6164.592 6322.000 6069.000
Memory\Cache Bytes 37049968.095 42672128.000 22970368.000
Process(fdhost)\Private Bytes 72012309.207 126963712.000 18948096.000
Process(fdhost)\Virtual Bytes 27535638746.130 55433195520.000 496201728.000
Process(fdlauncher)\Private Bytes 1773737.657 1810432.000 1728512.000
Process(fdlauncher)\Virtual Bytes 26888616.142 27893760.000 25796608.000
Process(sqlservr)\Private Bytes 273409090.651 279138304.000 257171456.000
Process(sqlservr)\Virtual Bytes 8928812910.580 8935145472.000 8918106112.000

=> So here’s a snapshot from Perfmon which indicates that fdhost.exe consumes all virtual memory (RAM+Page file) on the machine until the machine runs out of memory and fdhost dies.

virtualbytes-fdhost

Further from the crawl logs which was collected yesterday, here’s what we notice:

From SQLFT0000700005.LOG:-
{
spid18s Error ‘0x80070005’ occurred during full-text index population for table or indexed view ‘[Super7].[ram5].[TEXT]’ (table or indexed view ID ‘405576483’, database ID ‘7’), full-text key value ‘105135’. Attempt will be made to reindex it.
spid18s Informational: Full-text Auto population completed for table or indexed view ‘[Super7].[ram5].[TEXT]’ (table or indexed view ID ‘405576483’, database ID ‘7’). Number of documents processed: 1. Number of documents failed: 1. Number of documents that will be retried: 1.
spid33s A full-text retry pass of Auto population started for table or indexed view ‘[Super7].[ram5].[TEXT]’. Table or indexed view ID is ‘405576483’. Database ID is ‘7’.
spid32s Error ‘0x80070005’ occurred during full-text index population for table or indexed view ‘[Super7].[ram5].[TEXT]’ (table or indexed view ID ‘405576483’, database ID ‘7’), full-text key value ‘105135’. Attempt will be made to reindex it.
}

From SQLFT0000700005.1 :-
{
spid45s Error ‘0x80070005’ occurred during full-text index population for table or indexed view ‘[Super7].[ram5].[TEXT]’ (table or indexed view ID ‘405576483’, database ID ‘7’), full-text key value ‘16779605’. Attempt will be made to reindex it.
spid45s Error ‘0x80070005’ occurred during full-text index population for table or indexed view ‘[Super7].[ram5].[TEXT]’ (table or indexed view ID ‘405576483’, database ID ‘7’), full-text key value ‘16779606’. Attempt will be made to reindex it.


spid20s Error ‘0x80070005’ occurred during full-text index population for table or indexed view ‘[Super7].[ram5].[TEXT]’ (table or indexed view ID ‘405576483’, database ID ‘7’), full-text key value ‘16782193’. Attempt will be made to reindex it.
spid33s Error ‘0x80070005’ occurred during full-text index population for table or indexed view ‘[Super7].[ram5].[TEXT]’ (table or indexed view ID ‘405576483’, database ID ‘7’), full-text key value ‘16782194’. Attempt will be made to reindex it.

spid67s Informational: Full-text retry pass of Manual population completed for table or indexed view ‘[Super7].[ram5].[TEXT]’ (table or indexed view ID ‘405576483’, database ID ‘7’). Number of retry documents processed: 2343. Number of documents failed: 2343.

Followed by the below :-
{
spid28s Error ‘0x80070005’ occurred during full-text index population for table or indexed view ‘[Super7].[ram5].[TEXT]’ (table or indexed view ID ‘405576483’, database ID ‘7’), full-text key value ‘114045’. Attempt will be made to reindex it.
spid28s Informational: Full-text Auto population completed for table or indexed view ‘[Super7].[ram5].[TEXT]’ (table or indexed view ID ‘405576483’, database ID ‘7’). Number of documents processed: 1. Number of documents failed: 1. Number of documents that will be retried: 1.
spid47s A full-text retry pass of Auto population started for table or indexed view ‘[Super7].[ram5].[TEXT]’. Table or indexed view ID is ‘405576483’. Database ID is ‘7’.
spid43s Error ‘0x80070005’ occurred during full-text index population for table or indexed view ‘[Super7].[ram5].[TEXT]’ (table or indexed view ID ‘405576483’, database ID ‘7’), full-text key value ‘114045’. Attempt will be made to reindex it.
}

=> Error Code “0x80070005” implies “Access is Denied”. This would makes SQL server service unable to open and release the shared memory sections created by FDhost.exe.

Further to this From SQLFT0000700005.2 :-
{
spid83s Error ‘0x800705af’ occurred during full-text index population for table or indexed view ‘[Super7].[ram5].[TEXT]’ (table or indexed view ID ‘405576483’, database ID ‘7’), full-text key value ‘108279’. Attempt will be made to reindex it.
spid76s Error ‘0x800705af’ occurred during full-text index population for table or indexed view ‘[Super7].[ram5].[TEXT]’ (table or indexed view ID ‘405576483’, database ID ‘7’), full-text key value ‘108280’. Attempt will be made to reindex it.
spid73s Informational: Full-text retry pass of Manual population completed for table or indexed view ‘[Super7].[ram5].[TEXT]’ (table or indexed view ID ‘405576483’, database ID ‘7’). Number of retry documents processed: 15003. Number of documents failed: 15003.


spid44s Error ‘0x800705af’ occurred during full-text index population for table or indexed view ‘[Super7].[ram5].[TEXT]’ (table or indexed view ID ‘405576483’, database ID ‘7’), full-text key value ‘114045’. Attempt will be made to reindex it.
spid44s Informational: Full-text Auto population completed for table or indexed view ‘[Super7].[ram5].[TEXT]’ (table or indexed view ID ‘405576483’, database ID ‘7’). Number of documents processed: 1. Number of documents failed: 1. Number of documents that will be retried: 1.
spid30s A full-text retry pass of Auto population started for table or indexed view ‘[Super7].[ram5].[TEXT]’. Table or indexed view ID is ‘405576483’. Database ID is ‘7’.
spid36s Error ‘0x800705af’ occurred during full-text index population for table or indexed view ‘[Super7].[ram5].[TEXT]’ (table or indexed view ID ‘405576483’, database ID ‘7’), full-text key value ‘108281’. Attempt will be made to reindex it.
spid36s Error ‘0x800705af’ occurred during full-text index population for table or indexed view ‘[Super7].[ram5].[TEXT]’ (table or indexed view ID ‘405576483’, database ID ‘7’), full-text key value ‘108282’. Attempt will be made to reindex it.
}

=> Error Hex Code 0x800705af implies “The paging file is too small for this operation to complete.” which most likely is a result of the fact that Fdhost.exe has eaten up all the memory.

Now this is a known issue with Fdhost.exe which is documented under:
A memory leak occurs in FDHOST.exe process in SQL Server 2008 on a Windows Server 2003 Domain Controller
http://support.microsoft.com/kb/2014053

Please note that even though this talks about DC’s but upon further research I can confirm that the issue does not just occur on SQL Server instances running on a DC, it can even occur on any server where the account under which the FDHOST Launcher service is started does not have the correct permissions. We have in fact introduced a new rule in the SQL Server 2008 R2 Best Practice Analyser to check that the account used for the FDHOST Launcher service has the correct permissions or not.

CONCLUSION:-

Looks like the service account doesn’t have enough permission for which it is unable to open and release the shared memory sections created by FDhost.exe.

Please change the service account to a Domain account which has sufficient permission to get rid of the issue.

REFERENCES:-

2159124: How to setup the FDHOST Launcher service to work properly
http://support.microsoft.com/default.aspx?scid=kb;EN-US;2159124

How to: Set the FDHOST Launcher (MSSQLFDLauncher) Service Account for Full-Text Search (SQL Server Configuration Manager)
http://msdn.microsoft.com/en-us/library/ms345189.aspx

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: