By: Christian Holslin

My Average Wednesday

The other day I was installing FAST Search Server 2010 for SharePoint onto two servers of a six-server SharePoint 2010 farm.  The instructions say you need to setup the Admin server first.  After hand-coding the elaborate deployment.xml file I copied it to the Admin server and ran the configuration wizard.

The first attempt failed.  I wasn’t fazed.  It usually fails the first time, either because you forget something or typo something or leave whitespace in the deployment.xml file… No sweat, double-check everything and run it again.

The second attempt failed: a little perplexing, but not uncommon.  After reviewing everything a second time I became slightly perturbed.  No worries, run it again.

The third attempt failed.

The fourth attempt failed.

After the fifth attempt I picked up the phone and dialed Microsoft Incident Support.  Hi, my name is Christian, yes that is my partner ID, this is concerning FAST Search Server 2010 for SharePoint… After speaking with the technical router for a few minutes I waited a couple hours for my call-back.  Lunch seemed appropriate; I grabbed my phone and took a walk outside.

My Favorite Support Call of All Time

Slightly more than two hours later I was exchanging email addresses with the next guy on the phone.  We immediately got down to business.  I sent him my deployment.xml file.  I sent all the specifications for the server.  I sent a copy of the log file output from the configuration wizard and finally screen shots of a number of different dialog boxes.  After dispatching the barrage of evidence he confirmed receipt and promptly put us on mute.  He told us he had to research the problem.

Two hours passed.

An eerie silence that persisted throughout was suddenly broken by the phrase, “You need to reduce the number of cores.”  A long pause; I tried my best to figure out what he was talking about.  Cores… you mean document processors; what exactly are “cores” in the context of FAST Search, I wondered silently to myself.

“Excuse me?” I asked in response.

“You need to reduce the number of cores in the BIOS,” he repeated, this time the operative word being “BIOS.”  It took a while to sink in.

“You mean I have to restart the machine, go into the BIOS, and disable all the extra CPU cores?”

“Yes,” he said. “You have too many cores for FAST Search.”  Keep in mind during this entire dialog he did not mention the words “processor,” “proc,” or “CPU.”  What he meant was that the computer onto which I was installing FAST Search was simply too fast for FAST Search, rather, for the FAST Search installer to execute properly, to be accurate.  The script was getting hung up on IIS as it attempted to update the web.config file for the Resource Store.  The exact error message is as follows:

10/27/2010 11:25:23 AM Warning Utility.Execute - "D:\FASTSearch\bin\ResourceStoreInstaller.exe" StdError - resourcestoreinstaller.exe - C:\Windows\system32\inetsrv\appcmd.exe SET config "ResourceStore" /section:requestFiltering /verbs.allowUnlisted:false /commit:apphost - ERROR ( message:Unknown attribute "verbs.allowUnlisted".  Replace with -? for help. ) 

10/27/2010 11:25:23 AM Warning Utility.Execute - Return code for binary "D:\FASTSearch\bin\ResourceStoreInstaller.exe" is not 0. This may indicate that binary didn't execute successfully

If you have too many CPU cores, or logical CPUs, the installer for the Resource Store will crash on this line as the CPUs compete for access to the web.config file.  It would appear that the appcmd.exe commands are executing in parallel across the various CPU cores leading to this glitch.

The true and actual cause was never revealed, but the problem was indeed solved by lowering the number of CPU cores to 1 each, and disabling HyperThreading, on each of the two processors.

“Wow,” I said to the support guy, and he hung up and closed the case.

Summary

The Microsoft technician recommended enabling no more than 4 CPU cores while installing FAST Search Server 2010 for SharePoint.  The server’s dual 6-core ass-kicking Nehalem Xeons flagrantly neglected this recommendation by packing a grand total of 12 physical CPU cores.  Sorry boys, we need to tone it down a notch… or eight.

If you run into this situation yourself don’t forget about HyperThreading (I didn’t ask the Microsoft support technician about that feature); that feature brings the number of logical CPUs to 24.  Here’s the math:

Processors X Cores X 2 for HyperThreading = # of Logical CPUs

2 X 6 X 2 = 24 Logical CPUs

“Seriously?” you ask.  Yep.  The good news is: you can enable them afterwards.

By: Christian Holslin