Search Index deleted content over weekend - cannot do full crawl

Question

I've had a SharePoint 2010 server running since March, with a crawl containing around 32,000 searchable items. The crawling was setup to perform an incremental crawl nightly and a full crawl weekly. Until this last weekend it's been working just fine.

However, on Friday night when the last full crawl ran, it deleted every item from the index, and now when I try a full crawl it completes after 1 minute 40 seconds, with a single Top Level Error - http://myserver01:100 - The crawler could not communicate with the server. Check that the server is available and that the firewall access is configured correctly.

The server at http://myserver01:100 is extended to port 443 (for public access), and that extended web application works just fine. I can use the public facing side of the web application just fine, except for crawling.

If I remote desktop into the server box and try to access http://myserver01:100 I get errors saying that The resource object with key 'personalactions_menu_ak' was not found. This makes me think two things:

The web application at http://myserver01:100 is running, otherwise there wouldn't be any .NET errors (it would just show something like a HTTP 503 error?), and
There is a problem with the base web application - probably permissions. The error is saying that an out of the box resource key from wss.resx (which is confirmed as being in C:\inetpub\wwwroot\wss\VirtualDirectories\100\App_GlobalResources). I don't know if this would impact the crawl or not, as I would expect the crawl to look at the content database rather than try to render the pages?

I could really use some help trying to work out why my crawls won't work any more, and/or if the error with the resource keys might be the issue, and if so, how to resolve them. Again, they are the out of the box wss.resx files causing the issue, which I've not changed at all. I did add a custom resx file, but that's not appearing in the error messages.

---

A true software developer is one who, when told to 'Go to Hell', sees the 'go to', rather than the destination, as harmful.

Answer 1

Hi Kevin,

Check if your proxy server settings have changed. 

You can also specify the proxy setting for crawler from Central Admin console > Manage Service > Search Service > Search Administration> there is a Proxy Server setting that can be entered.

-chandra

---

Chandra

Answer 2

The content index got deleted because it can't access the site for some reason.  Since you can access the site externally, you need to look at that server and see why the site won't come up on that browser.  Don't focus on the resx files at first.  Ping the server name from the server and see if it resolves to the right address.  Also check the bindings of the web site in IIS and make sure that a hostheader exists that matches that name.  Then check alternate access mappings and makes sure the default zone for that web app also matches.

---

Corey Roth - SharePoint Server MVP blog: www.dotnetmafia.com twitter: @coreyroth

Answer 3

Chandra,

I checked the proxy, and it's currently set as 'None'. I don't know what proxy settings could be added there, but I don't believe we've ever had a proxy set up, so not sure that anything changed here.

Kevin

---

A true software developer is one who, when told to 'Go to Hell', sees the 'go to', rather than the destination, as harmful.

Answer 4

Corey,

I did a ping on myserver01, and it responded correctly (time<1ms). The bindings for the site in IIS (on port 100) have no Host Name (and I don't believe have ever had one). I added a Host Name, so the bindings now read:

type: http
Host Name: myserver01
Port: 100
IP Address: *
Binding Information: *blank*

Alternate Access Mappings are unchange, and are as follows:

http://myserver01:100 - Default - http://myserver01:100
https://www.mysite.com - Intranet - https://www.mysite.com
http://myserver01:12345 - Default - http://myserver01:12345 (for CA)

We're not able to find anything that has changed over the weekend, or in the last week on the server side of thing, but I do have custom code in a .WSP file that I create on a dev server, package into that .WSP file and then I run powershell to update the solution

stsadm -o upgradesolution -name mycode.wsp -filename mycode.asp -immediate -allowGacDeployment

I've tried putting previous versions of this .wsp file back (and running the same PS command). The assumptions I'm making are:

1. if I roll back to an earlier version of the .wsp, the above command is valid.
2. the solution is retracted first and the new one is added by that command
3. iisreset is not required (and reboot is definitely not required)

We're thinking it's more likely that something I did, by way of changing the code/configs on the server between the last successful full crawl (around May 12th) and the crawl that deleted the searchable items (May 19th) is the cause, but I did try rolling back those changes, without seeing any positive impact when trying another full crawl.

Any other ideas, based on the expanded information I just provided? Thanks for the help so far.

Kevin

---

A true software developer is one who, when told to 'Go to Hell', sees the 'go to', rather than the destination, as harmful.

Answer 5

Can you get to http://myserver01:100 from your desktop with the crawl reader account? 

---

de Nada

Answer 6

From my desktop? no. That port is not open on the server for external access. From the server I can open a browser to that site, but I get a .NET error (see the original question where I mention the .resx files)

---

A true software developer is one who, when told to 'Go to Hell', sees the 'go to', rather than the destination, as harmful.

Answer 7

ok.  what changes/rollback did you do?   Could you provide some more detail?   Thanks.

---

de Nada

Answer 8

The rollbacks I did were to use the following Powershell command, with an earlier version of my .wsp file, and then to follow the assumptions listed below.

stsadm -o upgradesolution -name mycode.wsp -filename mycode.asp -immediate -allowGacDeployment

The assumptions I'm making are:

1. if I roll back to an earlier version of the .wsp, the above command is valid.
2. the solution is retracted first and the new one is added by that command
3. iisreset is not required (and reboot is definitely not required)

My SharePoint hosting provider is telling me that as the 'base' website (on port 100) is showing .NET errors (welcome.ascx says 'The resource object with key 'personalactions_menu_ak' was not found.'), we need to resolve this issue first and then the search crawl should work. Does that make sense?

---

A true software developer is one who, when told to 'Go to Hell', sees the 'go to', rather than the destination, as harmful.

Answer 9

Yes.  what does the mycode.wsp contain?   Does it change items on the menu or the master page or anything that would fire or run when a user goes to the default page?   I'm wondering if you had any hard coded urls in the wsp.   i assume not but the could cause the url in the code not finding the resources.  

---

de Nada

Answer 10

The mycode.wsp has quite a few user controls and custom pages. The master page does use some of those user controls. The part that confuses me is that the base site on port 100 uses the same solution (mycode.wsp) as the extended site on port 443, so anything that 443 uses should be accessible for 100 as well.

The further muddy the waters, I extended the base site again, this time to port 111, and that site works just fine, again using the same solution and all my config file changes (AppSettings mostly).

However, I just decided to copy the web.config from site 111 to site 100, and now site 100 works. Clearly there's something in old web.config for site100, so I'm going to go through them with DiffMerge and see which item causes the problem.

Thanks for the assistance with this. Once I know exactly which settings on web.config caused the issue, I'll report back for thread completeness.

Kevin

---

A true software developer is one who, when told to 'Go to Hell', sees the 'go to', rather than the destination, as harmful.

Answer 11

Well, I found the line in the web.config that was causing the problem, though I'm still not exactly sure why.

<globalization resourceProviderFactoryType="Communifire.Providers.LocalizationProviders.DyveResourceProviderFactory, Communifire.Providers" />

The Communifire application is a 3rd party system we use for Social Networking, and my SharePoint site uses some Single Sign On code to allow SP authenticated users to get access to the social area without double login. On the 443 site this item in the web.config seems to work just fine, but on the 100 site, it didn't.

At least now I know. 

Kevin

---

A true software developer is one who, when told to 'Go to Hell', sees the 'go to', rather than the destination, as harmful.