User Profile Service unprovisions itself

Question

We have had the UPSS running successfully, and then it started stopping every day. Looking at it closer, it is actually unprovisioning itself, stopping, then disabling both FIM services. We can restart UPSS without issue (given we add the farm account to the local admins group), it will just stop roughly 24 hours later. My understanding is that the usual response will be to create a new User Profile Service service app with new databases and tie UPSS to the new app. If we do create a new UPS service app, what information will I loose by moving to new Sync/Profile/User databases? SP 2010 Ent, Feb 2011 CU.

 

Looking at the ULS logs, I see nothing more than a call to the timer job "ProfileSynchronizationUnprovisionJob" which then goes about unprovisioning service instance User Profile Synchornization Service, and at that point the service is stopped.

---

http://sharepoint.nauplius.net

Answer 1

A little more information, I restarted the service and ran a full import.  During the import process on one of our domains, this is helpful, by the way (ugh):

UserProfileApplication.SynchronizeMIIS: Unexpected Exception: System.Management.ManagementException: Generic failure

And the FIM services stop, unprovision, etc.

---

http://sharepoint.nauplius.net

Answer 2

Hi Trevor,

 

As I understand, you noticed that the user profile unprovisioned itself automatically.

 

Firstly, ensure that the account you've used to configure the User Profile Synchronization Service has the following permissions when you didn't install it on an AD controller:

 

- Replicate Directory Changes on your AD

- Replicate Directory Changes permissions on your configuration partition (AD)

- Is local admin on the machine where you want to configure User Profile Service( UPS) service

 

Secondly, stop all instances of the User Profile Service (UPS) and User Profile Sync Service (UPSS). If the UPSS is stuck in starting state, look for a One-Time job named User Profile Service provisioning (unprovisioning) job. My general recommendation is to have only one instance of the UPS running. You should also have only one instance of UPSS running in your farm.

 

Finally, open Central administration, click Monitoring, select configure diagnostic logging, expand SharePoint server category, select User Profile, then set the threshold of event and trace log to verbose.

 

Thanks,

Rock Wang

---

Regards, Rock Wang Microsoft Online Community Support

Answer 3

Hi Rock, none of those appear to be the issue. It looks like it is the SharePoint backup we're performing.  We are backing up via the Backup-SPFarm directory, using a Full backup on a daily basis.

When the backup initiates a backup of the Profile database, the FIM service shuts down.  Our backup window does not correspond with an Active Directory sync.

---

http://sharepoint.nauplius.net

Answer 4

It looks like the FIM services may start back up if the Farm Admin account is left as a local administrator (I haven't 100% validated this).  At the very least, the UPSS can be restarted once the backup is done if the Farm Admin is a local administrator, but this is not desirable nor best practice.

Any further suggestions?

---

http://sharepoint.nauplius.net

Answer 5

I also have this problem, on three seperate farms. Everything is fine for a few days but stops after a while, potentially when a backup is perfomed now you mention it. Leaving the Farm user account in the local administrators group will allow it to start again but everything says don't leave it in there...

Any ideas?

EDIT: Just confirmed that running a backup (Full or specifically of the UPS parts) unprovisions and then provisions the UPS services. If the farm account is a local administrator at the time, it works fine, but this isn't the case for normal operation so it stops and cannot start.

Answer 6

Thanks, I'm glad someone else has seen this. I do have a case open with PSS, but wanted to see if anyone else could replicate it or had a solution.

---

http://sharepoint.nauplius.net

Answer 7

Sounds good, I have it replicated in a development/test environment too so let me know if you want me to try anything.

Answer 8

Any luck with this?

Answer 9

No, still working with PSS.

---

http://sharepoint.nauplius.net

Answer 10

No, still working with PSS.

---

http://sharepoint.nauplius.net

Answer 11

Did PSS make any progress?  I've seen a couple of farms with this issue as well.

Answer 12

No, my case has been escalated, but I'm off work until tomorrow :)

---

http://sharepoint.nauplius.net

Answer 13

Any update on this bug? Verified it on our test farm as well.

---

Jason Bradley

Answer 14

Nothing from my engineer (and he is aware of this thread, so hopefully when we discover what is going on, he can provide the answer directly), however I do have a call today.

Over the weekend I did some checks with .NET Reflector, and the unprovisioning appears to be by design during the backup (to prevent any attempts at synchronization between FIM and Active Directory).

If you use Process Monitor, it *appears* (again, not validated by my engineer) that the UPSS is checking for group membership:

   ULS.SendTraceTag(0x39693230, ULSCat.msoulscat_SPS_UserProfiles, ULSTraceLevel.Medium, "ILM Configuration: Validating the system groups...");
    ValidateConfigurationResult(this.validateGroupsEx(this.mmsEdition, "WSS_ADMIN_WPG", "FIMSyncOperators", "FIMSyncJoiners", "FIMSyncBrowse", "FIMSyncPasswordSet"));

For some reason, this check is failing (the check uses some unmanaged code, so how it actually works, I can't tell).  But if you compare this portion of the check between a the Farm Admin account having versus not having Local Administrator access with Process Monitor, you can tell that lsass is doing something different.

If the Farm Admin account has Local Administrator rights, the check succeeds (even if the Farm Admin account is not a member of all of the above groups, which by default, it isn't).  If the Farm Admin account does not have those rights, something else happens and FIM reports ERR_INVALID_GROUPS, even if you've added the Farm Admin account to all of the above groups.

---

http://sharepoint.nauplius.net

Answer 15

Hi, how are you getting on with this?

Answer 16

Appiled SP1 and the June 2011 CU which includes a UPS re-write (as well as final bits for FIM) with no change.  I updated my case with Microsoft so my guess is there will probably be a documentation change to indicate that local admin is required if using the SharePoint internal backup solution.

---

http://sharepoint.nauplius.net

Answer 17

Try this solution: http://todd-carter.com/post/2011/07/01/The-User-Profile-Service-Heats-up-With-a-June-CU-Regression.aspx

 

---

-Todd

Answer 18

Try this one Trevor -- http://todd-carter.com/post/2011/07/01/The-User-Profile-Service-Heats-up-With-a-June-CU-Regression.aspx

---

-Todd

Answer 19

Todd, I'm not sure why you marked your post as an answer.  This is an on going issue with backup and has no relation to your link.  There is going to be a code and/or TechNet article change.

To expand on this, the Farm Admin account is attempting to open a handle to the local SAM.  Only Administrators can do this, so the way the unmanaged code is functioning will not allow a non-administrator to validate the local group membership.

This can be repro'ed with RTM through June 2011 CU and .NET 4.0 isn't a factor.

---

http://sharepoint.nauplius.net

Answer 20

Hi guys,

I'm having the very same issue over here. I just did a clean SP 2010 Server deployment and applied both SP1 and July 2011 CU. This customer has chosen to utilize OOTB SharePoint backups, which I have scheduled to run daily. The UPS gets unprovisioned prior to the back up, but then it fails to be reprovisioned as we didn't leave the Farm Service account in the administrators group. The only workaround that I found so far was to leave the Farm Service in the local administrators, which goes against the suggested practice. Any updates from Microsoft on a real fix for this? Thanks.

Regards,
Max

Answer 21

No updates, but I do have a new case open for this (the original case I had spiraled into something else...).  I'll make sure to keep this thread updated on the outcome.

 

---

http://sharepoint.nauplius.net

Answer 22

Update: In my new case, tier 1 support was able to repro the issue as expected.  They are engaging additional resources at this time.

---

http://sharepoint.nauplius.net

Answer 23

Update: This might be my last update to this.  I'm in the process of filing a Business Impact Statement to see if we can get this fixed, but that doesn't mean the group responsible for the code will accept the BIS and implement a fix or acceptable workaround.

---

http://sharepoint.nauplius.net

Answer 24

Update: The bug has been filed and the product team is in the process of determining if it can be fixed within the scope of a hotfix.

---

http://sharepoint.nauplius.net

Answer 25

I'm having a similar issue where my UPS service is in that intermediary state that requires an IISReset before it starts working. Basically at two different points in the day the UPS service unprovisions itself and re-provisions itself. The reprovision works but because a provision of the UPS requires an IISReset to take effect (if on the same box as central admin), it doesn't "fix" itself as it should.

 

I believe our backup agent (we use CommVault) is what's triggering the unprovision. The ULS logs indicate that the CommVault service is starting at the same time the unprovision kicks off. Is this a SharePoint issue or a CommVault issue? If CommVault is only doing what the SharePoint documentation suggests (unprovision before backing up) then I believe it to be a SharePoint issue.

Answer 26

It is likely still a SharePoint issue.  If CommVault is calling a native method in SharePoint that performs the same function as the SharePoint-integrated backup, it will have the same effect: Unprovision/reprovision UPA, and if UPA is on a Central Admin server, require an iisreset prior to functioning.  I believe both of these issues have been accepted by the product team(s), it is just a matter of deciding when to create and publish a fix.

---

http://sharepoint.nauplius.net

Answer 27

Another side effect I noticed is that the User Profile Service requires a Full Profile Synchronization after a re-provision. As a result the event log shows several warnings (6127, 6126):

The management agent "MOSS-faec92f3-d1f2-4973-86a3-fc81ef5955de" completed run profile "MOSS_DELTASYNC_d822df85-330a-44f8-b383-26bcf8c1c3d4" with a delta import or delta synchronization step type. The rules configuration has changed since the last full synchronization.

User Action

To ensure the updated rules are applied to all objects, a run with step type of full synchronization should be completed

Answer 28

Please add the UserProfile service Account to the farm admin group and check the behavior. If you create a new UPA youwould loose your profiles and audiences.

 

Answer 29

Hi,

This behavior is normal if you use SharePoint farm backup. During the backup, SharePoint trys to pause all service applications in aim to be able to backup everything. UPS is the only service that can't really be paused, so SharePoint unprovision the service and the reprovision it after the backup operation is done. For that to success, you need to have farm admin as a local administartor. I have seen many papers telling you that you need to remove it from local administrators group when you are done, but I think it should be there, especially in this case.

  

---

Regards
Henrik A. Halmstrand
sharepointrevealed.com
getspconfig.codeplex.com
Please click Mark As Answer; if a post solves your problem or Vote As Helpful; if a post has been useful to you.

Answer 30

While this is currently normal behavior, the TechNet documentation is incorrect with regards to removing the Farm Admin account from the Local Administrators group once provisioned.  Also, since a DCR has been filed, hopefully the behavior will be changed.

---

http://sharepoint.nauplius.net

Answer 31

Hi,

This behavior is normal if you use SharePoint farm backup. During the backup, SharePoint trys to pause all service applications in aim to be able to backup everything. UPS is the only service that can't really be paused, so SharePoint unprovision the service and the reprovision it after the backup operation is done. For that to success, you need to have farm admin as a local administartor. I have seen many papers telling you that you need to remove it from local administrators group when you are done, but I think it should be there, especially in this case.

  

---

Regards
Henrik A. Halmstrand
sharepointrevealed.com
getspconfig.codeplex.com
Please click Mark As Answer; if a post solves your problem or Vote As Helpful; if a post has been useful to you.

Despite this being normal, an unprovision/reprovision requires an IISReset before the User Profile Service interface in central admin is usable. I've verified that even though the interface is not functioning after a reprovision, the service itself still runs incremental synchronizations as expected on the user profiles. Being able to see how many profiles are in the store is only available after an IISReset post reprovision.

So regardless, it's broken.

Answer 32

Unprovisioning the User Profile Application is 'by design' per Microsoft.  This is not something that will be changed.  Online documentation will be updated to reflect that the Farm Administrator must also be a local administrator.

---

http://sharepoint.nauplius.net

Answer 33

When? I can't find anything to support your statement that it will be changed...

---

Kevin W. Gagel

Answer 34

When? I can't find anything to support your statement that it will be changed...

---

Kevin W. Gagel

It won't be changed, per my post.  It is by design and will not be changed.  Only online documentation will eventually be updated.

---

http://sharepoint.nauplius.net

Answer 35

I guess I'm asking how do you know that?

---

Kevin W. Gagel

Answer 36

I had opened a PSS case and submitted a DCR (design change request) which was declined by the Product Team responsible for FIM/UPS.  They said the design won't change, but they will at some point update the product documentation to reflect the required additional permissions for the Farm Admin account during backup when backing up the UPS.

---

http://sharepoint.nauplius.net

Answer 37

Hi Trevor,

I was working 2 weeks with MS support due to an UPS issue related to the fact that the spbackup-farm was running before it and the unprovision caused the known issue.

For me the solution is to stop using the backup farm and use the SQL backup for content DDBBs, config and service application´s DDBBs.

It´s a pity I didn´t see your post before. Thanks