none
Online Backup fails with error: The current operation failed due to an internal service error [0x81BC]. Please retry the operation after sometime. (ID 100066) RRS feed

  • Question

  • I'm currently testing Windows Azure Online Backup with DPM 2012 SP1 and I keep running into this error when running online backups.

    The current operation failed due to an internal service error [0x81BC]. Please retry the operation after sometime. (ID 100066).

    I've had a look at the CBEngineCurr.errlog file but don't know what I am looking at.

    Can anybody shed any light on why these jobs consistently fail?

    Tuesday, January 1, 2013 8:34 PM

Answers

All replies

  • Hi Adam,

    it would help to get a snippet of the error log above and below with text "[ErrorCode:" at the time when the backup failed. we are looking for request Id made by your client to our service so that we can correlate. an sample trace might look similar to this.

    0954 0F78 01/01 05:14:54.818 69 CBEngineWcfClientHelper.cs(298)  CF07D239-CB57-4661-A88E-89A055953001 NORMAL Setting RequestId header for outgoing request | Params: {RequestId = 93007e06-79bf-44dd-b886-e45f16b78586}{WorkitemId = 2f5ac141-e81d-43a4-beea-5be713d50f96}{TaskId = cf07d239-cb57-4661-a88e-89a055953001}

    Thanks

    Prashanth

    Thursday, January 3, 2013 6:27 PM
  • Prashanth,

    Here you go... Hopefully it shows you what you need:

    11EC 2428 01/03 07:05:49.558 69 CBEngineWcfClientHelper.cs(298) FF2690BC-0587-4B4F-A8B7-6E08E5141E6B NORMAL Setting RequestId header for outgoing request | Params: {RequestId = a1fa8fd4-ab3c-45f0-9c7f-a1b9ca8bf2d9}{WorkitemId = 95f92a7f-6f85-4960-80dd-b2a3e41e6482}{TaskId = ff2690bc-0587-4b4f-a8b7-6e08e5141e6b}
    11EC 2428 01/03 07:05:49.710 79 WcfClient.cs(892) FF2690BC-0587-4B4F-A8B7-6E08E5141E6B NORMAL Finished web service call | Params: {ServiceInterface = Microsoft.Internal.CloudBackup.Common.Protection.Interface.IProtectionServiceChannel}{Description = GetStatus}{TargetEndpoint = https://pod1-prot1.ne.onlinebackup.microsoft.com/ProtectionService.svc}
    11EC 2428 01/03 07:05:49.710 79 WcfClient.cs(739) FF2690BC-0587-4B4F-A8B7-6E08E5141E6B NORMAL FMBlock: Retry succeeded! | Params: {OperationCode = WcfProxyGenericAsyncOperationPollingCall}{CurrentAttempt = 3}
    11EC 2428 01/03 07:05:49.710 79 WcfClient.cs(871) FF2690BC-0587-4B4F-A8B7-6E08E5141E6B NORMAL Executing web service call | Params: {ServiceInterface = Microsoft.Internal.CloudBackup.Common.Protection.Interface.IProtectionServiceChannel}{Description = PerformIntegrityCheck}{TargetEndpoint = https://pod1-prot1.ne.onlinebackup.microsoft.com/ProtectionService.svc}
    11EC 2428 01/03 07:05:49.710 79 SWTTokenValidator.cs(56) FF2690BC-0587-4B4F-A8B7-6E08E5141E6B WARNING Cached Token Expired | Params: {ErrorCode = IdTokenExpiredError}{ExpiryTimeWindow (ulong) = 7200}
    11EC 2428 01/03 07:05:49.711 69 MachineIdentity.cs(163) FF2690BC-0587-4B4F-A8B7-6E08E5141E6B NORMAL ACS SWT token expired, requesting new token 
    11EC 2428 01/03 07:05:49.811 69 MachineIdentity.cs(349) FF2690BC-0587-4B4F-A8B7-6E08E5141E6B NORMAL Fetched ACS Token (from ACS service) for Machine | Params: {MachineName = backup1.riddlesdown.local}{Thumbprint = B5D59E9E02BB5750F9E6E854828F0257B059E0CF}
    11EC 2428 01/03 07:05:49.812 69 CBEngineWcfClientHelper.cs(298) FF2690BC-0587-4B4F-A8B7-6E08E5141E6B NORMAL Setting RequestId header for outgoing request | Params: {RequestId = 547e397e-f017-4d4f-9bf2-e9f8fe4660ae}{WorkitemId = f4d2205d-934f-40bb-bb0d-68d7037db5f9}{TaskId = ff2690bc-0587-4b4f-a8b7-6e08e5141e6b}
    11EC 2428 01/03 07:05:49.917 79 WcfClient.cs(892) FF2690BC-0587-4B4F-A8B7-6E08E5141E6B NORMAL Finished web service call | Params: {ServiceInterface = Microsoft.Internal.CloudBackup.Common.Protection.Interface.IProtectionServiceChannel}{Description = PerformIntegrityCheck}{TargetEndpoint = https://pod1-prot1.ne.onlinebackup.microsoft.com/ProtectionService.svc}
    11EC 2428 01/03 07:05:49.918 32 salcataloghelper.cpp(1217) FF2690BC-0587-4B4F-A8B7-6E08E5141E6B NORMAL PerformIntegrityCheck API with replicaId = {737FE9B0-90B9-453D-A459-7D9935845008}, replica snapshot id = {979D6E38-4E8C-4BFC-BDA9-1A3CCCFF3B0E}, IC level = 2 succeeded
    11EC 2428 01/03 07:05:49.918 32 fileprovider.cpp(1254) [00000000203E8C90] FF2690BC-0587-4B4F-A8B7-6E08E5141E6B NORMAL <--FileProvider::EndData
    11EC 2428 01/03 07:05:49.918 70 acceptdatasetsubtask.cpp(1291) [000000001A1AF7E0] FF2690BC-0587-4B4F-A8B7-6E08E5141E6B NORMAL CAcceptDatasetSubTask::SubTaskDone - Fixup
    11EC 1AC8 01/03 07:05:49.918 18 dsmreceiversubtaskbase.cpp(320) [000000001A1A8FD0] FF2690BC-0587-4B4F-A8B7-6E08E5141E6B NORMAL CDsmReceiverSubTaskBase received completion for session closed in CLEANUP state
    11EC 0BBC 01/03 07:05:49.918 70 onlinesubtask.cpp(496) [000000001A1AF7E0] FF2690BC-0587-4B4F-A8B7-6E08E5141E6B WARNING Failed: Hr: = [0x00000000] COnlineSubTask::UpdateStatus => hr
    11EC 1AC8 01/03 07:05:49.918 70 acceptdatasetsubtask.cpp(1413) [000000001A1AF7E0] FF2690BC-0587-4B4F-A8B7-6E08E5141E6B NORMAL CAcceptDatasetSubTask::CleanUp [000000001A1AF7E0]
    11EC 0BBC 01/03 07:05:49.919 70 onlinesubtask.cpp(436) [000000001A1AF7E0] FF2690BC-0587-4B4F-A8B7-6E08E5141E6B ACTIVITY COnlineSubTask::DeactivateSubTask => Deactivating SubTask

    Cheers

    Adam.


    Thursday, January 3, 2013 7:00 PM
  • Hi Adam,

    this set of trace is actually showing that the backup has succeeded, can you trigger a backup, wait for it to fail and send the errorlogs to waobsupport@microsoft.com

    Thanks

    Prashanth

    Friday, January 4, 2013 4:57 AM
  • I don't know if this is the fix, but since installing the rollup here http://support.microsoft.com/kb/2779569 I haven't experience this error.. 
    • Marked as answer by adamf83 Sunday, January 13, 2013 8:20 PM
    • Edited by adamf83 Sunday, January 13, 2013 8:20 PM
    Sunday, January 13, 2013 8:20 PM
  • Hi Adam,

    I had sent a reply with these investigation details on Jan7, please do check your proxy and system clock,

    Thanks

    Prashanth

    Can you check whether your proxy or isp is causing some trouble. I am seeing the following errors and we have seen other customers to hit it and the reason being ISP was throttling them. You can try to set bandwidth throttling and see whether the call succeeds.

    Also I am seeing the timestamp in the log as 11/02, is the system clock proper and in sync with current time.

    System.ServiceModel.EndpointNotFoundException: There was no endpoint listening at xxxxx that could accept the message. This is often caused by an incorrect address or SOAP action. See InnerException, if present, for more details. ---> System.Net.WebException: The remote name could not be resolved: 'xxxx'

    • Proposed as answer by Prashanth C Tuesday, January 29, 2013 2:45 AM
    Wednesday, January 16, 2013 2:27 AM