none
AppFabric 1.1 Fails to start

    Question

  • Quick background. This is for a SharePoint 2013 deployment where the Distributed Cache uses AppFabric Caching Service. There are three Windows Server 2008R2 Virtual Servers running on VMWare. Two are dedicated Web Front End Servers the third is a Search/App/Central Admin server. When Distributed Cache is enabled on any of the servers the web site takes at least 6 seconds to load. If I disable Distributed Cache on all servers it takes milliseconds to load. This is a new environment with no data out there yet and only a couple top level sites.

    When Distributed Cache is enabled on any of the servers it will start briefly and crash. In the Application log there are EVENT IDs of 1000 and 1026 being logged as noted here. 

    Log Name:      Application
    Source:        Application Error
    Date:          10/1/2013 1:06:08 PM
    Event ID:      1000
    Task Category: (100)
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:      [server name]
    Description:
    Faulting application name: DistributedCacheService.exe, version: 1.0.4632.0, time stamp: 0x4eafeccf
    Faulting module name: KERNELBASE.dll, version: 6.1.7601.18015, time stamp: 0x50b8479b
    Exception code: 0xe0434352
    Fault offset: 0x0000000000009e5d
    Faulting process id: 0xbd4
    Faulting application start time: 0x01cebed0e2fef851
    Faulting application path: c:\Program Files\AppFabric 1.1 for Windows Server\DistributedCacheService.exe
    Faulting module path: C:\Windows\system32\KERNELBASE.dll
    Report Id: 2518a099-2ac4-11e3-ae94-0050568c62e6
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="Application Error" />
        <EventID Qualifiers="0">1000</EventID>
        <Level>2</Level>
        <Task>100</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2013-10-01T18:06:08.000000000Z" />
        <EventRecordID>103058</EventRecordID>
        <Channel>Application</Channel>
        <Computer>[server name]</Computer>
        <Security />
      </System>
      <EventData>
        <Data>DistributedCacheService.exe</Data>
        <Data>1.0.4632.0</Data>
        <Data>4eafeccf</Data>
        <Data>KERNELBASE.dll</Data>
        <Data>6.1.7601.18015</Data>
        <Data>50b8479b</Data>
        <Data>e0434352</Data>
        <Data>0000000000009e5d</Data>
        <Data>bd4</Data>
        <Data>01cebed0e2fef851</Data>
        <Data>c:\Program Files\AppFabric 1.1 for Windows Server\DistributedCacheService.exe</Data>
        <Data>C:\Windows\system32\KERNELBASE.dll</Data>
        <Data>2518a099-2ac4-11e3-ae94-0050568c62e6</Data>
      </EventData>
    </Event>



    Log Name:      Application
    Source:        .NET Runtime
    Date:          10/1/2013 1:06:06 PM
    Event ID:      1026
    Task Category: None
    Level:         Error
    Keywords:      Classic
    User:          N/A
    Computer:     [server name]
    Description:
    Application: DistributedCacheService.exe
    Framework Version: v4.0.30319
    Description: The process was terminated due to an unhandled exception.
    Exception Info: System.ArgumentException
    Stack:
       at Microsoft.Fabric.Common.IOCompletionPortWorkQueue.Invoke(System.Threading.WaitCallback, System.Object)
       at Microsoft.Fabric.Common.IOCompletionPortWorkQueue.WorkerThreadStart()
       at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
       at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
       at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
       at System.Threading.ThreadHelper.ThreadStart()

    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name=".NET Runtime" />
        <EventID Qualifiers="0">1026</EventID>
        <Level>2</Level>
        <Task>0</Task>
        <Keywords>0x80000000000000</Keywords>
        <TimeCreated SystemTime="2013-10-01T18:06:06.000000000Z" />
        <EventRecordID>103057</EventRecordID>
        <Channel>Application</Channel>
        <Computer>[server name]</Computer>
        <Security />
      </System>
      <EventData>
        <Data>Application: DistributedCacheService.exe
    Framework Version: v4.0.30319
    Description: The process was terminated due to an unhandled exception.
    Exception Info: System.ArgumentException
    Stack:
       at Microsoft.Fabric.Common.IOCompletionPortWorkQueue.Invoke(System.Threading.WaitCallback, System.Object)
       at Microsoft.Fabric.Common.IOCompletionPortWorkQueue.WorkerThreadStart()
       at System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
       at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
       at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)
       at System.Threading.ThreadHelper.ThreadStart()
    </Data>
      </EventData>
    </Event>





    If I drill down to the Microsoft-Windows-Application Server-System Services/Admin Log I see EVENTID 6 and 111

    Log Name:      Microsoft-Windows-Application Server-System Services/Admin
    Source:        Microsoft-Windows Server AppFabric Caching
    Date:          10/1/2013 1:10:54 PM
    Event ID:      111
    Task Category: (1)
    Level:         Error
    Keywords:      
    User:          [Domain\User Account]
    Computer:      [Server Name]
    Description:
    AppFabric Caching service crashed with exception {System.ArgumentException: An entry with the same key already exists.
       at System.Collections.Generic.TreeSet`1.AddIfNotPresent(T item)
       at System.Collections.Generic.SortedDictionary`2.Add(TKey key, TValue value)
       at Microsoft.Fabric.Data.PartitionTable.UpdateEntry(LookupTableEntry newEntry)
       at Microsoft.Fabric.Data.PM.PMPartitionTable..ctor(PartitionManager pm, IList`1 partitions, Int64 savedVersion, Object lockObject)
       at Microsoft.Fabric.Data.PM.PartitionCache..ctor(PartitionManager pm, IPartitionManagerStore pmStore, LoadTable loadTable, IList`1 partitions, Int64 savedLookupVersion)
       at Microsoft.Fabric.Data.PM.PartitionManager.ProcessLoadPM(Object state)
       at Microsoft.Fabric.Common.IOCompletionPortWorkQueue.Invoke(WaitCallback callback, Object state)
       at Microsoft.Fabric.Common.IOCompletionPortWorkQueue.WorkerThreadStart()
       at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
       at System.Threading.ThreadHelper.ThreadStart()}. Check debug log for more information
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="Microsoft-Windows Server AppFabric Caching" Guid="{A77DCF21-545F-4191-B3D0-C396CF2683F2}" />
        <EventID>111</EventID>
        <Version>0</Version>
        <Level>2</Level>
        <Task>1</Task>
        <Opcode>111</Opcode>
        <Keywords>0x8000000000000000</Keywords>
        <TimeCreated SystemTime="2013-10-01T18:10:54.239292100Z" />
        <EventRecordID>13600</EventRecordID>
        <Correlation />
        <Execution ProcessID="4000" ThreadID="4144" />
        <Channel>Microsoft-Windows-Application Server-System Services/Admin</Channel>
        <Computer>[Server Name]</Computer>
        <Security UserID="S-1-5-21-2193036465-2839809817-2807360763-7132" />
      </System>
      <EventData>
        <Data Name="Source">AppFabricCachingService.Crash</Data>
        <Data Name="Param">System.ArgumentException: An entry with the same key already exists.
       at System.Collections.Generic.TreeSet`1.AddIfNotPresent(T item)
       at System.Collections.Generic.SortedDictionary`2.Add(TKey key, TValue value)
       at Microsoft.Fabric.Data.PartitionTable.UpdateEntry(LookupTableEntry newEntry)
       at Microsoft.Fabric.Data.PM.PMPartitionTable..ctor(PartitionManager pm, IList`1 partitions, Int64 savedVersion, Object lockObject)
       at Microsoft.Fabric.Data.PM.PartitionCache..ctor(PartitionManager pm, IPartitionManagerStore pmStore, LoadTable loadTable, IList`1 partitions, Int64 savedLookupVersion)
       at Microsoft.Fabric.Data.PM.PartitionManager.ProcessLoadPM(Object state)
       at Microsoft.Fabric.Common.IOCompletionPortWorkQueue.Invoke(WaitCallback callback, Object state)
       at Microsoft.Fabric.Common.IOCompletionPortWorkQueue.WorkerThreadStart()
       at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
       at System.Threading.ThreadHelper.ThreadStart()</Data>
      </EventData>
    </Event>

    Log Name:      Microsoft-Windows-Application Server-System Services/Admin
    Source:        Microsoft-Windows-Fabric
    Date:          10/1/2013 1:06:06 PM
    Event ID:      6
    Task Category: None
    Level:         Warning
    Keywords:      
    User:          [Domain\User Account]
    Computer:      [Server Name]
    Description:
    {e9d6998000000000000000000000000} failed to refresh lookup table, exception: {Microsoft.Fabric.Common.OperationCompletedException: Operation completed with an exception ---> Microsoft.Fabric.Federation.RoutingException: The target node explicitly aborted the operation
       --- End of inner exception stack trace ---
       at Microsoft.Fabric.Common.OperationContext.End()
       at Microsoft.Fabric.Federation.FederationSite.EndRoutedSendReceive(IAsyncResult ar)
       at Microsoft.Fabric.Data.ReliableServiceManager.EndRefreshLookupTable(IAsyncResult ar)}
    Event Xml:
    <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
      <System>
        <Provider Name="Microsoft-Windows-Fabric" Guid="{751C9DC0-4F51-44F6-920A-A620C7C2D13E}" />
        <EventID>6</EventID>
        <Version>0</Version>
        <Level>3</Level>
        <Task>0</Task>
        <Opcode>0</Opcode>
        <Keywords>0x8000000000000000</Keywords>
        <TimeCreated SystemTime="2013-10-01T18:06:06.981761700Z" />
        <EventRecordID>13597</EventRecordID>
        <Correlation />
        <Execution ProcessID="3028" ThreadID="3492" />
        <Channel>Microsoft-Windows-Application Server-System Services/Admin</Channel>
        <Computer>[Server Name]</Computer>
        <Security UserID="S-1-5-21-2193036465-2839809817-2807360763-7132" />
      </System>
      <EventData>
        <Data Name="param1">e9d6998000000000000000000000000</Data>
        <Data Name="param2">Microsoft.Fabric.Common.OperationCompletedException: Operation completed with an exception ---&gt; Microsoft.Fabric.Federation.RoutingException: The target node explicitly aborted the operation
       --- End of inner exception stack trace ---
       at Microsoft.Fabric.Common.OperationContext.End()
       at Microsoft.Fabric.Federation.FederationSite.EndRoutedSendReceive(IAsyncResult ar)
       at Microsoft.Fabric.Data.ReliableServiceManager.EndRefreshLookupTable(IAsyncResult ar)</Data>
      </EventData>
    </Event>



    Any Ideas? I think the biggest clue is the EventID 111 "An entry with the same key already exists" but searches don't seem to provide any clues about this.


    Tuesday, October 01, 2013 7:36 PM

Answers