locked
AppFabric 1.1 - problem with caching (crashing windows service) RRS feed

  • Question

  • I have a test installation of AppFabric 1.1 x64 (Hosting + Cache) on a single machine (domain-registered, Windows 7 Enterprise). Installation and configuration went flawlessly (cluster with just 1 machine, configuration stored in SQL), I'm running everything locally (including SQL Server 2008 R2). Problem occurs when I want to start "AppFabric Caching Service" - it crashes after few seconds and following entries appear in Administrative Events (in Event Viewer):

    Faulting application name: DistributedCacheService.exe, version: 1.0.4632.0, time stamp: 0x4eafeccf
    Faulting module name: KERNELBASE.dll, version: 6.1.7601.17651, time stamp: 0x4e21213c
    Exception code: 0xe0434352
    Fault offset: 0x000000000000cacd
    Faulting process id: 0x1928
    Faulting application start time: 0x01ccb8c5266c0fd5
    Faulting application path: C:\Program Files\AppFabric 1.1 for Windows Server\DistributedCacheService.exe
    Faulting module path: C:\Windows\system32\KERNELBASE.dll
    Report Id: 6457890a-24b8-11e1-b051-70f1a19c8456

    Application: DistributedCacheService.exe
    Framework Version: v4.0.30319
    Description: The process was terminated due to an unhandled exception.
    Exception Info: System.UriFormatException
    Stack:
       at Microsoft.ApplicationServer.Caching.VelocityWindowsService.StartServiceCallback(System.Object)
       at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)
       at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
       at System.Threading.ThreadPoolWorkQueue.Dispatch()
       at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()

    AppFabric Caching service crashed with exception {System.UriFormatException: Invalid URI: The hostname could not be parsed.
       at System.Uri.CreateThis(String uri, Boolean dontEscape, UriKind uriKind)
       at Microsoft.ApplicationServer.Caching.ServiceConfigurationManager.InitializeThisHostData()
       at Microsoft.ApplicationServer.Caching.ServiceConfigurationManager.InitializeDataFromGlobalConfig()
       at Microsoft.ApplicationServer.Caching.VelocityWindowsService.StartService(Boolean deleteTKT)
       at Microsoft.ApplicationServer.Caching.VelocityWindowsService.StartServiceCallback(Object context)
       at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean ignoreSyncCtx)
       at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem()
       at System.Threading.ThreadPoolWorkQueue.Dispatch()
       at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()}. Check debug log for more information

    Here's the output from basic PowerShell commands:

    Get-CacheHost

    HostName : CachePort Service Name            Service Status Version Info
    -------------------- ------------            -------------- ------------
    MW7GM0B50ROMDQ:22233 AppFabricCachingService DOWN           3 [3,3][1,3]

    Get-CacheHostConfig

    cmdlet Get-CacheHostConfig at command pipeline position 1
    Supply values for the following parameters:
    HostName: MW7GM0B50ROMDQ
    CachePort: 22233

    HostName        : MW7GM0B50ROMDQ
    ClusterPort     : 22234
    CachePort       : 22233
    ArbitrationPort : 22235
    ReplicationPort : 22236
    Size            : 3994 MB
    ServiceName     : AppFabricCachingService
    HighWatermark   : 99%
    LowWatermark    : 90%
    IsLeadHost      : True

    Caching Service is set up to be running as Network Service.

    Do you have any clue what may be wrong? I've seen "Invalid URI: The hostname could not be parsed." message, but name in configuration seems to be perfectly ok (even if it looks odd, it's a proper name of my computer). Any help would be appreciated.

    P.S. I had AppFabric 1.0 before and it seemed to work fine. I've uninstalled it before installing AppFabric 1.1.

    Monday, December 12, 2011 12:39 PM

All replies

  • I've managed to find out what was the reason.

    In my Windows host file I had several entries for loopback IP: localhost, {machine name} and {machine name FQDN}. When I removed all but {machine name FQDN}, configuration wizard was able to set up a proper configuration.

    This was the easy way, but I learned that later...

    My first (and more painful) way to fix the problem was to remove Caching Service configuration manually and re-create it from scratch using PowerShell cmdlets:

    Import-Module ApplicationServer
    Import-Module DistributedCacheAdministration
    Import-Module DistributedCacheConfiguration
    New-CacheCluster ...
    Register-CacheHost ...
    Add-CacheHost ...
    Add-CacheAdmin ...
    etc.
    

    Service starts proplerly, but sadly, when I verify setup with Get-CacheHost, I receive:

    HostName : CachePort                       Service Name            Service Status Version Info
    --------------------                       ------------            -------------- ------------
    MW7GM0B50ROMDQ.<domain part>:22233         AppFabricCachingService UP             0 [0,0][0,0]
    

    If you look at version number, you'd see that it's most likely improper. Unfortunately, when I try to update this info manually (‘Update-CacheHostAllowedVersions’), I get odd looking ‘access-denied’ errors:

    Update-CacheHostAllowedVersions : ErrorCode<ERRCAdmin026>:SubStatus<ES0001>:Remote registry access failed on host MW7GM0B50ROMDQ.<domain part>. Check if the required permissions are available and the host is not down.
    

    I have to make some more tests to find out if it's really a problem or just annoyance.

    Monday, December 12, 2011 8:54 PM
  • Update-CacheHostAllowedVersions : ErrorCode<ERRCAdmin026>:SubStatus<ES0001>:Remote registry access failed on host MW7GM0B50ROMDQ.<domain part>. Check if the required permissions are available and the host is not down.
    

    I have to make some more tests to find out if it's really a problem or just annoyance.

     

    Looks like its hitting some kind of permission/hostname issues while updating the registry, is your client able to connect to the cache server or do you get a version mismatch error ?


    My replies are my personal opinion and cannot be construed to be official Microsoft response. Microsoft will not be responsible for any consequences of my replies.
    Tuesday, December 13, 2011 6:34 AM
  • After short fiddling with hosts file I was able to create a proper (hopefully) cache host:

    HostName : CachePort Service Name            Service Status Version Info
    -------------------- ------------            -------------- ------------
    MW7GM0B50ROMDQ:22233 AppFabricCachingService UP             3 [3,3][1,3]

    However, when I try to connect using trivial client, AppFabric Caching Service crashes with a following, monstrous stacktrace:

    AppFabric Caching service crashed with exception {System.Runtime.CallbackException: Async Callback threw an exception. ---> System.Runtime.Serialization.SerializationException: Invalid enum value 'AdditionalRoutingProps' cannot be deserialized into type 'Microsoft.ApplicationServer.Caching.NamedCacheProperty'. Ensure that the necessary enum values are present and are marked with EnumMemberAttribute attribute if the type has DataContractAttribute attribute.
       at System.Runtime.Serialization.EnumDataContract.ReadEnumValue(String value, Int32 index, Int32 count)
       at System.Runtime.Serialization.EnumDataContract.ReadEnumValue(XmlReaderDelegator reader)
       at System.Runtime.Serialization.EnumDataContract.ReadXmlValue(XmlReaderDelegator xmlReader, XmlObjectSerializerReadContext context)
       at System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(XmlReaderDelegator reader, String name, String ns, Type declaredType, DataContract& dataContract)
       at System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(XmlReaderDelegator xmlReader, Int32 id, RuntimeTypeHandle declaredTypeHandle, String name, String ns)
       at ReadArrayOfNamedCachePropertyFromXml(XmlReaderDelegator , XmlObjectSerializerReadContext , XmlDictionaryString , XmlDictionaryString , CollectionDataContract )
       at System.Runtime.Serialization.CollectionDataContract.ReadXmlValue(XmlReaderDelegator xmlReader, XmlObjectSerializerReadContext context)
       at System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(XmlReaderDelegator reader, String name, String ns, Type declaredType, DataContract& dataContract)
       at System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(XmlReaderDelegator xmlReader, Int32 id, RuntimeTypeHandle declaredTypeHandle, String name, String ns)
       at ReadNamedCachePropertyListFromXml(XmlReaderDelegator , XmlObjectSerializerReadContext , XmlDictionaryString[] , XmlDictionaryString[] )
       at System.Runtime.Serialization.ClassDataContract.ReadXmlValue(XmlReaderDelegator xmlReader, XmlObjectSerializerReadContext context)
       at System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(XmlReaderDelegator reader, String name, String ns, Type declaredType, DataContract& dataContract)
       at System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(XmlReaderDelegator xmlReader, Type declaredType, DataContract dataContract, String name, String ns)
       at System.Runtime.Serialization.DataContractSerializer.InternalReadObject(XmlReaderDelegator xmlReader, Boolean verifyObjectName, DataContractResolver dataContractResolver)
       at System.Runtime.Serialization.XmlObjectSerializer.ReadObjectHandleExceptions(XmlReaderDelegator reader, Boolean verifyObjectName, DataContractResolver dataContractResolver)
       at System.Runtime.Serialization.XmlObjectSerializer.ReadObject(XmlDictionaryReader reader)
       at Microsoft.ApplicationServer.Caching.Utility.Deserialize(Byte[][] buffers, Boolean checkTypeToLoad, Object context, IEnumerable`1 knownTypes)
       at Microsoft.ApplicationServer.Caching.VelocityDataStore.GetNamedCacheProperties(RequestBody request)
       at Microsoft.ApplicationServer.Caching.VelocityDataStore.ProcessMessageRequest(RequestBody request)
       at Microsoft.ApplicationServer.Caching.DistributedObjectManager.OnReceiveMessageFromLocalDRM(RequestBody request, Object session)
       at Microsoft.ApplicationServer.Caching.ServerDRM.ProcessRequest(RequestBody request, Object session)
       at Microsoft.ApplicationServer.Caching.ServiceLayer.ProcessClientMsg(RequestBody req, Object session)
       at Microsoft.ApplicationServer.Caching.ServiceLayer.RcvCallback(IReplyContext replyContext)
       at Microsoft.ApplicationServer.Caching.WcfTransportChannel.InvokeCallback(Message message, IChannelContainer container)
       at Microsoft.ApplicationServer.Caching.WcfServerChannel.InvokeCallback(Message message, IChannelContainer container)
       at Microsoft.ApplicationServer.Caching.WcfServerChannel.CompleteProcessing(IAsyncResult result)
       at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
       --- End of inner exception stack trace ---
       at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
       at System.ServiceModel.Channels.SynchronizedMessageSource.ReceiveAsyncResult.OnReceiveComplete(Object state)
       at System.ServiceModel.Channels.SessionConnectionReader.OnAsyncReadComplete(Object state)
       at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)
       at System.Net.LazyAsyncResult.Complete(IntPtr userToken)
       at System.Net.Security.NegotiateStream.ProcessFrameBody(Int32 readBytes, Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)
       at System.Net.Security.NegotiateStream.ReadCallback(AsyncProtocolRequest asyncRequest)
       at System.Net.FixedSizeReader.CheckCompletionBeforeNextRead(Int32 bytes)
       at System.Net.FixedSizeReader.ReadCallback(IAsyncResult transportResult)
       at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)
       at System.ServiceModel.Channels.ConnectionStream.ReadAsyncResult.OnAsyncReadComplete(Object state)
       at System.ServiceModel.Channels.SocketConnection.AsyncReadCallback(Boolean haveResult, Int32 error, Int32 bytesRead)
       at System.Runtime.Fx.IOCompletionThunk.UnhandledExceptionFrame(UInt32 error, UInt32 bytesRead, NativeOverlapped* nativeOverlapped)
       at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)}. Check debug log for more information

    To be very honest, message "Invalid enum value 'AdditionalRoutingProps' cannot be deserialized into type 'Microsoft.ApplicationServer.Caching.NamedCacheProperty" makes it look like the issue is more related to AppFabric internals than my setup.

    Here's the client code:

                var servers = new List<DataCacheServerEndpoint>(1)
                              {
                                  new DataCacheServerEndpoint("MW7GM0B50ROMDQ",
                                                              22233)
                              };
     
                var configuration = new DataCacheFactoryConfiguration
                                    {
                                        Servers = servers,
                                        LocalCacheProperties = new DataCacheLocalCacheProperties()
                                    };
     
                DataCacheClientLogManager.ChangeLogLevel(System.Diagnostics.TraceLevel.Off);
     
                var myCacheFactory = new DataCacheFactory(configuration);
     
                var myCache = myCacheFactory.GetCache("TestCache");

    Obviously, "TestCache" was created with New-Cache earlier. When I check overall cache cluster health (Get-CacheClusterHealth), it doesn't show anything wrong:

    Cluster health statistics
    =========================

    HostName = MW7GM0B50ROMDQ.<my domain>
    ------------------------- 

        NamedCache = default
            Healthy               = 50.00
            UnderReconfiguration  = 0.00
            NotPrimary            = 0.00
            InadequateSecondaries = 0.00
            Throttled             = 0.00

        NamedCache = TestCache
            Healthy               = 50.00
            UnderReconfiguration  = 0.00
            NotPrimary            = 0.00
            InadequateSecondaries = 0.00
            Throttled             = 0.00

    Unallocated named cache fractions
    ---------------------------------

    Tuesday, December 13, 2011 2:27 PM
  • Are you using older version of caching server with newer clients? If yes, can you also upgrade your server to latest V1.1 and then use latest client SDK instead.
    • Edited by vvdeep Wednesday, December 14, 2011 2:38 AM
    Wednesday, December 14, 2011 2:36 AM
  • Are you using older version of caching server with newer clients? If yes, can you also upgrade your server to latest V1.1 and then use latest client SDK instead.

    All the tests are performed within one, single machine. It is a full Windows Server AppFabric installation - both server and client features. As far as I know there's no dedicated SDK as a separate download.

    *** Edit ***

    Actually, here's what I found out after some more digging.

    AppFabric's DLLs in GAC are still the ones installed in 1.0 distribution. One can find 1.1 DLLs only in AppFabric's directory in Program Files (or wherever you've chosen to install it). Anyway, vvdeep was partially right - I was (not intentionally) trying to access 1.1 server with 1.0 client. After updating the references in the project, everything works well.

    Thanks for all the suggestions.

    • Edited by Sebastian Gebski Thursday, December 15, 2011 7:07 AM
    • Proposed as answer by vvdeep Thursday, December 15, 2011 3:31 PM
    Wednesday, December 14, 2011 1:14 PM
  • After removing <IP address> <Machine name> from host file, I could start service. Thanks a lot for the help.

    Thanks & Regards, Preetham

    Thursday, June 21, 2012 8:39 AM
  • Perfect, solved my problem.   its the FDNQ in hosts file
    Wednesday, June 26, 2013 1:08 AM
  • Are you talking about the Windows host file?
    Wednesday, October 9, 2013 1:57 PM
  • I am getting the below error and struggling for the past 2 weeks. When we did the correlation we can see that Server SQL server is running out of memory when this error occurs. When we increased the SQL memory frequency is reduced to 2 per day than 8 times per day . We have to Applications sharing the same cluster and when we remove one application frequency further comes down to 1 per day. Even though server crashes it restarts however during this time users are impacted .

    AppFabric Caching service crashed with exception {System.Runtime.CallbackException: Async Callback threw an exception. ---> System.Runtime.Serialization.SerializationException: Invalid enum value 'AdditionalRoutingProps' cannot be deserialized into type 'Microsoft.ApplicationServer.Caching.NamedCacheProperty'. Ensure that the necessary enum values are present and are marked with EnumMemberAttribute attribute if the type has DataContractAttribute attribute.

       at System.Runtime.Serialization.EnumDataContract.ReadEnumValue(String value, Int32 index, Int32 count)

       at System.Runtime.Serialization.EnumDataContract.ReadEnumValue(XmlReaderDelegator reader)

       at System.Runtime.Serialization.EnumDataContract.ReadXmlValue(XmlReaderDelegator xmlReader, XmlObjectSerializerReadContext context)

       at System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(XmlReaderDelegator reader, String name, String ns, Type declaredType, DataContract& dataContract)

       at System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(XmlReaderDelegator xmlReader, Int32 id, RuntimeTypeHandle declaredTypeHandle, String name, String ns)

       at ReadArrayOfNamedCachePropertyFromXml(XmlReaderDelegator , XmlObjectSerializerReadContext , XmlDictionaryString , XmlDictionaryString , CollectionDataContract )

       at System.Runtime.Serialization.CollectionDataContract.ReadXmlValue(XmlReaderDelegator xmlReader, XmlObjectSerializerReadContext context)

       at System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(XmlReaderDelegator reader, String name, String ns, Type declaredType, DataContract& dataContract)

       at System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(XmlReaderDelegator xmlReader, Int32 id, RuntimeTypeHandle declaredTypeHandle, String name, String ns)

       at ReadNamedCachePropertyListFromXml(XmlReaderDelegator , XmlObjectSerializerReadContext , XmlDictionaryString[] , XmlDictionaryString[] )

       at System.Runtime.Serialization.ClassDataContract.ReadXmlValue(XmlReaderDelegator xmlReader, XmlObjectSerializerReadContext context)

       at System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(XmlReaderDelegator reader, String name, String ns, Type declaredType, DataContract& dataContract)

       at System.Runtime.Serialization.XmlObjectSerializerReadContext.InternalDeserialize(XmlReaderDelegator xmlReader, Type declaredType, DataContract dataContract, String name, String ns)

       at System.Runtime.Serialization.DataContractSerializer.InternalReadObject(XmlReaderDelegator xmlReader, Boolean verifyObjectName, DataContractResolver dataContractResolver)

       at System.Runtime.Serialization.XmlObjectSerializer.ReadObjectHandleExceptions(XmlReaderDelegator reader, Boolean verifyObjectName, DataContractResolver dataContractResolver)

       at System.Runtime.Serialization.XmlObjectSerializer.ReadObject(XmlDictionaryReader reader)

       at Microsoft.ApplicationServer.Caching.Utility.Deserialize(Byte[][] buffers, Boolean checkTypeToLoad, Object context, IEnumerable`1 knownTypes)

       at Microsoft.ApplicationServer.Caching.VelocityDataStore.GetNamedCacheProperties(RequestBody request)

       at Microsoft.ApplicationServer.Caching.VelocityDataStore.ProcessMessageRequest(RequestBody request)

       at Microsoft.ApplicationServer.Caching.DistributedObjectManager.OnReceiveMessageFromLocalDRM(RequestBody request, Object session)

       at Microsoft.ApplicationServer.Caching.ServerDRM.ProcessRequest(RequestBody request, Object session)

       at Microsoft.ApplicationServer.Caching.ServiceLayer.ProcessClientMsg(RequestBody req, Object session)

       at Microsoft.ApplicationServer.Caching.ServiceLayer.RcvCallback(IReplyContext replyContext)

       at Microsoft.ApplicationServer.Caching.WcfTransportChannel.InvokeCallback(Message message, IChannelContainer container)

       at Microsoft.ApplicationServer.Caching.WcfServerChannel.InvokeCallback(Message message, IChannelContainer container)

       at Microsoft.ApplicationServer.Caching.WcfServerChannel.CompleteProcessing(IAsyncResult result)

       at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)

       --- End of inner exception stack trace ---

       at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)

       at System.ServiceModel.Channels.SynchronizedMessageSource.ReceiveAsyncResult.OnReceiveComplete(Object state)

       at System.ServiceModel.Channels.SessionConnectionReader.OnAsyncReadComplete(Object state)

       at System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(IAsyncResult result)

       at System.Net.LazyAsyncResult.Complete(IntPtr userToken)

       at System.Net.Security.NegotiateStream.ProcessFrameBody(Int32 readBytes, Byte[] buffer, Int32 offset, Int32 count, AsyncProtocolRequest asyncRequest)

       at System.Net.Security.NegotiateStream.ReadCallback(AsyncProtocolRequest asyncRequest)

       at System.Net.FixedSizeReader.CheckCompletionBeforeNextRead(Int32 bytes)

       at System.Net.FixedSizeReader.ReadCallback(IAsyncResult transportResult)

       at System.Runtime.AsyncResult.Complete(Boolean completedSynchronously)

       at System.ServiceModel.Channels.ConnectionStream.IOAsyncResult.OnAsyncIOComplete(Object state)

       at System.ServiceModel.Channels.SocketConnection.OnReceiveAsync(Object sender, SocketAsyncEventArgs eventArgs)

       at System.Net.Sockets.SocketAsyncEventArgs.FinishOperationSuccess(SocketError socketError, Int32 bytesTransferred, SocketFlags flags)

       at System.Net.Sockets.SocketAsyncEventArgs.CompletionPortCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* nativeOverlapped)

       at System.Threading._IOCompletionCallback.PerformIOCompletionCallback(UInt32 errorCode, UInt32 numBytes, NativeOverlapped* pOVERLAP)}. Check debug log for more information

    Sunday, November 15, 2015 4:45 PM