Unable to make a cache cluster with two hosts

Question

Hello,

I'm trying to create a cache cluster with two machines using AppFabric 1.1.

I have one machine which created the cluster during its installation (using a XML conf. file) and another machine joining the cluster using the same XML configuration file. I used the default ports and "NETWORK SERVICES" as the cache account service (I don't know if that play in starting the cluster). Both machines are on the same domain.

I made sure that the Remote Registry is started on all machines and both machines have a rule allowing the ports for input/output traffic. I changed the ClusterConfig.xml to adjust the size and the host's name for fitting my network configuration.

I have an application using the cache cluster to do tests that runs on the machine where the cluster was created initially.

When I configure a cluster with the first machine only, the cluster starts and my test application can use the cache cluster.

When I configure a cluster with the second machine only, the cluster starts and my test application can use the cache cluster.

When I configure a cluster with the two machines, only the first starts, the second return a time out and its status is stuck at "STARTING". The error returned is ErrorCode<EERCAdmin025>:SubStatus<ES0001>.

Do you have any ideas about what I'm doing wrong in here ?

Thanks and best regards,

Eric GASPARD

Answer 1

Hi,

Can you confirm how you are starting the cluster? Are you using Start-CacheCluster?

Could you also export the cache cluster config using Export-CacheClusterConfig and put it here?

Thanks

---

Charu P [Microsoft]

Answer 2

Hello Charu,

Yes I'm starting the cluster using this command. Additionnally I tried to manually Start-CacheHost the deficient one but to no avail.

I'm not storing the configuration in a DataBase but through a shared XML file so here it is :

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <configSections>
    <section name="dataCache" type="Microsoft.ApplicationServer.Caching.DataCacheSection, Microsoft.ApplicationServer.Caching.Core" />
  </configSections>
  <dataCache size="Small">
    <caches>
      <cache consistency="StrongConsistency" name="default" minSecondaries="0">
        <policy>
          <eviction type="Lru" />
          <expiration defaultTTL="10" isExpirable="true" />
        </policy>
      </cache>
    </caches>
    <hosts>
      <host replicationPort="22236" arbitrationPort="22235" clusterPort="22234"
        hostId="150489490" size="512" leadHost="true" account="BI\DEVW7-WKS44$"
        cacheHostName="AppFabricCachingService" name="DEVW7-WKS44"
        cachePort="22233" />
      <host replicationPort="22236" arbitrationPort="22235" clusterPort="22234"
        hostId="1437885474" size="128" leadHost="true" account="BI\DEVW7-WKS24$"
        cacheHostName="AppFabricCachingService" name="DEVW7-WKS24"
        cachePort="22233" />
    </hosts>
    <deploymentSettings>
      <deploymentMode value="RoutingClient" />
    </deploymentSettings>
  </dataCache>
</configuration>

DEVW7-WKS44 is the initial machine that had the cluster created when installing, DEVW7-WKS24 is the secondary host and the one that do not start properly. Also I tried to put WKS24 with leadHost to false but that did not work either.

Thanks and best regards.

PS : I forgot to mention that both machines OS' are Windows 7

Answer 3

Okay I've something new:

I tried to install AppFabric on a Server 2008 R2 joining my existing Cache Cluster and it worked ! That means it seems my second W7 machine got a problem on its own.

After looking closely to the symptoms, I noticed that the service freeze on the STARTING status but then it just stops. And I finally noticed a related crash notification in event viewer :

 <Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
 <System>
  <Provider Name=".NET Runtime" /> 
  <EventID Qualifiers="0">1026</EventID> 
  <Level>2</Level> 
  <Task>0</Task> 
  <Keywords>0x80000000000000</Keywords> 
  <TimeCreated SystemTime="2012-06-22T14:05:55.000000000Z" /> 
  <EventRecordID>47213</EventRecordID> 
  <Channel>Application</Channel> 
  <Computer>DEVW7-WKS24.bi.local</Computer> 
  <Security /> 
</System>
 <EventData>
  <Data>Application: DistributedCacheService.exe Framework Version: v4.0.30319 Description: The process was terminated due to an unhandled exception. Exception Info: Microsoft.ApplicationServer.Caching.DataCacheException Stack: at Microsoft.ApplicationServer.Caching.VelocityWindowsService.ThrowCallback(System.Object) at System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean) at System.Threading.QueueUserWorkItemCallback.System.Threading.IThreadPoolWorkItem.ExecuteWorkItem() at System.Threading.ThreadPoolWorkQueue.Dispatch() at System.Threading._ThreadPoolWaitCallback.PerformWaitCallback()</Data> 
  </EventData>
  </Event>

<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
  <Provider Name="Application Error" /> 
  <EventID Qualifiers="0">1000</EventID> 
  <Level>2</Level> 
  <Task>100</Task> 
  <Keywords>0x80000000000000</Keywords> 
  <TimeCreated SystemTime="2012-06-22T14:05:56.000000000Z" /> 
  <EventRecordID>47214</EventRecordID> 
  <Channel>Application</Channel> 
  <Computer>DEVW7-WKS24.bi.local</Computer> 
  <Security /> 
  </System>
<EventData>
  <Data>DistributedCacheService.exe</Data> 
  <Data>1.0.4632.0</Data> 
  <Data>4eafeccf</Data> 
  <Data>KERNELBASE.dll</Data> 
  <Data>6.1.7600.16850</Data> 
  <Data>4e211da1</Data> 
  <Data>e0434352</Data> 
  <Data>000000000000a88d</Data> 
  <Data>22d0</Data> 
  <Data>01cd507f04ac8810</Data> 
  <Data>d:\BI\AppFabric 1.1 pour Windows Server\DistributedCacheService.exe</Data> 
  <Data>C:\Windows\system32\KERNELBASE.dll</Data> 
  <Data>623c11b0-bc73-11e1-94e4-005056c00008</Data> 
  </EventData>
  </Event>

There was also a report generated (sorry it's in french) :

Version=1
EventType=CLR20r3
EventTime=129848475564305461
ReportType=2
Consent=1
UploadTime=129848475583672921
ReportIdentifier=623c11b1-bc73-11e1-94e4-005056c00008
IntegratorReportIdentifier=623c11b0-bc73-11e1-94e4-005056c00008
Response.type=4
Sig[0].Name=Signature du problème 01
Sig[0].Value=distributedcacheservice.exe
Sig[1].Name=Signature du problème 02
Sig[1].Value=1.0.0.0
Sig[2].Name=Signature du problème 03
Sig[2].Value=4eafeccf
Sig[3].Name=Signature du problème 04
Sig[3].Value=DistributedCacheService
Sig[4].Name=Signature du problème 05
Sig[4].Value=1.0.0.0
Sig[5].Name=Signature du problème 06
Sig[5].Value=4eafeccf
Sig[6].Name=Signature du problème 07
Sig[6].Value=5
Sig[7].Name=Signature du problème 08
Sig[7].Value=13
Sig[8].Name=Signature du problème 09
Sig[8].Value=SWOUM0PNYW4I1S3EYHEY4VNB5OWO0LJ1
DynamicSig[1].Name=Version du système
DynamicSig[1].Value=6.1.7600.2.0.0.256.4
DynamicSig[2].Name=Identificateur de paramètres régionaux
DynamicSig[2].Value=1036
DynamicSig[22].Name=Information supplémentaire n° 1
DynamicSig[22].Value=01a4
DynamicSig[23].Name=Information supplémentaire n° 2
DynamicSig[23].Value=01a42e2d5ddd2179fba3bf21874ae0ad
DynamicSig[24].Name=Information supplémentaire n° 3
DynamicSig[24].Value=0035
DynamicSig[25].Name=Information supplémentaire n° 4
DynamicSig[25].Value=00357e27ba04e43dde5682ce9f34b5f6
UI[2]=d:\BI\AppFabric 1.1 pour Windows Server\DistributedCacheService.exe
UI[5]=Rechercher une solution en ligne (recommandé)
UI[6]=Rechercher une solution plus tard (recommandé)
UI[7]=Fermer
UI[8]=Microsoft(R) Windows(R) Server AppFabric a cessé de fonctionner et a été arrêté
UI[9]=Un problème est à l’origine du dysfonctionnement de cette application. Windows vous préviendra si une solution est disponible.
UI[10]=&Fermer
LoadedModule[0]=d:\BI\AppFabric 1.1 pour Windows Server\DistributedCacheService.exe
LoadedModule[1]=C:\Windows\SYSTEM32\ntdll.dll
LoadedModule[2]=C:\Windows\SYSTEM32\MSCOREE.DLL
LoadedModule[3]=C:\Windows\system32\KERNEL32.dll
LoadedModule[4]=C:\Windows\system32\KERNELBASE.dll
LoadedModule[5]=C:\Windows\system32\ADVAPI32.dll
LoadedModule[6]=C:\Windows\system32\msvcrt.dll
LoadedModule[7]=C:\Windows\SYSTEM32\sechost.dll
LoadedModule[8]=C:\Windows\system32\RPCRT4.dll
LoadedModule[9]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\mscoreei.dll
LoadedModule[10]=C:\Windows\system32\SHLWAPI.dll
LoadedModule[11]=C:\Windows\system32\GDI32.dll
LoadedModule[12]=C:\Windows\system32\USER32.dll
LoadedModule[13]=C:\Windows\system32\LPK.dll
LoadedModule[14]=C:\Windows\system32\USP10.dll
LoadedModule[15]=C:\Windows\system32\IMM32.DLL
LoadedModule[16]=C:\Windows\system32\MSCTF.dll
LoadedModule[17]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
LoadedModule[18]=C:\Windows\system32\MSVCR100_CLR0400.dll
LoadedModule[19]=C:\Windows\assembly\NativeImages_v4.0.30319_64\mscorlib\e0e5fbe72e8813a135fc878ff32b4bee\mscorlib.ni.dll
LoadedModule[20]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\nlssorting.dll
LoadedModule[21]=C:\Windows\system32\ole32.dll
LoadedModule[22]=C:\Windows\system32\CRYPTBASE.dll
LoadedModule[23]=C:\Windows\system32\CRYPTSP.dll
LoadedModule[24]=C:\Windows\system32\rsaenh.dll
LoadedModule[25]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clrjit.dll
LoadedModule[26]=C:\Windows\system32\OLEAUT32.dll
LoadedModule[27]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System\c9d45f7c0deab777cea3e4fe77c02031\System.ni.dll
LoadedModule[28]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.ServiceProce#\127b422c3177e9f042ee3390dc53abff\System.ServiceProcess.ni.dll
LoadedModule[29]=d:\BI\appfabric 1.1 pour windows server\Microsoft.ApplicationServer.Caching.Server.dll
LoadedModule[30]=d:\BI\appfabric 1.1 pour windows server\Microsoft.WindowsFabric.Federation.dll
LoadedModule[31]=d:\BI\appfabric 1.1 pour windows server\Microsoft.ApplicationServer.Caching.Core.dll
LoadedModule[32]=d:\BI\appfabric 1.1 pour windows server\Microsoft.WindowsFabric.Common.dll
LoadedModule[33]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Xml\9a1aea68b24af9040536b0677c6c35ab\System.Xml.ni.dll
LoadedModule[34]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Configuration\88d89c2eb5f36a33cec8d1734c311f23\System.Configuration.ni.dll
LoadedModule[35]=C:\Windows\system32\urlmon.dll
LoadedModule[36]=C:\Windows\system32\WININET.dll
LoadedModule[37]=C:\Windows\system32\iertutil.dll
LoadedModule[38]=C:\Windows\system32\CRYPT32.dll
LoadedModule[39]=C:\Windows\system32\MSASN1.dll
LoadedModule[40]=C:\Windows\system32\SspiCli.dll
LoadedModule[41]=C:\Windows\system32\ntmarta.dll
LoadedModule[42]=C:\Windows\system32\WLDAP32.dll
LoadedModule[43]=C:\Windows\system32\SHELL32.dll
LoadedModule[44]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Core\06e5638d1925f6cf87ff8fad1ef06d75\System.Core.ni.dll
LoadedModule[45]=d:\BI\appfabric 1.1 pour windows server\fr-FR\Microsoft.ApplicationServer.Caching.Core.resources.dll
LoadedModule[46]=C:\Windows\system32\ws2_32.dll
LoadedModule[47]=C:\Windows\system32\NSI.dll
LoadedModule[48]=C:\Windows\system32\mswsock.dll
LoadedModule[49]=C:\Windows\System32\wshtcpip.dll
LoadedModule[50]=C:\Windows\System32\wship6.dll
LoadedModule[51]=C:\Windows\system32\DNSAPI.dll
LoadedModule[52]=C:\Windows\system32\IPHLPAPI.DLL
LoadedModule[53]=C:\Windows\system32\WINNSI.DLL
LoadedModule[54]=C:\Windows\system32\rasadhlp.dll
LoadedModule[55]=C:\Windows\System32\fwpuclnt.dll
LoadedModule[56]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.ServiceModel\41ff109cc439d1cdb05465f9101261c3\System.ServiceModel.ni.dll
LoadedModule[57]=C:\Windows\assembly\NativeImages_v4.0.30319_64\SMDiagnostics\2d3c35ffc646287827a79c31eb91c21d\SMDiagnostics.ni.dll
LoadedModule[58]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Runtime.Dura#\b719608cfb73833aec4ffb15928325ec\System.Runtime.DurableInstancing.ni.dll
LoadedModule[59]=d:\BI\appfabric 1.1 pour windows server\Microsoft.WindowsFabric.Data.Common.dll
LoadedModule[60]=d:\BI\appfabric 1.1 pour windows server\Microsoft.WindowsFabric.Data.dll
LoadedModule[61]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.Runtime.Seri#\f68180d9f4ade9c313f9ad20422eb1c0\System.Runtime.Serialization.ni.dll
LoadedModule[62]=C:\Windows\assembly\NativeImages_v4.0.30319_64\System.IdentityModel\06cd9761b93a2882309d01af90f9cc9c\System.IdentityModel.ni.dll
LoadedModule[63]=C:\Windows\system32\pcwum.DLL
LoadedModule[64]=C:\Windows\system32\shfolder.dll
LoadedModule[65]=C:\Windows\system32\security.dll
LoadedModule[66]=C:\Windows\system32\SECUR32.DLL
LoadedModule[67]=C:\Windows\system32\credssp.dll
LoadedModule[68]=C:\Windows\system32\kerberos.DLL
LoadedModule[69]=C:\Windows\system32\cryptdll.dll
LoadedModule[70]=C:\Windows\system32\bcrypt.dll
LoadedModule[71]=C:\Windows\system32\bcryptprimitives.dll
LoadedModule[72]=C:\Windows\Microsoft.NET\Framework64\v4.0.30319\diasymreader.dll
LoadedModule[73]=C:\Windows\system32\VERSION.dll
FriendlyEventName=Fonctionnement arrêté
ConsentKey=CLR20r3
AppName=Microsoft(R) Windows(R) Server AppFabric
AppPath=d:\BI\AppFabric 1.1 pour Windows Server\DistributedCacheService.exe

Answer 4

Hello Eric,

In your clusterconfig.xml, the "size" attribute is different for 2 nodes - size is supposed to be homogeneous. All the nodes in the cluster should have same size.

Important
It is important that the cache cluster runs in a homogeneous environment of servers. It is not supported for cache hosts to have different physical specifications in memory, processor speed, and other capabilities Refer: http://msdn.microsoft.com/en-us/library/hh334311.aspx

One question: Are you registering/adding all the cache hosts first and then starting the cluster using (Start-CacheCluster)?

or

Registering one machine then start-cachecluster and then register another machine and start-cachehost on the second one?

Its better to register/add all the hosts first and then start-cachecluster.

Refer:

http://blogs.msdn.com/b/appfabric/archive/2011/07/06/lead-hosts-versus-sql-server-for-windows-server-appfabric-caching.aspx

http://msdn.microsoft.com/en-us/library/hh351477.aspx