we encounter a lot of request time out while accessing appfabric cache cluster. Even after restart appfabric cache, the issue is still exists. The error message are:
-
ErrorCode<ERRCA0018>:SubStatus<ES0001>:The request timed out
-
ErrorCode<ERRCA0016>:SubStatus<ES0001>:The connection was terminated, possibly due to server or network problems or serialized Object size is greater than MaxBufferSize on server. Result of the request is unknown.
-
Message : The socket connection was aborted. This could be caused by an error processing your message or a receive timeout being exceeded by the remote host, or an underlying network resource issue. Local socket timeout was '00:00:15'
The % Time in GC image is https://social.msdn.microsoft.com/Forums/getfile/1043001
Every 5 min we have captured the statistics.
Thursday, April 20, 2017 12:43:27 PM
CacheA
Size : 61943808
ItemCount : 1600
RegionCount : 1118
RequestCount : 232283
ReadRequestCount : 220629
WriteRequestCount : 5265
MissCount : 16846
IncomingBandwidth : 57237131
OutgoingBandwidth : 28868297165
ServerName : CACHE-1
Size : 29717504
ItemCount : 977
RegionCount : 726
NamedCacheCount : 3
RequestCount : 80295
MissCount : 1819
ServerName : CACHE-2
Size : 15409152
ItemCount : 959
RegionCount : 706
NamedCacheCount : 3
RequestCount : 69150
MissCount : 15369
ServerName : CACHE-3
Size : 30809088
ItemCount : 991
RegionCount : 740
NamedCacheCount : 3
RequestCount : 119366
MissCount : 15239
===========================================================
Thursday, April 20, 2017 12:48:27 PM
CacheA
Size : 62268416
ItemCount : 1816
RegionCount : 1226
RequestCount : 304394
ReadRequestCount : 290570
WriteRequestCount : 5898
MissCount : 19769
IncomingBandwidth : 63945483
OutgoingBandwidth : 36649595735
ServerName : CACHE-1
Size : 30449664
ItemCount : 1125
RegionCount : 802
NamedCacheCount : 3
RequestCount : 106927
MissCount : 2099
ServerName : CACHE-2
Size : 15943680
ItemCount : 1094
RegionCount : 775
NamedCacheCount : 3
RequestCount : 84761
MissCount : 18274
ServerName : CACHE-3
Size : 31399936
ItemCount : 1133
RegionCount : 813
NamedCacheCount : 3
RequestCount : 159747
MissCount : 18656
===========================================================
Thursday, April 20, 2017 12:53:27 PM
CacheA
Size : 62720000
ItemCount : 2016
RegionCount : 1308
RequestCount : 353606
ReadRequestCount : 337431
WriteRequestCount : 6688
MissCount : 23082
IncomingBandwidth : 69217072
OutgoingBandwidth : 42944998939
ServerName : CACHE-1
Size : 31029248
ItemCount : 1254
RegionCount : 861
NamedCacheCount : 3
RequestCount : 122586
MissCount : 2429
ServerName : CACHE-2
Size : 16546816
ItemCount : 1220
RegionCount : 828
NamedCacheCount : 3
RequestCount : 99822
MissCount : 21471
ServerName : CACHE-3
Size : 32122880
ItemCount : 1261
RegionCount : 869
NamedCacheCount : 3
RequestCount : 185841
MissCount : 22292
===========================================================
Thursday, April 20, 2017 12:58:27 PM
CacheA
Size : 62838784
ItemCount : 2122
RegionCount : 1366
RequestCount : 404695
ReadRequestCount : 386461
WriteRequestCount : 7404
MissCount : 25903
IncomingBandwidth : 74532993
OutgoingBandwidth : 49273053878
ServerName : CACHE-1
Size : 31485952
ItemCount : 1249
RegionCount : 901
NamedCacheCount : 3
RequestCount : 139302
MissCount : 2567
ServerName : CACHE-2
Size : 17624064
ItemCount : 1234
RegionCount : 883
NamedCacheCount : 3
RequestCount : 114248
MissCount : 24280
ServerName : CACHE-3
Size : 31704064
ItemCount : 1260
RegionCount : 919
NamedCacheCount : 3
RequestCount : 213783
MissCount : 25642
===========================================================
Thursday, April 20, 2017 1:03:27 PM
CacheA
Size : 62926848
ItemCount : 2238
RegionCount : 1402
RequestCount : 463397
ReadRequestCount : 443370
WriteRequestCount : 8027
MissCount : 28293
IncomingBandwidth : 80264982
OutgoingBandwidth : 55649800635
ServerName : CACHE-1
Size : 30999552
ItemCount : 1308
RegionCount : 946
NamedCacheCount : 3
RequestCount : 160311
MissCount : 2822
ServerName : CACHE-2
Size : 17143808
ItemCount : 1274
RegionCount : 925
NamedCacheCount : 3
RequestCount : 127770
MissCount : 26661
ServerName : CACHE-3
Size : 32014336
ItemCount : 1324
RegionCount : 969
NamedCacheCount : 3
RequestCount : 247015
MissCount : 29349
===========================================================
Thursday, April 20, 2017 1:08:27 PM
CacheA
Size : 62777344
ItemCount : 2288
RegionCount : 1442
RequestCount : 517318
ReadRequestCount : 495358
WriteRequestCount : 8708
MissCount : 30916
IncomingBandwidth : 85825768
OutgoingBandwidth : 62627439007
ServerName : CACHE-1
Size : 30520320
ItemCount : 1307
RegionCount : 983
NamedCacheCount : 3
RequestCount : 178792
MissCount : 3074
ServerName : CACHE-2
Size : 16750592
ItemCount : 1262
RegionCount : 963
NamedCacheCount : 3
RequestCount : 141380
MissCount : 29222
ServerName : CACHE-3
Size : 31470592
ItemCount : 1288
RegionCount : 998
NamedCacheCount : 3
RequestCount : 276617
MissCount : 32607
