locked
IIS 7.5 site 20,000 UNC virdirs via NetApp Filer Results in 500.19 NetBios Command limit reached errors RRS feed

  • Question

  • User-2141405750 posted

    Hi,

    I have been researching this issue for the past week, and would like some input/feedback and possibly answers from others who may have run into the same issue.

    We run a Windows 2008 R2 IIS 7.5 web server to host roughly 20,000 virtual directories (personal web sites). The directories are all UNC paths off of a share on our NetApp filer. We have had a similar setup with the same amount of virtual directories running in an all Windows enviornment for almost a decade (Windows 2000/IIS 5 and Windows file servers). With the new setup using NetApp CIFS, everything works for a little while, then all of the sudden we begin to get HTTP 500.19 errors for many of the virtual directories. The IIS tracing module allowed us to find the real cause which is "The network BIOS command limit has been reached. (0x80070038)". It appears to be running out of NetBIOS commands. I researched the issue and found that some SMB settings may need adjusted. I increased the MaxCmds for the lanmanworkstation service on the web server and it did not help. We are reluctant to change the cifs.max_mpx on the filer due to the many warrnings in NetApp documentation. We do not want to affect other clients accessing shares on the filer. Also, it is important to note that if I recycle the apppool for the web site that hosts the 20,000 unc virtual directories, everything works again for a little bit.  When the recycle happens I see a whole bunch of NT Cancel SMB commands and file closes in the network capture. But soon again, we get the 500.19 errors again. It appears the IIS web server is doing a Change Notify request for each new virtual directory access (I assume each dir requires one command to sit pending, which builds up I assume). My question is what can be done to fix this issue? This issue isn't present when using all Windows servers (IIS 5 web server and 2003 file server).  Is increasing the cifs.max_mpx on the filer and MaxCmds on the IIS 7.5 web server ok to do, or is it just a bandaid for something broken? We have 20,000 possible directories that could have change notifications on them, so eventually we would top NetApp's highest recommended cifs.max_mpx setting. Is it something with the way NetApp handles change notifications?  Can I tell IIS 7.5 to not request Change Notify for directories and/or not check for web.config files (I have no delegation enabled -- all config in apphost, and I already disabled web.config checks for subdirectories).   

     Thank you!

     

    Tuesday, October 5, 2010 4:06 PM

Answers

  • User1073881637 posted

    Glad the posts were useful.    Does NetApp support SMB 2.0?   SMB 2.0 greatly enhances SMB based communication over 1.0.  As far as I know, this only works between w2k8 and greater between each other.  If w2k8 connects to w2k3/xp, the OS reverts to SMB 1.0

     

    • Marked as answer by Anonymous Tuesday, September 28, 2021 12:00 AM
    Friday, October 8, 2010 10:59 PM

All replies

  • User-1757254373 posted
    Are you running the latest data ontap? Also, what is your cifs.max_mpx set to?
    Tuesday, October 5, 2010 11:57 PM
  • User-2141405750 posted

    Hi, thanks for responding.

    DataOntap 7.3.2

    cifs.max_mpx set to default of 50.

     Thanks again!

    Wednesday, October 6, 2010 11:13 AM
  • User-1757254373 posted

    Test in your lab, but I run something like your setup.

    cifs terminate
    options cifs.max_mpx 15365
    cifs restart
    iisreset the webnodes that use the netapp

    Wednesday, October 6, 2010 2:13 PM
  • User1073881637 posted

    Sounds like you have some good advice.  I'd suggest turning off TCP Chimney.

    Here are some good threads I've seen about UNC.

    http://weblogs.asp.net/steveschofield/archive/tags/UNC/default.aspx

    Wednesday, October 6, 2010 8:53 PM
  • User-2141405750 posted

    Test in your lab, but I run something like your setup.

    cifs terminate
    options cifs.max_mpx 15365
    cifs restart
    iisreset the webnodes that use the netapp

     Thanks for the reply and info!

    Early this morning I set cifs.max_mpx on the NetApp filter to 1124 (maximum recommended by NetApp Docs) and then restarted cifs, and then set MaxCmds to 1124 on the Windows 2008 R2 IIS 7.5 web server and restarted the server.  I will monitor the log and reply back with my findings.

    jimrPA, how did you come up with that 15365 number for cifs.max_mpx?  Was this recommened/supported by NetApp?  What version of Data OnTap are you running?

    Thanks again!

    Thursday, October 7, 2010 8:11 AM
  • User-2141405750 posted

     

    Sounds like you have some good advice.  I'd suggest turning off TCP Chimney.

    I saw this in my research on this issue, and already disabled TCP Chimney, RSS, and NetDMA.  We normally disable chimney as it causes so many issues. :)

     

     

    Here are some good threads I've seen about UNC.

    http://weblogs.asp.net/steveschofield/archive/tags/UNC/default.aspx

    I used that page in my research and troubleshooting.  Thanks for making and maintaining it!  :)

     

    Thursday, October 7, 2010 8:21 AM
  • User1073881637 posted

    Glad the posts were useful.    Does NetApp support SMB 2.0?   SMB 2.0 greatly enhances SMB based communication over 1.0.  As far as I know, this only works between w2k8 and greater between each other.  If w2k8 connects to w2k3/xp, the OS reverts to SMB 1.0

     

    • Marked as answer by Anonymous Tuesday, September 28, 2021 12:00 AM
    Friday, October 8, 2010 10:59 PM
  • User-1757254373 posted

    Yep, the netapp filers do support SMB2.0 depending on your ontap version

     I run Data OnTap 7.3.2P4. I am not the storage guy, but I am pretty sure netapp support was called for this and this is what we set our netapps at. If you have access to the netapp support page, may find some useful things in there for cifs.

     

    Hope all works out in the end!

    Friday, October 8, 2010 11:19 PM
  • User-2141405750 posted

    Test in your lab, but I run something like your setup.

    cifs terminate
    options cifs.max_mpx 15365
    cifs restart
    iisreset the webnodes that use the netapp

     Thanks for the reply and info!

    Early this morning I set cifs.max_mpx on the NetApp filter to 1124 (maximum recommended by NetApp Docs) and then restarted cifs, and then set MaxCmds to 1124 on the Windows 2008 R2 IIS 7.5 web server and restarted the server.  I will monitor the log and reply back with my findings.

    jimrPA, how did you come up with that 15365 number for cifs.max_mpx?  Was this recommened/supported by NetApp?  What version of Data OnTap are you running?

    Thanks again!

    Update:

    Hi, well, it looks like setting the cifs.max_mpx setting to 1124 on the filer and setting the MaxCmds registry setting on the IIS server to 1124 fixed the issue!  I have yet to seen another 500.19 error in the log for many days!

    The reason we didn't do this earlier is because based on the existing troubleshooting docs for this issue, we didn't need to (with exception to the IIS error and windows error code pointing to running into the command limit).  Furthermore, we were running a similar setup for years with IIS 5 and a windows file server.  During the troubleshooting I saw no "current commands" for Redirector running performance monitor on the IIS server.  So for Windows 2008 R2 IIS 7.5 connecting to a SMB 1.0 only enabled NetApp, it appears you can't use this performance stat to judge if you need to increase the cifs.max_mpx and MaxCmds.  Furthermore the Max Multiplex stat on the filer (doing cifs stat command) didn't get past 30 (50 would mean it hit the max_mpx default of 50), which should mean there is no need to increase max_mpx.  It was hard to convience my boss to let me change the filer setting, but I am glad I did, as I finally got rid of my headache! :) 

    Thank you so much to everyone!

     

     

    Tuesday, October 12, 2010 1:34 PM
  • User-1757254373 posted
    Glad it is resolved. Have a good one
    Tuesday, October 12, 2010 4:09 PM
  • User1073881637 posted

    Thanks for posting the solution.  Added to the UNC tag!

    http://bit.ly/adxN3c

    Tuesday, October 12, 2010 8:49 PM
  • User920506185 posted

    Nice that setting:

     options cifs.max_mpx 15365

    fixed the issue, while it seems  jumping from 50 to 15365 seems a little like a brute force to me.  If 50 was working with only a few random error for 20,000 vdirs, it just seems excessive to go from 50 to 15365?   While it's hard to argue with success :)  Just curious if the jump in the setting was a little excessive?

     

    Monday, October 18, 2010 1:58 PM
  • User-1757254373 posted

    I think he changed it to the netapp recommended 1124 for the max. I was telling him with my setup, to get around the issue we had, we set it to 15365.

    Tuesday, October 19, 2010 5:57 AM