500.19 errors when connecting to UNC share off Linux-based NAS

Question

User1639000639 posted
I have a Linux-based NAS that I'm trying to connect to my web farm, but am having a really difficult time doing it in a secure way.

I don't want to jump through the configuration hoops to make Samba join the domain, but I am rather attempting the "same  username and password on both sides" technique.  I can make it work flawlessly if I create an "administrator" user on the Linux server that has the same password as the one on the web server.  The issue comes in when I try to use any other user, which I'm trying to do for security.  No matter what I try, I get the dreaded 500.19 error, "cannot read configuration file", as if it's trying to read the web.config off a non-existent directory.  To be specific, it's trying to read the web.config off any file I request, as if the file I request is a directory.  For example, "\\?\UNC\10.10.179.144\Simple\Default.htm\web.config".  Default.htm isn't a directory.

In ways I think Schofield would approve, I've debugged this thing to death.  I've used Process Monitor, Wireshark, tcpdump, smbstatus, and the highest level of Samba logging.  The odd thing is that everything appears to be checking out.  The user (who I've set as the identity on the app pool) looks to be logging in fine.  You can see in this screenshot from Process Monitor, http://cl.ly/0d0g2r133g0O2P1g1V3u, that it's not getting any kind of access denial.  And here is a screenshot from smbstatus on the Linux side, http://cl.ly/1u1v3W3M0k1m1r2d2a2k, showing a successfully connected web server.

And here is the part of the Samba logs that shows that my "simpleuser" is able to log in fine and retrieve default.htm:
[2011/07/23 11:10:42, 2] auth/auth.c:check_ntlm_password(309)
  check_ntlm_password:  authentication for user [simpleuser] -> [simpleuser] -> [simpleuser] succeeded
[2011/07/23 11:10:42, 1] smbd/service.c:make_connection_snum(1077)
  windscale2 (10.10.179.136) connect to service Simple initially as user simpleuser (uid=502, gid=503) (pid 12819)
[2011/07/23 11:10:42, 2] auth/auth.c:check_ntlm_password(309)
  check_ntlm_password:  authentication for user [simpleuser] -> [simpleuser] -> [simpleuser] succeeded
[2011/07/23 11:10:42, 1] smbd/service.c:make_connection_snum(1077)
  windscale2 (10.10.179.136) connect to service Simple initially as user simpleuser (uid=502, gid=503) (pid 12819)
[2011/07/23 11:10:42, 2] smbd/open.c:open_file(391)
  simpleuser opened file default.htm read=Yes write=No (numopen=2)
[2011/07/23 11:10:57, 2] smbd/close.c:close_normal_file(406)
  simpleuser closed file default.htm (numopen=0) NT_STATUS_OK
[2011/07/23 11:11:09, 1] smbd/service.c:close_cnum(1274)
  windscale2 (10.10.179.136) closed connection to service Simple

So, I'm pulling my hair out.  It *appears* that, since everything looks to be checking out on the Linux side, that the 500.19 determination is strictly on the Windows side.  My guess is that it's throwing a fit about some permission issue it thinks it has.

Thank you for your help!

Ken

Answer 1

User1073881637 posted

I'm impressed! :)  That is the first time anyone put that in a post about me personally.  I'm humbled. 

I recall in one of the posts people putting a 'connect as' user in the vdir in IIS so it connects to linux share, which has that user on each side. If I read your post correctly., you did that and was able to connect, however the user on the linux side was an "administrator".  Is that correct? 

http://www.iislogs.com/tags/unc  (not sure you checked out the UNC tag I have on my blog).  That might help undercover something.

Answer 2

User1639000639 posted

Thanks, Steve!  Honored to have a reply from you.

That is actually one of the reasons I've backpedaled so much as to create an ultra-simple test website completely run off a UNC share, since my original problem (which I left out for brevity) was that I was having difficulty with accessing any content from a vdir off a UNC share to the NAS.  And, with that vdir problem, I tried every permutation possible, changing app pool identity, changing the "connect as" on the vdir, both, while jogging at the same time, etc.

To answer your question, I am able to get everything to run dandily if I have an "administrator" user on both the Linux side and the Windows side, and those "administrator" details are in the app pool identity and the "connect as" settings.

In the effort to make things more secure, I am now attempting to have a "simpleuser" user on both sides, and with those details all in IIS (simpleuser as the app pool identity, and set as the "connect as" details).  Then things fall apart with 500.19 errors, even though it appears IIS is able to connect to the NAS successfully, and to even retrieve content.  It just won't show it to me.

Thanks for the UNC tag reference on your blog, I will keep digging!

Ken

Answer 3

User1073881637 posted

You are going about it the right way, it can be tedious.  When doing something like this, try without worrying about security (firewalls, AV, permssions), get it working then back off permissions one step at a time.  From your description, few things come to mine

1) any firewalls in the way?

2) any anti-virus preventing something?

3) I'm not certain you need the windows side as a local administrator.  I would create a local or domain user with the same password as your linux counterpart.

4) One thing to verify is the local security policy > Local policies > security options  > Lan manager authentication level.

I would also increase the auditing on the windows side to collect both success and failure.  In my experience, when process monitor doesn't show any access denies and running as a local administrator, there is some permission in the local security policy that needs tweaked (user rights assignment).  Some of the popular ones are logon as a batch, impersonate a client after authentication, replace a process token level, adjust memory quotas for a process.  I'm not saying all need to adjusted, it's another place to look.  Hope that provides some direction.

Answer 4

User1639000639 posted

 Thanks, Steve.

 I've double-checked #1 (firewalls), #2 (anti-virus), and we're fine there.

I am trying to get everything going without using the main administrator account, but even with a local user on the Windows side with the same exact credentials as on the Samba side, I get the 500.19 errors.

I am not completely positive on #4 (lan manager auth level), but wouldn't that be eliminated as a possibility if everything's able to work using the parallel administrator accounts?

I also am unable to set up auditing on the share, since it's a Samba share on a Linux box.  If I try to set it up, it acts like it saves the setting, but it doesn't... Samba just discards it I suppose.  I do have plenty of tools on the Linux side like strace, tcpdump, Wireshark, and really verbose Samba logs to help.

Here's something new that may give you a hint of what's going on.  I just figured this out. 

The simple act of adding the local 'simpleuser' user account the local Administrators group makes everything work.  ('simpleuser' on the app pool identity, on the 'connect as', and as the Samba user).  That's progress, I think!  (But still a dud when it comes to security.)  Here's a screenshot of Process Monitor after a single successful request, with 'simpleuser' belonging to the Administrators group:  http://cl.ly/3a1D1Y1J331Z3X1o052g

If I remove 'simpleuser' from the Administrators group, then recycle, then the 500.19 re-appears.  Here is a screenshot of a 500.19-producing request, from Process Monitor's eyes:  http://cl.ly/1g1Y1b1D0Y1r350O052P

There are no access problems between the two screenshots, but you can see that some of the "options" are different.

Microsoft's only guidance regarding this specific 500.19 error, 0x8007010b, is:  "We have seen this error when the site content is pointing to some Non-NTFS File system. In such cases, it is advisable to test it by placing the content on a Windows/NTFS share."

Not very helpful in this case.  I'm really close to spinning up two fresh, clean VMs, one with Win2K8R2 and one with CentOS 5.6, and starting from even more scratch than I already am.

Answer 5

User1073881637 posted

I got my FreeBSD 8.2 server setup with SAMBA installed.  I have a user called winuser (both on BSD and Windows - local user) that runs as the app pool (the anonymous authentication module inherits the application pool identity).  The 'connect as user' for the vdir which is mapped to \\192.168.0.52\winuser   I was getting an error when trying while the IUSR account was used as the anonymous user, after I changed to 'inherit from application pool identity', did my aspx pages work from the SAMBA share.  The other thing I had to add the winuser account to the SAMBA database. 

http://www.us-webmasters.com/FreeBSD/Install/Samba/

I'm working on setting up a Fedora instance to replicate on a linux disto and not FreeBSD.  BSD is easier for me to get going on Hyper-V. 

Hope that helps, I hope to blog my findings in the near future.  Not sure it'll help anyone, but it's pretty cool to have a w2k8 r2 box using FreeBSD 8.2 as a SAMBA server. 

Answer 6

User1639000639 posted

I'm not sure if my last post is still in the moderation queue or not, but the TLDR version of it was that I checked all the things you listed, and that I made progress in that adding my test user (named "simpleuser" in my case), ceased to generate a 500.19 if they were added to the Administrators group.  Not great for security, but that was at least progress.

I've fired up a brand spanking fresh CentOS 5.6 instance, and a fresh Win2K8R2 instance, and you know what?  I'm not having the 500.19 issue.  :(  In the span of about 5 minutes I was able to set up the exact scenario I have set up on my other servers, and I can't get the 500.19 to happen.

At least I have something to compare with.  And, hopefully once I figure out what the difference is, this will be a help to someone else out there in Google land.  The only potential difference I see so far is that this Windows test instance isn't joined to a domain, whereas the other one I'm running tests on is on a domain.

The plot thickens...  Thank you for listening to all this, Steve.  :)

Ken

Answer 7

User1639000639 posted

Ok, I now know exactly why I'm getting 500.19 errors.  I lied when I said my setup on the new test machines was exactly the same as the other test servers.

My new test setup actually had a few less Samba parameters set up in the smb.conf.

The short of it is, if you have "case sensitive = yes" in your smb.conf, then you will get 500.19 errors when you try to authenticate with anyone other than an administrator.  That is odd.

The reason I have "case sensitive = yes" is that that majorly speeds up Samba when you're dealing with millions of folders and files when you're accessing it through IIS.  Otherwise, if Samba can't find a file that you request of it, it'll do a complete directory listing and do a string comparison of each and every file in there to see if there is a case insensitive match for the file you requested.  Big, huge, giant, staggering perf hit when you get any good size of content on your Linux NAS.

Now I'm up against another wall...  I will be Googling to see what I can find.  The more specialized I get, the quieter Google gets, so wish me luck.

Ken

Answer 8

User1639000639 posted

The world is silent on this.  I will dive into the world of the Samba mailing list for some help.  They'll likely be able to help me to diagnose what IIS is asking for.  Once I find out what IIS is asking for specifically, then I'll come back.  Am wondering if it's possible to write some kind of module or filter to intercept the request before it goes out to the Samba server to appease the "case sensitive = yes" gods.

I'm also curious if Samba4 is able to deal with this...

Ken

Answer 9

User1073881637 posted

I'm afraid I can't help on perf with Samba.  It would be great if you passed along anything to this thread.  It would help others.  The Linux Distro I'm using is CentOS 5.6.  First time I've worked with this Distro.

One remaining thing, did you change the anonymous authentication module from IUSR to INHERIT APPLICATION POOL Identity?

Answer 10

User-1672167363 posted

Hello,

Maybe this http://technet.microsoft.com/en-us/library/cc754351.aspx Technet information

"Subsystem for UNIX-based Applications and POSIX Compliance" helps.

Along with this Microsoft Support article http://support.microsoft.com/kb/929110 and

"A file system that was case sensitive becomes case insensitive after you install an update for the .NET Framework 2.0"

it has the Windows Registry Keys and settings.

I use "Open Suse"  linux and used this guide http://opensuse.swerdna.org/index.html  for settings

the Samba smb.conf might help ?

Cheers :)

Martin

 

Answer 11

User1639000639 posted

Wow, thank you, Martin.  Didn't know Windows supported case-sensitivity.  I will fire up a couple test VMs tomorrow and set up a test condition to see if that makes everything work.

Appreciate the help!

Ken

 

Answer 12

User1639000639 posted

Steve,

Thank you, I appreciate your help on this!

I will certainly post back if I can figure out what's going on.

Yes, I run my sites' anon auth module on "inherit application pool".  I've tried a few different ways, but that's the normal way I run it.  In this situation, it hasn't helped.

Ken

Answer 13

User-1672167363 posted

Hi,

Check this Technet Topic:

"Deploying and Configuring IIS 6.0 with Remotely Stored Content on UNC Servers and NAS Devices" and

Guide http://technet.microsoft.com/en-us/library/dd296641.aspx

which "Applies To: Windows 7, Windows Server 2008, Windows Server 2008 R2, Windows Vista" 

from IIS Team Blog  Tom Kaminski

http://blogs.iis.net/tomkmvp/archive/2009/04/28/deploying-and-configuring-internet-information-services-iis-6-0-with-remotely-stored-content-on-unc-servers-and-nas-devices.aspx .

Martin

 

 

Answer 14

User1639000639 posted

Thanks for the additional reading material, I've got that queued up.  Thanks, Martin.

I've made some additional progress as well.  To recap the story so far, I have parallel usernames and passwords on the Win2K8R2 box and on the Linux NAS (which uses Samba).  My Samba server has "case sensitive = yes" for perf reasons.  When that's turned off, I have no issues, but since I have to have it on, I have to deal with it.  So, knowing that, I had to find a workaround.  The (not good) workaround thus far has been adding that Windows user to the Administrators group.  When I do that, things work great.

The new development on the story is thusly:  So, knowing that the user being in the Administrators group solves everything, I set out to see if any other group membership would solve the issue.  So, I removed its membership from Administrators, and added every other group to it.  Recycled.  Boom, it works!  Great!  So I peeled off every group one by one until I found the group that did the trick.

It was "Backup Operators".  Which rang a bell, because if you look at the Process Monitor screenshots above, the one that works (where the user is part of the Administrators group) lists "Options: Open For Backup" when it goes to open a file from the UNC share.  (http://cl.ly/3a1D1Y1J331Z3X1o052g)  The screenshot of the scenario that does not work (when the user does not belong to any groups) instead has "Options: Open Reparse Point" when it goes to open a file from the UNC share. (http://cl.ly/1g1Y1b1D0Y1r350O052P)

What that means, I don't know, but the fact that IIS is attempting to open files using the "open for backup" method, it makes sense that Administrators and Backup Operators are able to pull that request off.

I still think this is a gross workaround, a hacky patch for some deeper underlying issue I'm not understanding.  But this at least gets me some additional security.  I can still protect this app pool from poking around the rest of the server.  I don't know enough what kind of vulnerabilities can be caused from belonging to Backup Operators just yet.  So, the journey isn't over yet.

Ken

Answer 15

User1073881637 posted
I would try this, take the user out of backup operators, open local security policy and add the user to the options as backup operators, ensure it works, then start to take out of the options 1 at a time until it breaks.   I'm pretty sure the local policy is preventing running as a regular user. 

Answer 16

User1639000639 posted

I hang my head low as I type this, but the positive thing is that everything is completely, utterly resolved.

Surely you'd think that Linux distros would have available the latest stable release of Samba in its repos.  And you'd be correct.  But wouldn't you think that "yum -y install samba" would install that latest release?  Yeah, me, too. 

Except that that innocent command inexplicably installed the ancient (their major version cycles take nearly a decade!) 3.0.x version of Samba.  Hey, but look, they offer the most current, stable release under the name of "samba3x".  So, I "yum remove samba" and then "yum install samba3x", and poof, all my authentication issues go away, regardless of the user being in the Administrators group, or the Backup Operators group.

So I'm now running on 3.5.4, and things are great.  Steve, what you should have asked was, "Do you have the most recent version of Samba installed?", which is embarrassingly close to "Is your computer plugged in?".

Thanks for your help, and I hope this thread can help some poor sap.  One hotly anticipated thing about the upcoming 3.6.0 release of Samba (they're only in release candidate mode at the moment) is that full SMB2 support is on the way.

Ken

Answer 17

User-1672167363 posted

Hi,

Glad you finally got it resolved. :D

Sorry to say I agree not all "Linux" distros are created nore updated the same.

I do have a suggestion ping and post the information at "Linux Forums" and maybe even  the Technet Social Forums.

 Cheers :

Martin