locked
Web Services suddenly stopped responding / working

    Question

  • I apologize if this isn't the correct forum . .but I'm not really sure even where to start.  We have 2 web services - one  a 'custom' web service that retrieves documents from a server and another one that retrieves documents from a FAND server.  'Suddenly' yesterday morning they stopped working.  We are getting errors such as:

    ***     System.Net.WebException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.IO.IOException: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host. ---> System.Net.Sockets.SocketException: An existing connection was forcibly closed by the remote host

       at System.Net.Sockets.Socket.EndReceive(IAsyncResult asyncResult)

       at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)

       --- End of inner exception stack trace ---

       at System.Net.Sockets.NetworkStream.EndRead(IAsyncResult asyncResult)

       at System.Net.PooledStream.EndRead(IAsyncResult asyncResult)

       at System.Net.CommandStream.ReadCallback(IAsyncResult asyncResult)

       --- End of inner exception stack trace ---

       at Microsoft.VisualBasic.MyServices.Internal.WebClientCopy.DownloadFile(Uri address, String destinationFileName)

       at Microsoft.VisualBasic.Devices.Network.DownloadFile(Uri address, String destinationFileName, ICredentials networkCredentials, Boolean showUI, Int32 connectionTimeout, Boolean overwrite, UICancelOption onUserCancel)

       at Microsoft.VisualBasic.Devices.Network.DownloadFile(Uri address, String destinationFileName, ICredentials networkCredentials, Boolean showUI, Int32 connectionTimeout, Boolean overwrite)

       at Gmc.Common.LoanDocuments.LoanDocumentGrid.ShowDocument(LoanDocument document)

       at Gmc.Common.LoanDocuments.LoanDocumentGrid.DataGridView1_CellContentClick(Object sender, DataGridViewCellEventArgs e)

       at System.Windows.Forms.DataGridViewCell.OnMouseUpInternal(DataGridViewCellMouseEventArgs e)

       at System.Windows.Forms.DataGridView.OnCellMouseUp(DataGridViewCellMouseEventArgs e)

       at System.Windows.Forms.DataGridView.OnMouseUp(MouseEventArgs e)

       at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)

       at System.Windows.Forms.Control.WndProc(Message& m)

       at System.Windows.Forms.DataGridView.WndProc(Message& m)

       at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)

       at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

     AND another error:

     

     

    System.Net.WebException: The underlying connection was closed: An unexpected error occurred on a receive.

       at Microsoft.VisualBasic.MyServices.Internal.WebClientCopy.DownloadFile(Uri address, String destinationFileName)

       at Microsoft.VisualBasic.Devices.Network.DownloadFile(Uri address, String destinationFileName, ICredentials networkCredentials, Boolean showUI, Int32 connectionTimeout, Boolean overwrite, UICancelOption onUserCancel)

       at Microsoft.VisualBasic.Devices.Network.DownloadFile(Uri address, String destinationFileName, ICredentials networkCredentials, Boolean showUI, Int32 connectionTimeout, Boolean overwrite)

       at Gmc.Common.LoanDocuments.LoanDocumentGrid.ShowDocument(LoanDocument document)

       at Gmc.Common.LoanDocuments.LoanDocumentGrid.DataGridView1_CellContentClick(Object sender, DataGridViewCellEventArgs e)

       at System.Windows.Forms.DataGridView.OnCellContentClick(DataGridViewCellEventArgs e)

       at System.Windows.Forms.DataGridView.OnCommonCellContentClick(Int32 columnIndex, Int32 rowIndex, Boolean doubleClick)

       at System.Windows.Forms.DataGridViewCell.OnMouseUpInternal(DataGridViewCellMouseEventArgs e)

       at System.Windows.Forms.DataGridView.OnCellMouseUp(DataGridViewCellMouseEventArgs e)

       at System.Windows.Forms.DataGridView.OnMouseUp(MouseEventArgs e)

       at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)

       at System.Windows.Forms.Control.WndProc(Message& m)

       at System.Windows.Forms.DataGridView.WndProc(Message& m)

       at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)

       at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)

       at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

     

    Can anyone help please?  We had not changed anything on the system that I know of when the services stopped.  We have tried stopping and restarting the services and stopping and restarting the entire web server.  Still not working.

    Saturday, April 18, 2009 11:55 AM

All replies

  • Perhaps the network guys installed a new proxy server or firewall?  Is that possible.
    http://www.alanjmcf.me.uk/ Please follow-up in the newsgroup. If I help, mark the question answered
    Saturday, April 18, 2009 10:16 PM
  • Thanks for Responding Alan!  I appreciate it.  I talked with the network guy and found out 2 things:  the problem appears to have started after the web server was restarted on friday morning (as it is every Friday).  From what he could tell something happened to the IP address of the website???  He changed the address to a different address and the web services seemed to work again...  Until I Published the TEST application again (using click-once deployment).  Now, once again the web services have stopped working for BOTH the Test and the Production application.   Urgh! 
    Can you help me? 
    I'm wondering about the System.Web.Extensions.  Can they be causing me a problem?  In References for the project, we have been using .Net 2.0 Ajax Extensions .. but in the web config file for the website, we are using 3.5.00.  I have NOT published the website since before we started having this problem . . if that makes any difference.
    I'm really stuck and need help . .. I have spent hours staring at this and I'm exhausted.  Any ideas would be appreciated..
    Thanks in advance for any help!
    Sunday, April 19, 2009 3:56 AM
  • Can you get a sniffer trace, or turn on System.Net tracing so we can see at what point the fault occurs. :-) See http://msdn2.microsoft.com/en-us/library/ty48b824.aspx and/or http://blogs.msdn.com/dgorti/archive/2005/09/18/471003.aspx  We want to have System.Net.Sockets tracing at least.
    http://www.alanjmcf.me.uk/ Please follow-up in the newsgroup. If I help, mark the question answered
    Sunday, April 19, 2009 9:08 AM
  • Good morning Alan!  I was so glad to see you had responded again this AM.  I may have done something incorrectly . .but when I added the Network Traces . . it wouldn't even let me log in!!!  ???  I tried only the System.Net.Sockets trace (removed System.Net and System.Net.Cache) and still couldn't log in.  This is what I got back:

        ***     Csla.DataPortalException: DataPortal.Fetch failed (System.TypeInitializationException: The type initializer for 'System.Data.SqlClient.SqlConnection' threw an exception. ---> System.TypeInitializationException: The type initializer for 'System.Data.SqlClient.SqlConnectionFactory' threw an exception. ---> System.TypeInitializationException: The type initializer for 'System.Data.SqlClient.SqlPerformanceCounters' threw an exception. ---> System.Configuration.ConfigurationErrorsException: The element <sources> may only appear once in this section. (C:\Documents and Settings\jrose\Local Settings\Apps\2.0\JEDCCJQY.1G9\5J5C265W.HLP\gems..tion_ea449c9a1f2f373c_0001.0026_640b0f8ba3df5380\Gems.exe.config line 63)

       at System.Configuration.BaseConfigurationRecord.EvaluateOne(String[] keys, SectionInput input, Boolean isTrusted, FactoryRecord factoryRecord, SectionRecord sectionRecord, Object parentResult)

       at System.Configuration.BaseConfigurationRecord.Evaluate(FactoryRecord factoryRecord, SectionRecord sectionRecord, Object parentResult, Boolean getLkg, Boolean getRuntimeObject, Object& result, Object& resultRuntimeObject)

       at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject)

       at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject)

       at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject)

       at System.Configuration.BaseConfigurationRecord.GetSection(String configKey, Boolean getLkg, Boolean checkPermission)

       at System.Configuration.BaseConfigurationRecord.GetSection(String configKey)

       at System.Configuration.ClientConfigurationSystem.System.Configuration.Internal.IInternalConfigSystem.GetSection(String sectionName)

       at System.Configuration.ConfigurationManager.GetSection(String sectionName)

       at System.Configuration.PrivilegedConfigurationManager.GetSection(String sectionName)

       at System.Diagnostics.DiagnosticsConfiguration.GetConfigSection()

       at System.Diagnostics.DiagnosticsConfiguration.Initialize()

       at System.Diagnostics.DiagnosticsConfiguration.get_SwitchSettings()

       at System.Diagnostics.Switch.InitializeConfigSettings()

       at System.Diagnostics.Switch.InitializeWithStatus()

       at System.Diagnostics.Switch.get_SwitchSetting()

       at System.Diagnostics.TraceSwitch.get_Level()

       at System.Data.ProviderBase.DbConnectionPoolCounters..ctor(String categoryName, String categoryHelp)

       at System.Data.SqlClient.SqlPerformanceCounters..ctor()

       at System.Data.SqlClient.SqlPerformanceCounters..cctor()

       --- End of inner exception stack trace ---

       at System.Data.SqlClient.SqlConnectionFactory..cctor()

       --- End of inner exception stack trace ---

       at System.Data.SqlClient.SqlConnection..cctor()

       --- End of inner exception stack trace ---

       at System.Data.SqlClient.SqlConnection..ctor()

       at System.Data.SqlClient.SqlConnection..ctor(String connectionString)

       at Gfm.Gems.BO.Security.GfmIdentity.DataPortal_Fetch(Criteria criteria)) ---> Csla.Server.CallMethodException: DataPortal_Fetch method call failed ---> System.TypeInitializationException: The type initializer for 'System.Data.SqlClient.SqlConnection' threw an exception. ---> System.TypeInitializationException: The type initializer for 'System.Data.SqlClient.SqlConnectionFactory' threw an exception. ---> System.TypeInitializationException: The type initializer for 'System.Data.SqlClient.SqlPerformanceCounters' threw an exception. ---> System.Configuration.ConfigurationErrorsException: The element <sources> may only appear once in this section. (C:\Documents and Settings\jrose\Local Settings\Apps\2.0\JEDCCJQY.1G9\5J5C265W.HLP\gems..tion_ea449c9a1f2f373c_0001.0026_640b0f8ba3df5380\Gems.exe.config line 63)

       at System.Configuration.BaseConfigurationRecord.EvaluateOne(String[] keys, SectionInput input, Boolean isTrusted, FactoryRecord factoryRecord, SectionRecord sectionRecord, Object parentResult)

       at System.Configuration.BaseConfigurationRecord.Evaluate(FactoryRecord factoryRecord, SectionRecord sectionRecord, Object parentResult, Boolean getLkg, Boolean getRuntimeObject, Object& result, Object& resultRuntimeObject)

       at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject)

       at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject)

       at System.Configuration.BaseConfigurationRecord.GetSectionRecursive(String configKey, Boolean getLkg, Boolean checkPermission, Boolean getRuntimeObject, Boolean requestIsHere, Object& result, Object& resultRuntimeObject)

       at System.Configuration.BaseConfigurationRecord.GetSection(String configKey, Boolean getLkg, Boolean checkPermission)

       at System.Configuration.BaseConfigurationRecord.GetSection(String configKey)

       at System.Configuration.ClientConfigurationSystem.System.Configuration.Internal.IInternalConfigSystem.GetSection(String sectionName)

       at System.Configuration.ConfigurationManager.GetSection(String sectionName)

       at System.Configuration.PrivilegedConfigurationManager.GetSection(String sectionName)

       at System.Diagnostics.DiagnosticsConfiguration.GetConfigSection()

       at System.Diagnostics.DiagnosticsConfiguration.Initialize()

       at System.Diagnostics.DiagnosticsConfiguration.get_SwitchSettings()

       at System.Diagnostics.Switch.InitializeConfigSettings()

       at System.Diagnostics.Switch.InitializeWithStatus()

       at System.Diagnostics.Switch.get_SwitchSetting()

       at System.Diagnostics.TraceSwitch.get_Level()

       at System.Data.ProviderBase.DbConnectionPoolCounters..ctor(String categoryName, String categoryHelp)

       at System.Data.SqlClient.SqlPerformanceCounters..ctor()

       at System.Data.SqlClient.SqlPerformanceCounters..cctor()

       --- End of inner exception stack trace ---

       at System.Data.SqlClient.SqlConnectionFactory..cctor()

       --- End of inner exception stack trace ---

       at System.Data.SqlClient.SqlConnection..cctor()

       --- End of inner exception stack trace ---

       at System.Data.SqlClient.SqlConnection..ctor()

       at System.Data.SqlClient.SqlConnection..ctor(String connectionString)

       at Gfm.Gems.BO.Security.GfmIdentity.DataPortal_Fetch(Criteria criteria)

       --- End of inner exception stack trace ---

       at System.Data.SqlClient.SqlConnection..ctor()

       at System.Data.SqlClient.SqlConnection..ctor(String connectionString)

       at Gfm.Gems.BO.Security.GfmIdentity.DataPortal_Fetch(Criteria criteria)

       at Csla.MethodCaller.CallMethod(Object obj, MethodInfo info, Object[] parameters)

       at Csla.Server.SimpleDataPortal.Fetch(Type objectType, Object criteria, DataPortalContext context)

       --- End of inner exception stack trace ---

       at System.Data.SqlClient.SqlConnection..ctor()

       at System.Data.SqlClient.SqlConnection..ctor(String connectionString)

       at Gfm.Gems.BO.Security.GfmIdentity.DataPortal_Fetch(Criteria criteria)

       at Csla.MethodCaller.CallMethod(Object obj, MethodInfo info, Object[] parameters)

       at Csla.Server.SimpleDataPortal.Fetch(Type objectType, Object criteria, DataPortalContext context)

       at Csla.DataPortal.Fetch(Type objectType, Object criteria)

       at Csla.DataPortal.Fetch[T](Object criteria)

       at Gfm.Gems.BO.Security.GfmIdentity.GetIdentity(String userName, String password)

       at Gfm.Gems.BO.Security.GfmPrincipal.LogOn(String userName, String password)

       at Gfm.Gems.UI.LogOn.OK_Click(Object sender, EventArgs e)

     
    And as you can probably guess .  I have no idea what all this means.

    I commented out the trace lines of code and was able to log in again.  And when I tried to run the command that uses the Web Service (retrieving the documents), I was back to the original posted error.

    Any ideas now?

    Sunday, April 19, 2009 12:38 PM
  • The first exception there is:
    System.Configuration.ConfigurationErrorsException:
    The element <sources> may only appear once in this section.
    (C:\Documents and Settings\jrose\Local Settings\Apps\2.0\JEDCCJQY.1G9\5J5C265W.HLP\gems..tion_ea449c9a1f2f373c_0001.0026_640b0f8ba3df5380\
    Gems.exe.config line 63)

    So something's gone wrong in editing the config.  Maybe it would be easier just to use the config file from the MSDN page verbatim for now. :-)

    http://www.alanjmcf.me.uk/ Please follow-up in the newsgroup. If I help, mark the question answered
    Sunday, April 19, 2009 12:48 PM
  • Thanks for your patience. :)  I changed the app.config so that the only thing in the configuration <system diagnositcs> section is the section from the article.

    Now when I run the app and try to pull down the documents . . it gets stuck and does not error out.  I have waited way beyond the time it should have errored out . . like 30 minutes..??  I finally canceled the request.

    My question is . . the messages I sent you previously have been emailed to me 'automatically' once an error occurs.  I don't know where the 'log' files are stored.  Is it possible it is writing 'stuff' to the log file even though it isn't erroring out and emailing me the results?  Is there some way I can find the log file and look at it?

    Thanks SO MUCH!

    Sunday, April 19, 2009 2:11 PM
  • It's me again.  I finally got a document to error out and send the message.  Here it is:

    System.Net.WebException: The underlying connection was closed: The server committed a protocol violation.

       at Microsoft.VisualBasic.MyServices.Internal.WebClientCopy.DownloadFile(Uri address, String destinationFileName)

       at Microsoft.VisualBasic.Devices.Network.DownloadFile(Uri address, String destinationFileName, ICredentials networkCredentials, Boolean showUI, Int32 connectionTimeout, Boolean overwrite, UICancelOption onUserCancel)

       at Microsoft.VisualBasic.Devices.Network.DownloadFile(Uri address, String destinationFileName, ICredentials networkCredentials, Boolean showUI, Int32 connectionTimeout, Boolean overwrite)

       at Gmc.Common.LoanDocuments.LoanDocumentGrid.ShowDocument(LoanDocument document)

       at Gmc.Common.LoanDocuments.LoanDocumentGrid.DataGridView1_CellContentClick(Object sender, DataGridViewCellEventArgs e)

       at System.Windows.Forms.DataGridView.OnCellContentClick(DataGridViewCellEventArgs e)

       at System.Windows.Forms.DataGridView.OnCommonCellContentClick(Int32 columnIndex, Int32 rowIndex, Boolean doubleClick)

       at System.Windows.Forms.DataGridViewCell.OnMouseUpInternal(DataGridViewCellMouseEventArgs e)

       at System.Windows.Forms.DataGridView.OnCellMouseUp(DataGridViewCellMouseEventArgs e)

       at System.Windows.Forms.DataGridView.OnMouseUp(MouseEventArgs e)

       at System.Windows.Forms.Control.WmMouseUp(Message& m, MouseButtons button, Int32 clicks)

       at System.Windows.Forms.Control.WndProc(Message& m)

       at System.Windows.Forms.DataGridView.WndProc(Message& m)

       at System.Windows.Forms.Control.ControlNativeWindow.OnMessage(Message& m)

       at System.Windows.Forms.Control.ControlNativeWindow.WndProc(Message& m)

       at System.Windows.Forms.NativeWindow.Callback(IntPtr hWnd, Int32 msg, IntPtr wparam, IntPtr lparam)

    Sunday, April 19, 2009 2:15 PM
  • Can you explain the systems here.  The exceptions all suggest that the failure is reported in a WinForms app.  Is that WinForms app not running on your PC?  I presumed you were downloading it with ClickOnce deployment.  What is it that is being emailed to you? :-)

    The System.Net tracing should have been written to a file on the local disk, presumably in the current working directory.  I guess that might be the local ClickOnce installation directory, something deep with c:\Users\Foo\ApplicationSettings\... or similar.  Maybe you can put a absolute path in the config file, to have it written somewhere obvious.

    Is the IP Address for the Web Service included in the ClickOnce config file?  Is it for the updated web server address.  When the app is trying to contact the web server, does netstat -n in a command prompt show an (attempted) connection to the correct address.

    Maybe you should contact Microsoft Support if this is urgent and might be something that changed due to different versions on the web server.
    http://www.alanjmcf.me.uk/ Please follow-up in the newsgroup. If I help, mark the question answered
    Sunday, April 19, 2009 4:06 PM
  • You are correct.  I am using a Click-Once Deployment for this app and the Winforms app that it is referring to is a custom written form that is a grid that lists documents.  You click on the document and click the VIEW button and then it goes out to the web service and retrieves the document based on information in a database table.

    Some information about the error is being emailed because that is the way the application is set up.  Any time an error occurs it is emailed to our 'technical support' mailbox.  I found the local ClickOnce directory, but the only thing in there is a user.config file that basically stores their user id.  However, I notice more than one directory in there for the application - some of them really old.  Would that do anything?

    The IP address is not included in the ClickOnce config file.  You are over my head with the 'netstat -n' thing..??

    A little background if you will so indulge me, I have been trying to do a Click-Once deployment on an application I've never done one on before . . .(the person who did do it is no longer with the company) and while I do the click-once deployment on another application -- it is no where near the 'monster' this one is.  So, I have a little knowledge but only enough to be dangerous.  Anyway, I've been trying to get all the settings etc. correct and in trying to do so I have run into several 'issues'. . So

    I'm going to ask you another question that seems to be 'out there' . but I've been wracking my brain on this issue since Friday AM.  The ONLY thing I can come up with that could have caused this problem is . .  I had been publishing and installing the Test machine all week with no problems.  On Thursday I switched to the production machine to hopefully do a live deployment this weekend (I checked settings, etc. to try it from there) and I did have one problem.  I had a missing reference to System.Web.Extensions.  I added the reference referred to in .NET .   .it didn't work . so I checked the settings on an old machine and it referenced the AJAX 2.0 Extension version.  I changed the Production machine to that version and deployed .  unfortunately, while that version was listed in References, when I dug further . . in Publish / Application Files . .it was supposed to be EXCLUDED.  Our DataPortal (btw:  we also use csla) already references the System.Web.Extensions -- but version 3.5.   Urgh!!!  I'm afraid that I sent something up to the server and now the references are messed up. 

    Could this be part of it .  .or am I way off base?

    I really, really, really appreciate your responses.  Thank you !  Thank you!!!
    Sunday, April 19, 2009 4:40 PM
  • I'm not a web services person so you might want to ask again in another forum, maybe at http://forums.asp.net or at http://social.msdn.microsoft.com/Forums/en-US/asmxandxml/threads

    If you try to open the webservices page using a web browser what do you see, do it seem to work there?


    When you are trying to troubleshoot this, can't you just run the app locally from within VisualStudio rather than have to run it via ClickOnce -- which makes it harder to redo the config etc.  Anyway, the System.net tracing config should be in a file called MyExeName.exe.config when the app is run, which is this case is: "C:\Documents and Settings\jrose\Local Settings\Apps\2.0\JEDCCJQY.1G9\5J5C265W.HLP\gems..tion_ea449c9a1f2f373c_0001.0026_640b0f8ba3df5380\Gems.exe.config".  If you look in that (very oddly named folder) do you see the tracing output?

    netstat.exe is a command-line program that shows whats happening on the TCP/IP stack.  "netstat -n" will show the open TCP/IP connections (and without name lookup).
    http://www.alanjmcf.me.uk/ Please follow-up in the newsgroup. If I help, mark the question answered
    • Proposed as answer by edhickey Monday, April 27, 2009 5:41 PM
    Monday, April 20, 2009 10:27 AM