Release management deploying components failing since update 2

Question

Since I updated Release Management with update 2 I am having issues deploying to my external servers.
Before update 2 it worked fine. Now everytime during the deployment it hangs on copying the component. It gives the following message:

Communication with the deployer was lost during deployment. Please make sure (1) the deployer machine has not rebooted during installation and (2) the component timeout is sufficient to copy the files from the drop location to the deployer machine and install the package.

After a while the deployment fails and goes into rejected state. In the log it just shows some random failed activity with an exit code. 

When deploying to servers in the local network it goes fine. Only difference is that internal servers use the UNC path copy action and the external the http copy action. I already tried to increase the component timeout, but of no avail.

It looks like there are some issues with the http copy deployment. Any ideas or patches available?
At the moment I cannot succesfully deploy to my external servers anymore. Since update 2.

Answer 1

Hi Chris, 

Thanks for your post.  

Recreate a new component to copy files from drop location to that external server in your RM Update 2 still receive the same error?   

For this scenario, please submit it to Microsoft Connect Feedback portal at: https://connect.microsoft.com/VisualStudio. Microsoft engineers will evaluate them seriously.

---

We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
Click HERE to participate the survey.

Answer 2

Tried it and still having same issue.
I also experienced that the process was hanging on single steps (like 'start website'). Many times it just ends up in 'Rejected' state and when you view the log you can only see succeeded steps. So no indication of something failing. Also nothing to be found in the event viewer on the server itself.

In the screenshot you see that the deployment in my local test environment succeeded (deployment through UNC path). When it goes to deploy on the remote server it fails. In the log everything is on 'succeeded'. It did however performed a rollback action I put in.

I also tried to restart the service and it didnt help. I can try to reinstall the deployer. It was updated by the server when I installed the update 2 version of the server.

The annoying part is that I can not find any failure messages in logs or something else. Any indications on where I can look for it?

Answer 3

Been searching more and found the following errors in the event viewer on the RM server

Message: The operation could not be performed because WorkflowApplication 58f73c24-9a1c-41b3-8cd0-9df66bbd6833 is aborted.: \r\n\r\n   at System.Activities.WorkflowApplication.ThrowIfAborted()
   at System.Activities.WorkflowApplication.ValidateStateForRun()
   at System.Activities.WorkflowApplication.InternalRun(TimeSpan timeout, Boolean isUserRun)
   at System.Activities.WorkflowApplication.ResumeBookmark(Bookmark bookmark, Object value, TimeSpan timeout)
   at System.Activities.WorkflowApplication.ResumeBookmark(Bookmark bookmark, Object value)
   at System.Activities.WorkflowApplication.ResumeBookmark(String bookmarkName, Object value)
   at Microsoft.TeamFoundation.Release.ServiceProcessor.Processor.DeploymentControllerServiceProcessor.ContinueDeploymentWithFinalComponentState(DeploymentLog log)
   at Microsoft.TeamFoundation.Release.ServiceProcessor.Processor.DeploymentControllerServiceProcessor.ContinueDeployment(String log)
Process Name: C:\Windows\SysWOW64\inetsrv\w3wp.exe

and

Message: Data at the root level is invalid. Line 2, position 1.: \r\n\r\n   at System.Xml.XmlTextReaderImpl.Throw(Exception e)
   at System.Xml.XmlTextReaderImpl.Throw(String res, String arg)
   at System.Xml.XmlTextReaderImpl.ParseRootLevelWhitespace()
   at System.Xml.XmlTextReaderImpl.ParseDocumentContent()
   at System.Xml.XmlTextReaderImpl.Read()
   at System.Xml.Linq.XDocument.Load(XmlReader reader, LoadOptions options)
   at System.Xml.Linq.XDocument.Parse(String text, LoadOptions options)
   at Microsoft.TeamFoundation.Release.Common.Helpers.WebServiceHelper.ExtractWebMethod(String pathInfo, Stream inputStream)

Answer 4

Hi Chris, 

Thanks for your reply.

Have you reinstalled your deploy agent? If reinstalled the deploy agent still receive the same result, I think it’s an issue in this new Release Management Update 2 product, please submit this issue in that feedback site. After you submitted it, please share your feedback link here.

---

We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
Click HERE to participate the survey.

Answer 5

Hi,
I have not reinstalled the agent, but increased all the timeout settings. On the components and in the settings area. I increased it quit a bit.
This solved it for now at least. Two different projects released fully now with no more failures.

One thing is though is that when you get failures because of timeouts, it doesnt show this. As in my screenshot, the release is rejected but all steps show up succeeded.

Answer 6

Back again. Although it does help, increasing timeout times, I still am experiencing this issue.
It often happens that its hanging on a step during deployment. As if communication between deployer and server just stop.
I now have releases hanging on pending for hours.

Looking into the event viewer of the RM server I find these with source 'Release Management Services':

Message: Time taken when trying to load the WorkflowApplication: 180801,2511 Milliseconds
Category: General
Priority: -1
EventId: 0
Severity: Error
Process Name: C:\Windows\SysWOW64\inetsrv\w3wp.exe

and

Message: Failed to load WorkflowApplication a7a51288-5379-43cd-9c5f-93fb2e9cef9c
Category: General
Priority: -1
EventId: 0
Severity: Error
Process Name: C:\Windows\SysWOW64\inetsrv\w3wp.exe

Answer 7

Hi Chris,  

Thanks for your reply.

Have you submitted this issue in feedback site? I think there need a patche to fix this issue in Release Management 2013 Update 2. 

---

We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
Click HERE to participate the survey.

Answer 8

Yes I did. I also now reinstalled the deployer on the server.
Now it seems to be working again.
Thanks for the info.

Answer 9

Hi Chris,  

Thanks for your reply.

Please share the feedback link here, we will tracking the replies in it.

---

We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
Click HERE to participate the survey.

Answer 10

Feedback link https://connect.microsoft.com/VisualStudio/feedback/details/879345/release-management-deploying-components-failing-since-update-2

Answer 11

I know this post is old, but I've been struggling for the past week or so with this workflow "TimeTaken" event triggered by Release Manager 2013 (Update 4) - finally I've found an answer to my scenario and thought I'd share it...

The event below had been occuring every minute for months:

Message: WAEP.ContinueDeploymentWithFinalComponentState: Task:WorkflowInstanceId: InstanceOwnerId: - Failed to load WorkflowApplication. Time taken when trying to load the WorkflowApplication:1843.746 Milliseconds

For a while this had not impact, but then RM started hanging and anytime it did this, these errors would change to time taken values of 1million ms or higher! For a while our work around was to recycle the RM app pool.

After some log parsing (over 300k events!) I found that only 6 WorkflowInstanceIDs were erroring. I did some digging into the Release IDs these related to and found that all of them were in a 'cancelled' state - further to this I found that the deployment action for each of these release had a stale 'In progress' step!

So many hours of searching around the RM DB, I found this table [ReleaseManagement].[dbo].[ReleaseV2ActivityLog] and within said table, a column called 'Status'. This 'Status' column correlates to the action steps for every release in RM. Status value of 2 means 'in progress' - so you'd expect any currently running releases to appear in the table. However, what I found were the active releases plus my 6 additional 'cancelled' releases (with the same 6 familiar GUIDs related to them).

Running an update on the table to change Status = '2' to '3' where the workflowinstanceid matches the erroring GUIDs will flick the stage to 'Complete' and stop the error!

Long story short:-

1. Parse the event errors and extract the unique workflowinstanceid's
2. Check the table: [ReleaseManagement].[dbo].[ReleaseV2ActivityLog] where status = '2'
3. Compare the WorkflowInstanceID's you noted down against the results in SQL
4. Update the table for all the matching WorkflowInstanceIDs where status = '2' to '3'
5. Live an error free life... for now.

Answer 12

It's work for me, thanks