WCF Call History and WF Instance History Not Updating

Question

I've created a workflow service and hosted this in AppFabric and the WCF Call History and WF Instance History counts do not update when the service is executed. However the Persisted WF Instance count does get correctly updated.

I have checked the following

- Application Server Event Collection Service, didn't start automatically so I manually started this and restarted IIS after doing this

This is on Windows 2008 SP2

Any pointers to what I've missed.

Thanks

Owain

Answer 1

Hi,

Please take a look at the "StagingTable" and "WcfEvents" tables in the database.  If you see rows in the StagingTable and not in the WcfEvents table then you may have the following issue:

When the system is working properly WCF events move from the staging table into the wcf events table.  For SQL Express, this will happen automatically.  For other SQL SKUs, the SQL agent is responsible for moving the events.  Please make sure the SQL Agent is running.

If this is not the problem then you may have a permissions error.  Ensure that when you initially configured your monitoring database that you setup permissions correctly.

To perform more in depth diagnostics you can check the following event logs using eventvwr.exe.  These logs are used by the Event Collector service.

    Microsoft-Windows-Application Server-System Services\Admin
    Microsoft-Windows-Application Server-System Services\Debug

Please let me know if you see any errors in there.  Thanks.

Adam

Answer 2

Thank you Adam

The SQL Agent thing did the trick in my case. I am using SQL Server Std, not Express.

Regards

Javier

Answer 3

I have the same problem, the StagingTable is full of rows, but WcfEvents is empty. I am running SQL Express on Windows 7. I don't think it's a permissions error, I used Database Configuration in AppFabric to initialize my database, and it has added the users and groups (AS_Administrators, AS_Observers, IIS_IUSRS) to the database. 

I have enabled the Admin and Debug logs that you mentioned. The Admin log is still empty, and the Debug log only contains these messages: 

Collecting commands from database 'Data Source=.\SQLEXPRESS;Initial Catalog=ApplicationServerExtensions;Integrated Security=True;Asynchronous Processing=True'.

and

Collected '0' ready commands and '0' future commands from database 'Data Source=.\SQLEXPRESS;Initial Catalog=ApplicationServerExtensions;Integrated Security=True;Asynchronous Processing=True'.

Any idea what could be wrong? WF Persistence works, by the way, to the same database and shows up in the Dashboard

Answer 4

I should add that I also tried having the monitoring and persistence database on an SQL Server Developer Edition (instead of Express) as well, but with the same results. Everything just stays in the StagingTable. And yes, the SQL Server Agent is running.

/Johan

Answer 5

Hi Johan,

The lack of Admin and Debug logs from the Event Collector Service (the logs you are seeing on collected commands are from Workflow Management Service) suggests that the Event Collector service isn't running. Could you please verify the following in the Services management console (services.msc)?

1) The Application Server Event Collector service is running, restart if needed
2) The SQL Server Agent service is running, restart if needed
3) What identities are the Event Collector Service and SQL Server Agent service running as (in Beta1 they should both be Local System I believe)

Thanks,
Miguel

Answer 6

Hi Miguel,

That was my thought too, but

1) The Application Server Event Collector is running, and I have restarted both it and the computer many times

2) The SQL Server Agent was running when I used SQL Developer Ed., although I was under the impression that it is not needed for SQL Express? At least that's what it says in the answer above.

3) Local System for the Event Collector, SQL Agent was running as Network Service (which I believe is the default)

But it still doesn't work...

/Johan

Answer 7

Thanks Johan

Could you please provide the following details,

1) In your application's Monitoring configuration settings (you can reach these by navigating to your app in IIS Manager, and clicking on 'Configure...' in the Action Pane in the right). Is this currently pointing to a connection string name for a SQLExpress or Sql Dev Edition database? If it is SqlExpress, then Windows Service Broker is used. If it is Sql Dev/Enterprise edition, then Sql Agent is used.
    i) If it is using Sql Dev Edition, could you please go to Sql Management Studio -> Sql Agent -> Jobs-> and verify when was the last time the 'ImportEvents' or 'ImportWCFEvents' jobs was run. Was the latest runs successful or where there any errors?
    ii) (if Sql Dev ed) Could you please try changing the identity of the Sql Agent service to Local System and restart it?

2) After ensuring Event Collector is running, try calling the application again (to generate new events), are these new events stuck in the Staging Table, get processed to the WCF/WFEventsTable, or do not reach the database (StagingTable) at all?

Thanks,
Miguel

Answer 8

1) It is an SqlExpress database.

2) Yes, the Event Collector service is running, and the events get to the StagingTable, but get stuck there. The StagingTable now contains about 1300 rows, but both the WCFEventsTable and the WFEventsTable are emtpy.

/Johan

Answer 9

Thanks Johan for providing us with these details.

Could you try manually running the ImportEvents stored procedure for that particular database? This should make the events drain from the Staging table to the processed views. If it doesn't, can you please verify the FailedStagingEventsTable and look for any errors shown?

The ImportEvents stored procedure is the one that moves events from the StagingTable into the WCF and WF event tables. The tools (dashboard) read data from the views of these tables.

In SQLExpress, Windows Service Broker schedules and runs the ImportEvents stored procedure. In the case of Sql Dev and above (Standard/Enterprise), SqlAgent is used to schedule a job that runs this stored procedures.  The creation of the windows service broker schedules/sql agent jobs are done during database creation (i.e. when running InitializeMonitoringDatabase for example). The account that was used to create the DB is the account that will be set as owner of this windows service broker schedule/Sql agent jobs. If these jobs are not being scheduled properly, it usually points to a permissions issue with this account. The recommendeded approach is that when creating the database, a local account with dbo privileges is used.

Thanks,
Miguel

Answer 10

Running the ImportEvents procedure worked fine, and as far as I can tell, all events from the StagingTable were imported into WcfEvents and WfEvents. 

I did use "InitializeMonitoringDatabase" in IIS when creating the database, and I just accepted the default values, that is "ACTIVE701\AS_Administrators" for Administrators, "BUILTIN\IIS_IUSRS" for Users and "ACTIVE701\AS_Observers" as Observers (my computer name is ACTIVE701). I was running with my normal account, which has dbo privileges to the database.

Is there any way to see how and if the windows service broker schedule fails?

Answer 11

Thanks Johan,

You can verify if the Service Broker is enabled, by going to the properties of that database, and in Options->Service Broker->Broker Enabled should be set to 'True'.

In addition, if this is set to true and you still continue to see these errors, you can also see the JobsTable, for the last time the schedule was run, and if the run was successful (Select * from JobsTable). 

On the permissions issue, was the account you were running under when executing the InitializeMonitoringDatabase a local account or a domain account? If a domain account was used, and the machine is later disconnected from the domain, the jobs/schedules will not be able to authenticate the user's permissions they are running under. If the scenario requires to enable disconnecting from the domain, then a local account with dbo priveleges is recommended to be used.

Please let us know if this fixes the issues you encountered.

Thanks,
Miguel