polling external source every minute - can a web app do this?

Question

User-23818800 posted

Hi

As per the subject, i need to poll an XML file every minute and update my database with a status 

Is there anyway to do this in a asp.net web app (maybe using latest framework?)

 

i dont really want to create a service to run on the server to do this

 

Thanks

Answer 1

User-1853252149 posted

Does this XML file really change every minute?  As far as polling, the best method would be a service or to do this from within SQL.  A web app is not the place to do timed events.

Jeff 

Answer 2

User-1618234021 posted

Hi

You can use the Global.asax to schedule timed operations. Following is the code for that:

Public Class Global_asax
    Inherits System.Web.HttpApplication
    Shared t As Timers.Timer
    Sub Application_Start(ByVal sender As Object, ByVal e As EventArgs)
        ' Fires when the application is started
        t = New Timers.Timer(60000) 'means every 60 seconds
        t.Enabled = True
        AddHandler t.Elapsed, AddressOf doit
    End Sub
    Sub doit(ByVal sender As Object, ByVal e As Timers.ElapsedEventArgs)
     'write here the code to poll the xml file and do the processing
    End Sub
   
    Sub Application_End(ByVal sender As Object, ByVal e As EventArgs)
        t.Enabled = False
        t.Dispose()
    End Sub

End Class

Answer 3

User-23818800 posted

Does this XML file really change every minute?  As far as polling, the best method would be a service or to do this from within SQL.  A web app is not the place to do timed events.

Jeff 

 

 

It changes more than every min, but want to reduce the server load

 

Answer 4

User-23818800 posted

Hi

You can use the Global.asax to schedule timed operations. Following is the code for that:

 

view plaincopy to clipboardprint?

1. Public Class Global_asax   
2.     Inherits System.Web.HttpApplication   
3.     Shared t As Timers.Timer   
4.     Sub Application_Start(ByVal sender As Object, ByVal e As EventArgs)   
5.         ' Fires when the application is started   
6.         t = New Timers.Timer(60000) 'means every 60 seconds   
7.         t.Enabled = True  
8.         AddHandler t.Elapsed, AddressOf doit   
9.     End Sub  
10.     Sub doit(ByVal sender As Object, ByVal e As Timers.ElapsedEventArgs)   
11.      'write here the code to poll the xml file and do the processing   
12.     End Sub  
13.       
14.     Sub Application_End(ByVal sender As Object, ByVal e As EventArgs)   
15.         t.Enabled = False  
16.         t.Dispose()   
17.     End Sub  
18.   
19. End Class  

Public Class Global_asax
    Inherits System.Web.HttpApplication
    Shared t As Timers.Timer
    Sub Application_Start(ByVal sender As Object, ByVal e As EventArgs)
        ' Fires when the application is started
        t = New Timers.Timer(60000) 'means every 60 seconds
        t.Enabled = True
        AddHandler t.Elapsed, AddressOf doit
    End Sub
    Sub doit(ByVal sender As Object, ByVal e As Timers.ElapsedEventArgs)
     'write here the code to poll the xml file and do the processing
    End Sub
   
    Sub Application_End(ByVal sender As Object, ByVal e As EventArgs)
        t.Enabled = False
        t.Dispose()
    End Sub

End Class

 

 

Awesome!

Thanks fella,
Is this limited to any version of .net (will 2.0 be ok?)

Are there any issues that i may encounter or is this a good solution?

Answer 5

User-952121411 posted

This subject seems to be one of the most talked about topics in this forum, and it is a good one.  Typically the best formula for stability and longevity when needing to poll most anything in .NET is to create a Windows Service to house this operation.  There are indeed circumstances when a site is hosted and the owner can not use a Windows Service, and in these situations a solution within the actual web app might work but I caution against it due to stability factors with keeping a web site open on the client indefinitely.

Your quickest and easiest option is to use a Windows Service.  If you have never created one before, no worries - it is not that difficult.  Take a look below for the information you need to successfully implement running a periodic process to poll on a set interval using a Windows Service:

Running a Periodic Process in .NET using a Windows Service:

http://allen-conway-dotnet.blogspot.com/2009/12/running-periodic-process-in-net-using.html

Hope this helps!

Answer 6

User-1618234021 posted

Hi

Yeah its fine with .net 2.0.

Answer 7

User-23818800 posted

 I caution against it due to stability factors with keeping a web site open on the client indefinitely.

 

Oh

I didn't realise i'd need to keep the website open on a client to get this working?

if that is the case, then this solution wont work, i need the acutal web app to manage the polling and not be reliant on clients having the website open

If someone could double confirm this then i'll need to go the windows service route

thanks

 

Answer 8

User-1618234021 posted

Hi

No this is not the case. You do not need to keep the website opened on the client end. The code is a server side code embedded in the Global.asax. When your application is hosted and started on the server side. Application_Start will setup the timer and will keep on polling the tick event of the timer, irrespective of the fact that it is opened on client end or not.

It is purely happening on the server side. So you can safely go ahead with this solution.

Answer 9

User-952121411 posted

Ok my correction.   I have not done it that way before.  I guess I ask why do it this way as opposed to a tool that was designed for this sort of task: a Windows Service?

Would there be no concern of Process Recycling in IIS and having the 'Application_End' event be called; couldn't this potentially disrupt service or polling (even with overlapped recycling)?  If your app depends at all on Session state, then that could be lost during recycling as well and cause issues.  The nature of IIS is to recycle the applications to free up unused resources and allow applications to run smoothly. I even know some servers where batch files force IIS to reset in the middle of the night to provide for best app performance.  It to me always seems that IIS is not the best tool to house processes that run indefinitely as 'background processes', but rather is best at what its main intention is: act as a web server to host and run web applications,FTP, SMTP, etc.

I suppose if you did venture down this road, you would want to place this application in its own Application Pool and monitor it closely, to make sure its continuous operation is seamless and is always running as needed. I understand that even during a recycle, the app should seamlessly be restarted and thus the 'Application_Start' event would be called restarting the timers, but I still wouldn't use this as Option # 1 in my book for this sort of operation.  To me it is one of those 'It can be done, but certainly not the preferred way' scenarios.  I would still opt for a Windows Service, that would only stop running your service if the server was rebooted (which also would bring IIS down), or the programming was bad (an issue neither method is immune to).

IIS Process Recycling:

http://msdn.microsoft.com/en-us/library/ms525803.aspx

Keep IIS Alive:

http://bytes.com/topic/asp-net/answers/661317-keep-iis-alive

 

Answer 10

User-389939489 posted

atconway:

 > I guess I ask why do it this way as opposed to a tool that was designed for this sort of task: a Windows Service?

A win service, as anything else, is the right tool for some jobs, not for all.

From a conceptual point of view: there are occasions when it just makes no sense to have the "service" in question run when the web application is not...

Form a technical point of view: first, there is absolutely no technical issue of any kind with this approach (which is indeed canonical); second, let's not forget that this is the only option in most shared hosting scenarios.

-LV

Answer 11

User-23818800 posted

Yes shared hosting will be the issue, not now, but in the future, so it needs to be right from the start

 

so basically, if the app pool resets, then the application will stop running until someone requests a page from the site,

then the site will start, and the polling will start again

is this the summary of the problem? or have i got it wrong?
if so, what solutions are there for starting the app (remember, i want to try and house this in one application)

 

thanks

Answer 12

User-952121411 posted

so basically, if the app pool resets, then the application will stop running until someone requests a page from the site,

then the site will start, and the polling will start again

I believe that once the application recycling occurs, the application will continue to run (calling Application_Start) and does not require a user to start it again.  However, Session state will be lost as I understand, and the behavior of the timers will have to be monitored to make sure that the process is seamless.

I do agree that if the site is hosted, that a Windows Service is not feasible and this design may be the primary option.  I will let others chime in as to the low level specifics of this design because as I mentioned before I have not set up a service in this manner using IIS; they might be able to speak to these details better.

If the questions get very IIS specific and can not be answered here you may also want to re-post or check out the IIS forums.  They know all the ins and outs of how IIS performs in relation to a service/app of this type.

IIS Forums:

http://forums.iis.net/

As a side note, I updated the 2nd link in my last post, as it was incorrect. 

(http://bytes.com/topic/asp-net/answers/661317-keep-iis-alive)

Answer 13

User437720957 posted

You can use an external polling service (such as webcron.org) to trigger the update, or you can use it just to keep the HttpApplication alive, making the Timer approach a bit more viable.

Answer 14

User-1199946673 posted

It changes more than every min, but want to reduce the server load

 

Even if the data changes more than every minute, is it important that you keep the data up to date, even if you don't have visitors on your site? Why don't you retrieve the data at the moment a request is made to the page(s) that show this data, showing the most up to date data, and only retrieve the data when it is needed?

Answer 15

User-389939489 posted

so basically, if the app pool resets, then the application will stop running until someone requests a page from the site, then the site will start, and the polling will start again

is this the summary of the problem? or have i got it wrong?

Please check the MSDN about Application Pool Settings:

http://msdn.microsoft.com/en-us/library/aa719533(VS.71).aspx

You are here particularly interested in Recycling, Performance, and Health Application Pool Settings: you can control when and how worker processes recycle and much more.

Few more considerations:

atconway:
> However, Session state will be lost as I understand

I don't know much about it, but there are even options for "web gardening".

atconway:
> and the behavior of the timers will have to be monitored to make sure that the process is seamless.

There is no more need to monitor the behaviour of the timers than there is to monitor that your web app is running... or your win service! In fact, keep in mind that the scheduling component itself is the same, that you control it from a win service or from a web application. The only point in question is about the different life-cycles of a win service compared to a web application, not about the timers and how reliable they are, as you need timers in both cases.

gunteman:
> You can use an external polling service (such as webcron.org) to trigger the update, or you can use it just to keep the HttpApplication alive, making the Timer approach a bit more viable.

Unless I am still missing something, you just don't need any polling service with an Asp.Net application (Fwk 2 at least).

hans_v:
> Even if the data changes more than every minute, is it important that you keep the data up to date, even if you don't have visitors on your site.

Who can say so? To expand a bit on the conceptual side: there are occasions when you just DO NOT want your service to -say- send notifications unless the web application is running. In those occasions, running a scheduling component within the web application is just the best way to go, rather than running it in the context of a win service, and then having to check what is the state of the web application...

-LV

Answer 16

User437720957 posted

Unless I am still missing something, you just don't need any polling service with an Asp.Net application

No, I just presented it as an option. An easy one at that.

> Even if the data changes more than every minute, is it important that you keep the data up to date, even if you don't have visitors on your site.

Who can say so?

I think Hans just forgot the question mark. It was a question.

Answer 17

User-389939489 posted

gunteman:

> > Unless I am still missing something, you just don't need any polling service with an Asp.Net application

> No, I just presented it as an option. An easy one at that.

An easy one at what? Solving a problem that does not exist?

> > > Even if the data changes more than every minute, is it important that you keep the data up to date, even if you don't have visitors on your site.

> > Who can say so?

> I think Hans just forgot the question mark. It was a question.

If you say so.

-LV

Answer 18

User-952121411 posted

I don't know much about it, but there are even options for "web gardening".

 

In a 'hosted' environment which is what the original poster said would be the ultimate destination, I don't think this sort of configuration would be possible or at the hands of the developer.  Also this requires the use of an out of process location for Session like a 'State Server' or 'SQL Server' and then all Session variables must be serializable for this setup to work properly.

"Caution: If you enable Web-garden mode by setting the webGarden attribute to true in the processModel element of the application's Web.config file, do not use InProc session state mode. If you do, data loss can occur if different requests for the same session are served by different worker processes."

http://msdn.microsoft.com/en-us/library/ms178586.aspx

For the original poster - if in-memory server Session State is not used in your application then the points surrounding this feature are unimportant. 

 

Answer 19

User-389939489 posted

> > I don't know much about it, but there are even options for "web gardening".

> In a 'hosted' environment which is what the original poster said would be the ultimate destination, I don't think this sort of configuration would be possible or at the hands of the developer.

It was *you* to complain about lost session state, despite it was *you* to push for the win service, and it was *to you* that I was then replying: the OP is surely *not* interested. Reality is (just reread the thread): you don't even know what you are talking about. Still, it seems you have decided that, whatever I say, you are going to complain, even if you can't find one single sensible counter-argument. What fun.

-LV

Answer 20

User-389939489 posted

P.S. Caution here and caution there?? Rather, JUST READ THE ***MANUAL!!!

(Man, how sick I am of the situation in the software arena...)

-LV

Answer 21

User-952121411 posted

I sincerely apologize.  I didn't realize I was messing with your bread and butter design each time I disregard using an ASP.NET app as an indefinite process, as I see you have been recommending this for 3+ years:

http://forums.asp.net/p/1062969/1530812.aspx#1530812

It seems like you get the response not to tackle running a long running process like this in an ASP.NET process often, but refuse to understand this is not the best solution.  Your consistent defensive tone on this topic makes sense now; this is the method you have been using for years, and get offended if another method is brought up.  But keep on touting the Timers in a IIS ASP.NET web app as often as you like.  We are all listening to see how you defend them next...

Answer 22

User-389939489 posted

And what is an "indefinite process"? You seem to refuse you are a beginner: rather, all you have done, here and elsewhere, is making it personal (apparently your only weapon).

-LV