none
Monitoring multiple(6000) email(accounts) programatically RRS feed

  • Question

  • Hi ,

    We have decided to montor incoming and outgoing emails and save them( subject To,From Timestamp) in database.

    I have around 6000 email accounts, now I have use EWS example to monitor single mail box(using EWS-streaming)  and store them in database.

    Is it feasible to monitor 6000 mail box using EWS is there any other way to store every informaion of mail (sent and received) to store in database?

    I was also looking at transort agent but it is with disclaimer that exhchange performance may get affected.

    I will be using c# and may be sql as database. Any suggestion or solution how to achieve my goal?

    Regards,

    Amit


    • Edited by ABBhagwat Thursday, September 27, 2012 1:58 PM spelling mistake
    Thursday, September 27, 2012 1:57 PM

Answers

  • Amit,

    I'd use either OnCategorizedMessageHandler or OnRoutedMessageHandler.

    These handlers will see all messages that make it past the Edge server's SMTP receive handler (i.e. all email that is not spam) - all inbound, outbound and internal email.

    Some thing to remember / consider:

    • Your handler/s will be called for each email
    • Exchange will be processing concurrently / multiple emails will be processed at the same time. Exchange gives you a new thread for each one and (this is important) the parameters are only within scope in that handler.
    • The number of threads depends on how many cores your hub server has (although this isn't documented anywhere), with 2007 and 2010 it's 6 threads per core i.e. 8 cores = 48 threads. Not sure if that holds for 2013 yet.
    • On the plus side you don't have to be concerned about the threads management as Exchange does it all (there are cases where you might need to do your own threading but your case is fairly simple so I don't think you'll need to).
    • You can't take too long processing in an event handler, just do what you need to do and get out. If you slow down Exchange too much there are consequences: With Exchange 2007 the queue will keep on growing and the hub can go into back pressure, with Exchange 2010 the hub will stop accepting emails once all the threads are in use (and I don't know about 2013 yet).

    So in your handler get the data you need, and push it somewhere / offload it as quickly as possible and get out. That's why I suggested MSMQ (or similar) because you can just keep on queuing data to it.

    You may even want to consider just dumping the data to files locally and have another application pick-up the information and store it in a database etc. The only constraint then is disk space.

    If you are reliant on storing your data to a SQL Server (or similar) within the handler then you have to consider what happens when SQL goes off-line. As above, if you use all the threads waiting for a SQL Server (that's crashed) then Exchange will stop receiving emails or go into back pressure sooner or later. In this situation you have to decide (A) Should your handler just wait indefinitely for the SQL Server to come online again ? or (B) stop logging and allow Exchange to continue processing?

    Regards,


    Scott Quinn | C# developer & messaging specialist (for hire). Contact me at http://au.linkedin.com/in/scottquinn

    • Marked as answer by ABBhagwat Monday, October 8, 2012 4:56 AM
    Saturday, October 6, 2012 12:54 AM

All replies

  • Amit,

    Well... you could try to use EWS but for that many accounts ?! Not feasible I think.

    I think a routing agent would make more sense.

    So all you want to store is the subject, datetime, sender and recipients (and I'd add Message ID) ? That's not that much to store really and a well setup SQL (or Oracle) server should handle it easily. You could push the data via a MSMQ (or a web service) to wherever.

    You can choose whether to log in submission (before DLs are expanded) or when the email has been routed or categorised.

    Routing agents can run many threads to do the work (this is set by the number of cores in your server/s, the more cores the more threads the agent can run).

    You know if that all you want to track you could also look at the logs.

    There are also commercial applications that can do all this for you.

    Regards,


    Scott Quinn | C# developer & messaging specialist (for hire). Contact me at http://au.linkedin.com/in/scottquinn

    Thursday, September 27, 2012 3:43 PM
  • Hi Scott,

    Thanks for the information, yes I would like to track subject, date time, sender and recipients , messageID(or maybe conversation ID) and body also(not bothered about attachments).

    I have done some study of EWS and as you suggested it will be very difficult or may be impossible to track so many email accounts.

    I am in process of doing exchange server setup, I will look more into routing and Transport agent.

    In case if you have some link that shares the code of routing/transport agent please share.

    Regards,

    Amit

    Friday, September 28, 2012 12:27 PM
  • Amit,

    The Transport Agents SDK should have enough sample code for you.

    I'd look at the hub asynchronous logging sample.

    Regards,


    Scott Quinn | C# developer & messaging specialist (for hire). Contact me at http://au.linkedin.com/in/scottquinn


    • Edited by Scott Quinn Monday, October 1, 2012 4:06 AM Clarification
    Monday, October 1, 2012 4:04 AM
  • Might want to investigate using journaling to aggregate the data for you.

    [string](0..33|%{[char][int](46+("686552495351636652556262185355647068516270555358646562655775 0645570").substring(($_*2),2))})-replace " "

    Monday, October 1, 2012 4:34 AM
  • By all means investigate journaling but I can't see it being the best solution...

    You'd need to use Premium Journaling (old style mailbox journaling wouldn't really work in this situation).

    Two things to consider then are:

    • Different licensing conditions apply for Premium Journaling (i.e. it may cost you $$$)
    • You'll get the whole email - so make sure you have heaps of storage space as you're effectively storing twice as much data if you do journal for all recipients. 

    Premium Journaling is achieved using a Transport Agent (so are Transport Rules), with your own Transport Agent however you get a lot more control, you can have the agent do whatever you like with the data - use as much or as little of it as you like. All you need is the know-how and tools to write one.

    Regards,


    Scott Quinn | C# developer & messaging specialist (for hire). Contact me at http://au.linkedin.com/in/scottquinn

    Monday, October 1, 2012 6:53 AM
  • Hi Scott,

    I will look at the sample examples  my setup is not ready will explore more once got it.

    Thanks for the information.

    Regards,

    Amit

    Monday, October 1, 2012 11:47 AM
  • Hi mjolinor,

    I would also look at journaling . Thanks

    Regards,

    Amit

    Monday, October 1, 2012 11:48 AM
  • I wouldn't have considered it an option until the second post where he said he also wanted to capture the body of the email.

    Up until then, it seemed to be a fairly simple exercise in capturing header information. 


    [string](0..33|%{[char][int](46+("686552495351636652556262185355647068516270555358646562655775 0645570").substring(($_*2),2))})-replace " "

    Monday, October 1, 2012 1:07 PM
  • Hi Scott,

    I have further read the Transport Agent documentation(unfortunately not able to test my code as exchange server setup not ready) I haved decided to use the RoutingAgent , haven't decided whether to use

    OnRoutedMessageHandler or <mshelp:link keywords="b33986ab-b0ca-ee9d-bcf9-a02cd6a84de0" tabIndex="0">OnSubmittedMessage</mshelp:link> 

    As per documentation I should get all the messages(I am assuming both incoming and outgoing messages of all users) if hub server or edge server role is set.

    (RoutedMessageEventSource source, QueuedMessageEventArgs args)

    I will put arg in MSMQ and will process this queue(either on this machine or different machine) to put the required information of email like body, subject, messageid datetime stamp etc in database.

    I was just wondering if I can directly access QueuedMessageEventArgs  from different machine instead of having my own MSMQ.(I may be asking wrong questions but just thought came to my mind).

    I appreciate the guidance you have provdied.

    Regards,

    Amit

    Friday, October 5, 2012 5:38 AM
  • Amit,

    I'd use either OnCategorizedMessageHandler or OnRoutedMessageHandler.

    These handlers will see all messages that make it past the Edge server's SMTP receive handler (i.e. all email that is not spam) - all inbound, outbound and internal email.

    Some thing to remember / consider:

    • Your handler/s will be called for each email
    • Exchange will be processing concurrently / multiple emails will be processed at the same time. Exchange gives you a new thread for each one and (this is important) the parameters are only within scope in that handler.
    • The number of threads depends on how many cores your hub server has (although this isn't documented anywhere), with 2007 and 2010 it's 6 threads per core i.e. 8 cores = 48 threads. Not sure if that holds for 2013 yet.
    • On the plus side you don't have to be concerned about the threads management as Exchange does it all (there are cases where you might need to do your own threading but your case is fairly simple so I don't think you'll need to).
    • You can't take too long processing in an event handler, just do what you need to do and get out. If you slow down Exchange too much there are consequences: With Exchange 2007 the queue will keep on growing and the hub can go into back pressure, with Exchange 2010 the hub will stop accepting emails once all the threads are in use (and I don't know about 2013 yet).

    So in your handler get the data you need, and push it somewhere / offload it as quickly as possible and get out. That's why I suggested MSMQ (or similar) because you can just keep on queuing data to it.

    You may even want to consider just dumping the data to files locally and have another application pick-up the information and store it in a database etc. The only constraint then is disk space.

    If you are reliant on storing your data to a SQL Server (or similar) within the handler then you have to consider what happens when SQL goes off-line. As above, if you use all the threads waiting for a SQL Server (that's crashed) then Exchange will stop receiving emails or go into back pressure sooner or later. In this situation you have to decide (A) Should your handler just wait indefinitely for the SQL Server to come online again ? or (B) stop logging and allow Exchange to continue processing?

    Regards,


    Scott Quinn | C# developer & messaging specialist (for hire). Contact me at http://au.linkedin.com/in/scottquinn

    • Marked as answer by ABBhagwat Monday, October 8, 2012 4:56 AM
    Saturday, October 6, 2012 12:54 AM
  • Hi Scott,

    Thanks for the detailied explaination. I will take care of all the things you mentioned in my code.

    I will put the messages in MSMQ and will process it in another application. Also I will take care that if queue size grows beyond some limit will send notification(as this scenario will appear when cosumer thread has stopped consuming) and if database is not responding in exception handling will put that in file.

    I haven't got the setup yet but now quite clear about the approach.

    Thanks for all help.

    Regards,

    Amit

    Monday, October 8, 2012 5:00 AM