none
Throttling when using impersonation RRS feed

  • Question

  • Hi,

    I am building a windows service that does a lot of read write actions to several user mailboxes with EWS. I am running against a problem with Throttling. How can I disable Throttling for my AD impersonation account? It is not a mailbox account so I can't set a Throttling policy. Can I create a mailbox for this account and then set a policy?

    Further more I ran across this article.

    http://blogs.technet.com/b/exchange/archive/2010/08/02/3410563.aspx

    There it said that when using impersonation from for example a website it runs under machine account and default hard coded policy is used. I thought EWS always runs under a AD account. I can't create a impersonation account that is a machine account?

    Tuesday, April 23, 2013 9:33 AM

All replies

  • Are you using EWS Impersonation eg http://blogs.msdn.com/b/exchangedev/archive/2009/06/15/exchange-impersonation-vs-delegate-access.aspx or are you just impersonating the Mailbox owner directly?. If your using EWS Impersonation and you have the correct updates applied http://blogs.msdn.com/b/exchangedev/archive/2012/04/03/exchange-online-ewsmaxsubscriptions-throttling-budget-calculation-has-been-updated.aspx then a copy of the mailbox owner's throttling budget should be getting charged. Short of disabling throttling for those mailboxes there isn't much you can do if your still going over budget (you may want to look at the EWS logs on the server to work out what operations are costing the most and if your using batching you may need to optomize your code a bit I would suggest having a look at http://blogs.msdn.com/b/mstehle/archive/2012/05/04/matt-on-channel9-talking-about-ews.aspx). Probably the easiest fix would be to create a Service Account with Mailbox and give it a special throttling policy and just use delegation. (But it really depends what your doing eg if your doing a lot of calendaring task then Impersonation may still be a better option for some requests).

    Cheers
    Glen

    Wednesday, April 24, 2013 8:29 AM
  • Hi ,

    I am using EWS impersonation (I think). What do you mean with impersonate mailbox directly? With my service account I access 15 mailboxes. I create a subscription for each mailbox and I run a SyncFolderItems every  15 minutes. Then for each item(s) I find I do a CopyItems to several subscribed users. Each of the 15 mailboxes contain more than 15000 items. So the first time I start my service with SyncFolderItems I get a lot off calls getting alle the items in the mailbox and then copying them to my users. So when I create a mailbox for my service account but do impersonate another user (1 of the 15 mailboxes). Does the policy added to this mailbox apply?

    How would I connect with delegate access to another users mailbox? I now set the impersonationId of my Exchnage connection. When I only pass network credential how do I get items for another Mailbox?

    Regards

    Danny


    Wednesday, April 24, 2013 10:29 AM
  • If your going to use Delegation the first step is to make sure you grant you service account full access to all the mailboxes in the process using Add-mailboxpermissions

    To connect to another mailbox using delegation just use Mailbox overload in the FolderId to specify the Mailbox you want to connect to eg

                FolderId fiFolderid = new FolderId(WellKnownFolderName.Inbox, "user@domain5.onmicrosoft.com");
                Folder Inbox = Folder.Bind(service, fiFolderid);

    What size are the batches your using with CopyItems ? From what your talking about I'm surprised your having issues with throttling but if your batches are too large then that maybe your issue. (or are you using multiple threads ? and simulate nous operations)

    Cheers
    Glen

    Thursday, April 25, 2013 6:08 AM
  • Hi Glen, I have 15 threads running for each of my source mailboxes. 

    This is the flow I use:

    1. I create a new Exchange connection with my service account as network credential and my SourceMailBox as impersonation id

    2. I call the SyncFolderItems with a max of 100 items

    3. I retrieve the users subscribed to the sourcemailbox. In my test case only 1

    4. I do a for each for the users. For every user I create a new Exchange connection with impersonation id of the target user. 

    5. I do a call to find Folder with new connaction to get target folder in user mailbox

    6. With the 100 items found in step 2 I create a SearchFilter Collection ( 100 is the max after that I get to many filters why??)

    7. With connection created for target user in step 4 I do a findItems to check which items are already present

    8. With sub selection of 100 items I do a copy items

    I am testing with 2 sourcemailboxes each containing 4000 items.

     So the SyncFolderItems runs 40 times * 2 mailboxes.

    Then 40 * 2 calls to GetFolder

    After that 40 * 2 calls to FindItems and for last 40*2 calls to copy items.

    In eventlog I get message that my service account has exceed the throttle policy with 219 times in 1 minute for a maximum (setting) of 100

    This is my class. The calls to Create connection and GetFolder Are in another class

    using System;
    using System.Linq;
    using System.Threading;
    using Microsoft.Exchange.WebServices.Data;
    using System.Collections.Generic;
    using log4net;
    using ExchangeFolderSync.Data;
    
    namespace ExchangeFolderSync.Processing
    {
        public class SyncController
        {
            #region members
    
            public static volatile bool _Run = true;
            private bool _LocalRun = true;
            private ExchangeService _ExchangeService;
            private Thread _BackroundSyncThread;
            private static StreamingSubscription _StreamingSubscription;
            private StreamingSubscriptionConnection _StreamingSubscriptionConnection;
            private ILog _Log;
            private int profileId;
            private string profileSmtpAddress;
            private volatile string profileLastSyncState;
            private ExtendedPropertyDefinition extendedPropertyDefinition = new ExtendedPropertyDefinition(new Guid("{C11FF724-AA03-4555-9952-8FA248A11C3E}"), "originalid", MapiPropertyType.String);
    
            #endregion
    
            #region public methods
    
            public void RunSubscribeToStreaming(int profileId)
            {
                this.profileId = profileId;
    
                //Create log instance
                _Log = LogManager.GetLogger((this.GetType()));
                var currentProfile = DataManagement.GetProfile(profileId);
                this.profileLastSyncState = currentProfile.LastSyncState;
                this.profileSmtpAddress = currentProfile.MailBoxAddress;
    
                if (currentProfile != null)
                {
                    //Create impersonated Exchange connection
                    _ExchangeService = ExchangeEwsHelper.CreateExchangeConnection(currentProfile.MailBoxAddress);
    
                    // Process all items in the folder on a background-thread.
                    _BackroundSyncThread = new Thread(SynchronizeChangesPeriodically);
                    _BackroundSyncThread.Start();
    
                    CreateSubscription(_ExchangeService, new FolderId[] { new FolderId(currentProfile.SourceFolderId) });
                    _Log.InfoFormat("Subscription created for profile: {0}", currentProfile.MailBoxAddress);
                }
            }
    
            #endregion
    
            #region private methods
    
            private StreamingSubscriptionConnection CreateStreamingSubscription(ExchangeService service, StreamingSubscription subscription)
            {
                var connection = new StreamingSubscriptionConnection(service, 30);
                connection.AddSubscription(subscription);
                connection.OnNotificationEvent += OnNotificationEvent;
                connection.OnSubscriptionError += OnSubscriptionError;
                connection.OnDisconnect += OnDisconnect;
                connection.Open();
                return connection;
            }
    
            private void SynchronizeChangesPeriodically()
            {
                while (_Run && _LocalRun)
                {
                    try
                    {
                        // Get all changes from the server and process them according to the business rules.
                        var currentProfile = DataManagement.GetProfile(profileId);
                        if (currentProfile != null)
                        {
                            SynchronizeChanges(new FolderId(currentProfile.SourceFolderId), currentProfile.MailBoxAddress, currentProfile.CutOff);
                        }
                    }
                    catch (Exception ex)
                    {
                        _Log.InfoFormat("Failed to synchronize items for profile: {0}  Error: {1}", profileSmtpAddress, ex.Message);
                    }
                    // Since the SyncFolderItems operation is a rather expensive operation, only do this every 10 minutes
                    Thread.Sleep(TimeSpan.FromMinutes(15));
    
                    //See if profile has been changed by management tool
                    CheckProfileChanged();
                }
    
                StopSynchronizeChanges();
            }
    
            private void SynchronizeChanges(FolderId folderId, string profileSmtpAddress, int profileCutOff)
            {
                bool moreChangesAvailable;
                do
                {
                    _Log.InfoFormat("Running SyncFolderItems for profile: {0}", profileSmtpAddress);
                    // Get all changes since the last call. The synchronization cookie is stored in the _SynchronizationState field.                                
                    var changes = _ExchangeService.SyncFolderItems(folderId, new PropertySet(BasePropertySet.IdOnly, ItemSchema.DateTimeReceived), null, 100, SyncFolderItemsScope.NormalItems, profileLastSyncState);
    
                    if (changes.Count > 0)
                    {
                        var newMails = changes.Where(item => item.Item != null && item.Item.DateTimeReceived >= DateTime.Now.AddDays(0 - profileCutOff)).
                            Select(i => i.ItemId);
    
                        if (newMails.Count() > 0)
                        {
                            ProcessEmailItems(newMails, true);
                        }
                    }
    
                    // Update the synchronization cookie                
                    profileLastSyncState = changes.SyncState;
                    DataManagement.UpdateProfileLastSyncState(profileId, profileLastSyncState);
    
                    // If more changes are available, issue additional SyncFolderItems requests.
                    moreChangesAvailable = changes.MoreChangesAvailable;
                } while (moreChangesAvailable);
            }
    
            private void CreateSubscription(ExchangeService service, FolderId[] folders)
            {
                // Create a new subscription            
                _StreamingSubscription = service.SubscribeToStreamingNotifications(folders, EventType.NewMail);
                // Create new streaming notification conection            
                _StreamingSubscriptionConnection = CreateStreamingSubscription(service, _StreamingSubscription);
            }
    
            private void StopSynchronizeChanges()
            {
                _Log.InfoFormat("Synchronizing is stopping for profile: {0}", profileSmtpAddress);
                // Close the connection
                CloseStreamingSubscriptionConnection();
                // Finally, unsubscribe from the Exchange server
                UnsubscribeStreamingSubscription();
            }
    
            private void CloseStreamingSubscriptionConnection()
            {
                if (_StreamingSubscriptionConnection != null && _StreamingSubscriptionConnection.IsOpen)
                {
                    _StreamingSubscriptionConnection.Close();
                    _StreamingSubscriptionConnection.RemoveSubscription(_StreamingSubscription);
                    _StreamingSubscriptionConnection.Dispose();
                    _Log.InfoFormat("Synchronizing connection closed for profile: {0}", profileSmtpAddress);
                }
            }
    
            private void UnsubscribeStreamingSubscription()
            {
                if (_StreamingSubscription != null)
                {
                    _StreamingSubscription.Unsubscribe();
                    _Log.InfoFormat("Synchronizing subscription unsubscribed for profile: {0}", profileSmtpAddress);
                }
            }
    
            private void CheckProfileChanged()
            {
                var oldSourceMailBoxEmail = profileSmtpAddress;
                var currentProfile = DataManagement.GetProfile(profileId);
    
                if (currentProfile == null)
                {
                    _LocalRun = false;
                }
                else
                {
                    if (!oldSourceMailBoxEmail.Equals(currentProfile.MailBoxAddress))
                    {
                        _Log.InfoFormat("Profile has changed. Resetting subscription and connection.");
                        this.profileSmtpAddress = currentProfile.MailBoxAddress;
                        this.profileLastSyncState = currentProfile.LastSyncState;
                        // Close the connection
                        CloseStreamingSubscriptionConnection();
                        // Finally, unsubscribe from the Exchange server
                        UnsubscribeStreamingSubscription();
                        //new impersonated connection with new smtp address
                        _ExchangeService = ExchangeEwsHelper.CreateExchangeConnection(currentProfile.MailBoxAddress);
                        //new subscription
                        CreateSubscription(_ExchangeService, new FolderId[] { new FolderId(currentProfile.SourceFolderId) });
                    }
                }
            }
    
            private void ProcessEmailItems(IEnumerable<ItemId> mailItems, bool shouldDisableUser)
            {
                var currentProfile = DataManagement.GetProfile(profileId);
    
                if (currentProfile != null)
                {
                    _Log.DebugFormat("Setting originalid extended property for items to be copied");
                    var bindResponse = _ExchangeService.BindToItems(mailItems, new PropertySet(BasePropertySet.IdOnly, extendedPropertyDefinition));
                    var items = bindResponse.Select(res => res.Item);
                    var searchFilterCollection = new SearchFilter.SearchFilterCollection(LogicalOperator.Or);
    
                    _Log.DebugFormat("Setting searchfilter for items to be copied");
    
                    foreach (var item in items)
                    {
                        if (item.ExtendedProperties.FirstOrDefault(ep => ep.PropertyDefinition.PropertySetId == extendedPropertyDefinition.PropertySetId) == null)
                        {
                            item.SetExtendedProperty(extendedPropertyDefinition, item.Id.UniqueId);
                            item.Update(ConflictResolutionMode.AutoResolve);
                        }
    
                        searchFilterCollection.Add(new SearchFilter.IsEqualTo(extendedPropertyDefinition, item.Id.UniqueId));
                    }
    
                    _Log.DebugFormat("Starting copy to users");
                    foreach (var user in currentProfile.ProfileUsers.Where(user => user.Enabled))
                    {
                        try
                        {
                            CopyItemsForUser(mailItems, currentProfile, user, searchFilterCollection, shouldDisableUser);
                        }
                        catch (Exception ex)
                        {
                            _Log.ErrorFormat("Copy items failed from profile {0} to user {1} Error: {2}", profileSmtpAddress, user.UserMailboxAddress, ex.Message);
                        }
                    }
                }
            }
    
            private void CopyItemsForUser(IEnumerable<ItemId> mailItems, Profile currentProfile, ProfileUser user, SearchFilter.SearchFilterCollection searchFilterCollection, bool shouldDisableUser)
            {
                _Log.DebugFormat("Getting or creating folder for user {0}", user.UserMailboxAddress);
                var destinationFolder = ExchangeEwsHelper.GetUserDestinationFolder(user.UserMailboxAddress, user.UserFolderId, currentProfile.DestinationFolder);
    
                var exchangeService = ExchangeEwsHelper.CreateExchangeConnection(user.UserMailboxAddress);
                var view = new ItemView(int.MaxValue) { PropertySet = new PropertySet(BasePropertySet.IdOnly, ItemSchema.Subject, extendedPropertyDefinition) };
    
                _Log.DebugFormat("Checking items already present for user {0}", user.UserMailboxAddress);
                var findResult = exchangeService.FindItems(destinationFolder.Id, searchFilterCollection, view);
    
                if (findResult.TotalCount == 0)
                {
                    CopyItems(exchangeService, mailItems, destinationFolder, user, shouldDisableUser);
                }
                else
                {
                    var filtered = findResult.Select(k => k.ExtendedProperties.FirstOrDefault(p => p.PropertyDefinition.Name.ToLower() == "originalid").Value.ToString());
                    var toCopyItems = mailItems.Where(x => !filtered.Contains(x.UniqueId));
    
                    if (toCopyItems.Count() > 0)
                    {
                        CopyItems(exchangeService, toCopyItems, destinationFolder, user, shouldDisableUser);
                    }
                }
            }
    
            private void CopyItems(ExchangeService exchangeService, IEnumerable<ItemId> toCopyItems, Folder destinationFolder, ProfileUser user, bool shouldDisableUser)
            {
                _Log.DebugFormat("Starting copy items for user {0}", user.UserMailboxAddress);
                var response = exchangeService.CopyItems(toCopyItems, destinationFolder.Id);
    
                if (response.OverallResult == ServiceResult.Success)
                {
                    _Log.InfoFormat("Copy items from profile {0} to user {1} completed", profileSmtpAddress, user.UserMailboxAddress);
                }
                else
                {
                    var error = response.FirstOrDefault();
                    _Log.ErrorFormat("Copy items failed from profile {0} to user {1} ErrorCode: {2} ErrorMessage: {3}", profileSmtpAddress, user.UserMailboxAddress, error.ErrorCode, error.ErrorMessage);
                    if (shouldDisableUser)
                    {
                        DataManagement.EnabledProfileUser(user.Id, false);
                        _Log.WarnFormat("User {0} disabled for profile {1}", user.UserMailboxAddress, profileSmtpAddress);
                    }
                }
            }
    
            #endregion
    
            #region eventhandlers
    
            private void OnDisconnect(object sender, SubscriptionErrorEventArgs args)
            {
                if (_Run && _LocalRun)
                {
                    // Cast the sender as a StreamingSubscriptionConnection object.           
                    var connection = (StreamingSubscriptionConnection)sender;
                    connection.Open();
                    _Log.Info("Connection has been reopend.");
                }
            }
    
            private void OnNotificationEvent(object sender, NotificationEventArgs args)
            {
                // Extract the item ids for all NewMail Events in the list.
                var newMails = from e in args.Events.OfType<ItemEvent>()
                               where e.EventType == EventType.NewMail
                               select e.ItemId;
    
                if (newMails.Count() > 0)
                {
                    ProcessEmailItems(newMails, false);
                }
            }
    
            private void OnSubscriptionError(object sender, SubscriptionErrorEventArgs args)
            {
                // Handle error conditions. 
                var e = args.Exception;
                _Log.Error("The following error occured:", e);
            }
    
            #endregion
        }
    }



    Thursday, April 25, 2013 1:29 PM
  • I forgot to mention that I add a originalid extended prop to the mailmessages I copy so that I can check if an item has been copied before. I do this because items copied by the subscription event also show up in SyncFolderItems. Also my admin user can do a manual sync for all items if there where errors. So I need to verify if a message is already present and copied before. Are there other existing fields that I can use to compare messages?

    I get throttle error when running 2 mailbox threads with 1 target user each.

    When I use 1 mailbox with 1 target user at a time no throttle error.

    Thursday, April 25, 2013 1:41 PM
  • >> I have 15 threads running for each of my source mailboxes.

    That would be a red flag for the default throttling policy because its would limit you to 10 concurrent connections

    >> 6. With the 100 items found in step 2 I create a SearchFilter Collection ( 100 is the max after that I get to many filters why??)

    ??? This doesn't sound correct or something you would want to do, if your trying to use a SearchFilter collection this way your just creating a very inefficient query that's going to cause a lot of Load and I would say this would be what's causing the throttle to be exceeded. 

    Cheers
    Glen

    Friday, April 26, 2013 11:25 AM
  • Can I change the policy so I can get more concurrent connections?

    How can I do this?

    I can't set a policy for my service account because it has no mailbox.

    For the searchfiler I do a query with the 100 items but it is 1 call. So this should not cause the problem I think.

    The call also takes a minimum amount of time. Less than 1 second.

    If I pass more that 125 items I get error to many restrictions. Why is this?

    is there another way I can check if an item I am going to copy is already present in the mailbox of the user?

    Friday, April 26, 2013 11:43 AM
  • To adjust the Throttling policies you need to use the Exchange Management Shell cmdlets http://technet.microsoft.com/en-us/library/dd298094%28v=exchg.150%29.aspx

    >If I pass more that 125 items I get error to many restrictions. Why is this?

    There is a limit to how big a restriction can be, like any database query loading up the where predicates is generally a bad idea.

    >> is there another way I can check if an item I am going to copy is already present in the mailbox of the user?

    If its time based eg messages received today, you can probably grab the messages you received today in one call, index those and test against the index. Instead of telling the server to scan every message in the folder for every message you want to test.

    Cheers
    Glen

    Friday, April 26, 2013 11:51 PM
  • 1 more question about delegation instead of impersonantion. When you delegate do you create the connection with the user you want to get items from? In this way you would have to know the passwords of the users. I am only allowed to get the login information of the service account. 
    Saturday, April 27, 2013 9:44 AM
  • >>When you delegate do you create the connection with the user you want to get items from?

    No you connect with your service account, these are the only credentials you specify and this service account needs to be granted Full Access to the mailboxes you want to connec to to via add-mailboxpermission. To connect to the Delegates mailboxs folder you specify the Primary Email address in the request eg

             FolderId fiFolderid = new FolderId(WellKnownFolderName.Inbox, "user@domain5.onmicrosoft.com");
             Folder Inbox = Folder.Bind(service, fiFolderid);
    Cheers
    Glen
    Monday, April 29, 2013 6:48 AM