none
Extended properties and Exchange search indexing RRS feed

  • Question

  • Hello,

    I read through the list of properties indexed by Exchange search and from my research it seems that extended properties are not able to be indexed for Exchange searches.  Is this actually the case or am I mistaken?

    In our implementation we repeatedly do a FindItems call and as part of our search we look for an extended property that we add to our messages.  I'm wondering if anyone knows a way to improve the performance of the search with regards to the extended property.

    Thank you for your time,

    -Michael

    Wednesday, November 4, 2015 12:23 AM

Answers

  • >>We have hit a problem though.  For a few users, when we try to locate the search folder we are unable to find it.  When this happens our program tries to create the search folder, it does so and we get an error saying "the folder already exists".  So for these users the search folder clearly exists (We can also see the folder in Outlook) but we can't locate it through the EWS call.  This happens for 3 out of ~30 users, so we have plenty of successful instances of our strategy working. 

    That doesn't sounds correct to me I guess the thing to understand about Search Folders is once you do create them they aren't meant to be an instant search they populated in the background http://blogs.msdn.com/b/dgoldman/archive/2008/07/01/microsoft-exchange-and-search-folders.aspx this is a little old but still quite good https://technet.microsoft.com/en-us/library/cc535025.aspx. In the affected mailboxes are there are very large number of Items it maybe just a timing issue that you need to wait on but it really sounds to me like a logic or bug issue in your code somewhere so I'd suggest getting it reviewed (you can always confirm object creation with a MAPI editor like OutlookSpy of MFCMapi)

    To show all the SearchFolders in a Mailbox I would suggest something like this where you search from the NonIPMRoot

    function Connect-Exchange{ 
        param( 
        	[Parameter(Position=0, Mandatory=$true)] [string]$MailboxName,
    		[Parameter(Position=1, Mandatory=$true)] [System.Management.Automation.PSCredential]$Credentials,
    		[Parameter(Position=2, Mandatory=$false)] [string]$url
        )  
     	Begin
    		 {
    		Load-EWSManagedAPI
    		
    		## Set Exchange Version  
    		$ExchangeVersion = [Microsoft.Exchange.WebServices.Data.ExchangeVersion]::Exchange2010_SP2
    		  
    		## Create Exchange Service Object  
    		$service = New-Object Microsoft.Exchange.WebServices.Data.ExchangeService($ExchangeVersion)  
    		  
    		## Set Credentials to use two options are availible Option1 to use explict credentials or Option 2 use the Default (logged On) credentials  
    		  
    		#Credentials Option 1 using UPN for the windows Account  
    		#$psCred = Get-Credential  
    		$creds = New-Object System.Net.NetworkCredential($Credentials.UserName.ToString(),$Credentials.GetNetworkCredential().password.ToString())  
    		$service.Credentials = $creds      
    		#Credentials Option 2  
    		#service.UseDefaultCredentials = $true  
    		 #$service.TraceEnabled = $true
    		## Choose to ignore any SSL Warning issues caused by Self Signed Certificates  
    		  
    		Handle-SSL	
    		  
    		## Set the URL of the CAS (Client Access Server) to use two options are availbe to use Autodiscover to find the CAS URL or Hardcode the CAS to use  
    		  
    		#CAS URL Option 1 Autodiscover  
    		if($url){
    			$uri=[system.URI] $url
    			$service.Url = $uri    
    		}
    		else{
    			$service.AutodiscoverUrl($MailboxName,{$true})  
    		}
    		Write-host ("Using CAS Server : " + $Service.url)   
    		   
    		#CAS URL Option 2 Hardcoded  
    		  
    		#$uri=[system.URI] "https://casservername/ews/exchange.asmx"  
    		#$service.Url = $uri    
    		  
    		## Optional section for Exchange Impersonation  
    		  
    		#$service.ImpersonatedUserId = new-object Microsoft.Exchange.WebServices.Data.ImpersonatedUserId([Microsoft.Exchange.WebServices.Data.ConnectingIdType]::SmtpAddress, $MailboxName) 
    		if(!$service.URL){
    			throw "Error connecting to EWS"
    		}
    		else
    		{		
    			return $service
    		}
    	}
    }
    
    function Load-EWSManagedAPI{
        param( 
        )  
     	Begin
    	{
    		## Load Managed API dll  
    		###CHECK FOR EWS MANAGED API, IF PRESENT IMPORT THE HIGHEST VERSION EWS DLL, ELSE EXIT
    		$EWSDLL = (($(Get-ItemProperty -ErrorAction SilentlyContinue -Path Registry::$(Get-ChildItem -ErrorAction SilentlyContinue -Path 'Registry::HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Exchange\Web Services'|Sort-Object Name -Descending| Select-Object -First 1 -ExpandProperty Name)).'Install Directory') + "Microsoft.Exchange.WebServices.dll")
    		if (Test-Path $EWSDLL)
    		    {
    		    Import-Module $EWSDLL
    		    }
    		else
    		    {
    		    "$(get-date -format yyyyMMddHHmmss):"
    		    "This script requires the EWS Managed API 1.2 or later."
    		    "Please download and install the current version of the EWS Managed API from"
    		    "http://go.microsoft.com/fwlink/?LinkId=255472"
    		    ""
    		    "Exiting Script."
    		    exit
    		    } 
      	}
    }
    
    function Handle-SSL{
        param( 
        )  
     	Begin
    	{
    		## Code From http://poshcode.org/624
    		## Create a compilation environment
    		$Provider=New-Object Microsoft.CSharp.CSharpCodeProvider
    		$Compiler=$Provider.CreateCompiler()
    		$Params=New-Object System.CodeDom.Compiler.CompilerParameters
    		$Params.GenerateExecutable=$False
    		$Params.GenerateInMemory=$True
    		$Params.IncludeDebugInformation=$False
    		$Params.ReferencedAssemblies.Add("System.DLL") | Out-Null
    
    $TASource=@'
      namespace Local.ToolkitExtensions.Net.CertificatePolicy{
        public class TrustAll : System.Net.ICertificatePolicy {
          public TrustAll() { 
          }
          public bool CheckValidationResult(System.Net.ServicePoint sp,
            System.Security.Cryptography.X509Certificates.X509Certificate cert, 
            System.Net.WebRequest req, int problem) {
            return true;
          }
        }
      }
    '@ 
    		$TAResults=$Provider.CompileAssemblyFromSource($Params,$TASource)
    		$TAAssembly=$TAResults.CompiledAssembly
    
    		## We now create an instance of the TrustAll and attach it to the ServicePointManager
    		$TrustAll=$TAAssembly.CreateInstance("Local.ToolkitExtensions.Net.CertificatePolicy.TrustAll")
    		[System.Net.ServicePointManager]::CertificatePolicy=$TrustAll
    
    		## end code from http://poshcode.org/624
    
    	}
    }
    
    function Enum-SearchFolders{ 
        param( 
    		[Parameter(Position=0, Mandatory=$true)] [Microsoft.Exchange.WebServices.Data.ExchangeService]$service,
    		[Parameter(Position=2, Mandatory=$false)] [Microsoft.Exchange.WebServices.Data.FolderId]$FolderId
        )  
     	Begin
    	{
    		#Define Extended properties  
    		$psPropset= new-object Microsoft.Exchange.WebServices.Data.PropertySet([Microsoft.Exchange.WebServices.Data.BasePropertySet]::FirstClassProperties)  
    		$PR_Folder_Path = new-object Microsoft.Exchange.WebServices.Data.ExtendedPropertyDefinition(26293, [Microsoft.Exchange.WebServices.Data.MapiPropertyType]::String);				
    		$psPropset.Add($PR_Folder_Path)
    		$Folders = @()
    		$PR_FOLDER_TYPE = new-object Microsoft.Exchange.WebServices.Data.ExtendedPropertyDefinition(13825,[Microsoft.Exchange.WebServices.Data.MapiPropertyType]::Integer);  
    		#Define the FolderView used for Export should not be any larger then 1000 folders due to throttling  
    		$fvFolderView =  New-Object Microsoft.Exchange.WebServices.Data.FolderView(1000)  
    		#Deep Transval will ensure all folders in the search path are returned  
    		$fvFolderView.Traversal = [Microsoft.Exchange.WebServices.Data.FolderTraversal]::Deep;  
         	#Add Properties to the  Property Set  
    		$fvFolderView.PropertySet = $psPropset;  
    		#The Search filter will exclude any Search Folders  
    		$sfSearchFilter = new-object Microsoft.Exchange.WebServices.Data.SearchFilter+IsEqualTo($PR_FOLDER_TYPE,"2")  
    		$fiResult = $null  
    		#The Do loop will handle any paging that is required if there are more the 1000 folders in a mailbox  
    		do {  
    		    $fiResult = $Service.FindFolders($folderId,$sfSearchFilter,$fvFolderView)  
    		    foreach($ffFolder in $fiResult.Folders){  
    		        $foldpathval = $null  
    		        #Try to get the FolderPath Value and then covert it to a usable String   
    		        if ($ffFolder.TryGetProperty($PR_Folder_Path,[ref] $foldpathval))  
    		        {  
    		            $binarry = [Text.Encoding]::UTF8.GetBytes($foldpathval)  
    		            $hexArr = $binarry | ForEach-Object { $_.ToString("X2") }  
    		            $hexString = $hexArr -join ''  
    		            $hexString = $hexString.Replace("FEFF", "5C00")  
    		            $fpath = ConvertToString($hexString)  
    		        }  
    				$ffFolder | Add-Member -Name "FolderPath" -Value $fpath -MemberType NoteProperty
    				$Folders += $ffFolder
    		    } 
    		    $fvFolderView.Offset += $fiResult.Folders.Count
    		}while($fiResult.MoreAvailable -eq $true)  
    		return $Folders	
    	}
    }
    
    function Get-SearchFolders{
    
        param( 
        	[Parameter(Position=0, Mandatory=$true)] [string]$MailboxName,
    		[Parameter(Position=1, Mandatory=$true)] [System.Management.Automation.PSCredential]$Credentials
        )  
     	Begin
    	{
    		$service = Connect-Exchange -MailboxName $MailboxName -Credentials $Credentials
    		# Bind to the NON_IPM_ROOT Root folder  
    		$folderid= new-object Microsoft.Exchange.WebServices.Data.FolderId([Microsoft.Exchange.WebServices.Data.WellKnownFolderName]::Root,$MailboxName)   
    		$folders = Enum-SearchFolders -service $service -FolderId $folderid
    		foreach($folder in $Folders){
    			Write-Host ($folder.FolderPath)
    		}
    	}
    
    }

    the run Get-SearchFolders -MailboxName mailbox@domain.com

    Cheers
    Glen

    • Marked as answer by MikeHix Monday, November 16, 2015 4:57 PM
    Thursday, November 12, 2015 3:06 AM

All replies

  • Extended properties aren't indexed and the indexes themselves aren't configurable, What sort of property and how large is it that you searching for eg are you searches mainly static or does the value of the property vary greatly across a number of items.

    Some things you can do is use a Search Folder if you queries are mainly static or consider using an indexed property (eg if your property is a simple string then maybe a category will work). Are you searching on a number of constraints ? if that the case you may find dropping the non indexed constrainsts and just filtering them client side will yield better performance.

    Cheers
    Glen

    Wednesday, November 4, 2015 3:55 PM
  • Thanks for the response Glen, I'm looking at using a search folder as you suggested because our query is static.  So far it looks promising, but I've run into some odd behavior when I do a call like:

    FindFoldersResults findResults = exchSvc.FindFolders(WellKnownFolderName.SearchFolders, folderView);

    on one user's mailbox the results showed 3 search folders, one of which is named "Unread Mail".  When I queried another user who also has an "Unread Mail" search folder, the unread mail folder did not show up in the results.  So the results seem inconsistent, I would expect the same search folder to show up for each user that has it.

    Additionally, we created a new search folder as a test, using one of the preset search folder options "Large Mail".  When I did another FindFolders call the new search folder did not show up in the list of results.

    If you or anyone has any input on why I'm seeing either of the above behaviors I would much appreciate it.Thanks for your time,

    -Michael

    Thursday, November 5, 2015 10:12 PM
  • >> on one user's mailbox the results showed 3 search folders, one of which is named "Unread Mail".  When I queried another user who also has an "Unread Mail" search folder, the unread mail folder did not show up in the results.  So the results seem inconsistent, I would expect the same search folder to show up for each user that has it.

    Search folders are typically created by the clients (eg Outlook, OWA etc) so depending if user has ever connected to the Mailbox using Outlook certain search folder may or maynot exist. You can create your own search folder which maybe a better idea if you need to ensure it will be available.

    >>Additionally, we created a new search folder as a test, using one of the preset search folder options "Large Mail".  When I did another FindFolders call the new search folder did not show up in the list of results.

    Where did you save the new search Folder to ? , typically search folders are located in the finder non_ipm root so if you did a findFolder on the MsgFolderRoot it wouldn't include that location.

    Cheers
    Glen

    Monday, November 9, 2015 3:17 AM
  • So based on previous questions/answers and more research we have developed a solution as follows:

    We create a search folder through EWS managed API, filtering on our specific message type.

    We then subscribe to StreamingNotification events on that search folder.

    This seems to work for most users just fine, the search folder's contents match what we expect and we are able to use the EWS managed API to find and subscribe to the search folder for events.  The subscription works fine, sending events only when we modify items that are in the search folder.

    We have hit a problem though.  For a few users, when we try to locate the search folder we are unable to find it.  When this happens our program tries to create the search folder, it does so and we get an error saying "the folder already exists".  So for these users the search folder clearly exists (We can also see the folder in Outlook) but we can't locate it through the EWS call.  This happens for 3 out of ~30 users, so we have plenty of successful instances of our strategy working. 

    I save the search folder as such: "searchFolder.Save(WellKnownFolderName.SearchFolders);"

    Any idea why we would be unable to locate the search folder?  Or if there is a way to verify the folder's existence/contents in powershell?

    Also any more in-depth articles on how search folders work would be much appreciated.

    Thank you,

    -Michael


    • Edited by MikeHix Wednesday, November 11, 2015 7:37 PM
    Wednesday, November 11, 2015 7:10 PM
  • >>We have hit a problem though.  For a few users, when we try to locate the search folder we are unable to find it.  When this happens our program tries to create the search folder, it does so and we get an error saying "the folder already exists".  So for these users the search folder clearly exists (We can also see the folder in Outlook) but we can't locate it through the EWS call.  This happens for 3 out of ~30 users, so we have plenty of successful instances of our strategy working. 

    That doesn't sounds correct to me I guess the thing to understand about Search Folders is once you do create them they aren't meant to be an instant search they populated in the background http://blogs.msdn.com/b/dgoldman/archive/2008/07/01/microsoft-exchange-and-search-folders.aspx this is a little old but still quite good https://technet.microsoft.com/en-us/library/cc535025.aspx. In the affected mailboxes are there are very large number of Items it maybe just a timing issue that you need to wait on but it really sounds to me like a logic or bug issue in your code somewhere so I'd suggest getting it reviewed (you can always confirm object creation with a MAPI editor like OutlookSpy of MFCMapi)

    To show all the SearchFolders in a Mailbox I would suggest something like this where you search from the NonIPMRoot

    function Connect-Exchange{ 
        param( 
        	[Parameter(Position=0, Mandatory=$true)] [string]$MailboxName,
    		[Parameter(Position=1, Mandatory=$true)] [System.Management.Automation.PSCredential]$Credentials,
    		[Parameter(Position=2, Mandatory=$false)] [string]$url
        )  
     	Begin
    		 {
    		Load-EWSManagedAPI
    		
    		## Set Exchange Version  
    		$ExchangeVersion = [Microsoft.Exchange.WebServices.Data.ExchangeVersion]::Exchange2010_SP2
    		  
    		## Create Exchange Service Object  
    		$service = New-Object Microsoft.Exchange.WebServices.Data.ExchangeService($ExchangeVersion)  
    		  
    		## Set Credentials to use two options are availible Option1 to use explict credentials or Option 2 use the Default (logged On) credentials  
    		  
    		#Credentials Option 1 using UPN for the windows Account  
    		#$psCred = Get-Credential  
    		$creds = New-Object System.Net.NetworkCredential($Credentials.UserName.ToString(),$Credentials.GetNetworkCredential().password.ToString())  
    		$service.Credentials = $creds      
    		#Credentials Option 2  
    		#service.UseDefaultCredentials = $true  
    		 #$service.TraceEnabled = $true
    		## Choose to ignore any SSL Warning issues caused by Self Signed Certificates  
    		  
    		Handle-SSL	
    		  
    		## Set the URL of the CAS (Client Access Server) to use two options are availbe to use Autodiscover to find the CAS URL or Hardcode the CAS to use  
    		  
    		#CAS URL Option 1 Autodiscover  
    		if($url){
    			$uri=[system.URI] $url
    			$service.Url = $uri    
    		}
    		else{
    			$service.AutodiscoverUrl($MailboxName,{$true})  
    		}
    		Write-host ("Using CAS Server : " + $Service.url)   
    		   
    		#CAS URL Option 2 Hardcoded  
    		  
    		#$uri=[system.URI] "https://casservername/ews/exchange.asmx"  
    		#$service.Url = $uri    
    		  
    		## Optional section for Exchange Impersonation  
    		  
    		#$service.ImpersonatedUserId = new-object Microsoft.Exchange.WebServices.Data.ImpersonatedUserId([Microsoft.Exchange.WebServices.Data.ConnectingIdType]::SmtpAddress, $MailboxName) 
    		if(!$service.URL){
    			throw "Error connecting to EWS"
    		}
    		else
    		{		
    			return $service
    		}
    	}
    }
    
    function Load-EWSManagedAPI{
        param( 
        )  
     	Begin
    	{
    		## Load Managed API dll  
    		###CHECK FOR EWS MANAGED API, IF PRESENT IMPORT THE HIGHEST VERSION EWS DLL, ELSE EXIT
    		$EWSDLL = (($(Get-ItemProperty -ErrorAction SilentlyContinue -Path Registry::$(Get-ChildItem -ErrorAction SilentlyContinue -Path 'Registry::HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Exchange\Web Services'|Sort-Object Name -Descending| Select-Object -First 1 -ExpandProperty Name)).'Install Directory') + "Microsoft.Exchange.WebServices.dll")
    		if (Test-Path $EWSDLL)
    		    {
    		    Import-Module $EWSDLL
    		    }
    		else
    		    {
    		    "$(get-date -format yyyyMMddHHmmss):"
    		    "This script requires the EWS Managed API 1.2 or later."
    		    "Please download and install the current version of the EWS Managed API from"
    		    "http://go.microsoft.com/fwlink/?LinkId=255472"
    		    ""
    		    "Exiting Script."
    		    exit
    		    } 
      	}
    }
    
    function Handle-SSL{
        param( 
        )  
     	Begin
    	{
    		## Code From http://poshcode.org/624
    		## Create a compilation environment
    		$Provider=New-Object Microsoft.CSharp.CSharpCodeProvider
    		$Compiler=$Provider.CreateCompiler()
    		$Params=New-Object System.CodeDom.Compiler.CompilerParameters
    		$Params.GenerateExecutable=$False
    		$Params.GenerateInMemory=$True
    		$Params.IncludeDebugInformation=$False
    		$Params.ReferencedAssemblies.Add("System.DLL") | Out-Null
    
    $TASource=@'
      namespace Local.ToolkitExtensions.Net.CertificatePolicy{
        public class TrustAll : System.Net.ICertificatePolicy {
          public TrustAll() { 
          }
          public bool CheckValidationResult(System.Net.ServicePoint sp,
            System.Security.Cryptography.X509Certificates.X509Certificate cert, 
            System.Net.WebRequest req, int problem) {
            return true;
          }
        }
      }
    '@ 
    		$TAResults=$Provider.CompileAssemblyFromSource($Params,$TASource)
    		$TAAssembly=$TAResults.CompiledAssembly
    
    		## We now create an instance of the TrustAll and attach it to the ServicePointManager
    		$TrustAll=$TAAssembly.CreateInstance("Local.ToolkitExtensions.Net.CertificatePolicy.TrustAll")
    		[System.Net.ServicePointManager]::CertificatePolicy=$TrustAll
    
    		## end code from http://poshcode.org/624
    
    	}
    }
    
    function Enum-SearchFolders{ 
        param( 
    		[Parameter(Position=0, Mandatory=$true)] [Microsoft.Exchange.WebServices.Data.ExchangeService]$service,
    		[Parameter(Position=2, Mandatory=$false)] [Microsoft.Exchange.WebServices.Data.FolderId]$FolderId
        )  
     	Begin
    	{
    		#Define Extended properties  
    		$psPropset= new-object Microsoft.Exchange.WebServices.Data.PropertySet([Microsoft.Exchange.WebServices.Data.BasePropertySet]::FirstClassProperties)  
    		$PR_Folder_Path = new-object Microsoft.Exchange.WebServices.Data.ExtendedPropertyDefinition(26293, [Microsoft.Exchange.WebServices.Data.MapiPropertyType]::String);				
    		$psPropset.Add($PR_Folder_Path)
    		$Folders = @()
    		$PR_FOLDER_TYPE = new-object Microsoft.Exchange.WebServices.Data.ExtendedPropertyDefinition(13825,[Microsoft.Exchange.WebServices.Data.MapiPropertyType]::Integer);  
    		#Define the FolderView used for Export should not be any larger then 1000 folders due to throttling  
    		$fvFolderView =  New-Object Microsoft.Exchange.WebServices.Data.FolderView(1000)  
    		#Deep Transval will ensure all folders in the search path are returned  
    		$fvFolderView.Traversal = [Microsoft.Exchange.WebServices.Data.FolderTraversal]::Deep;  
         	#Add Properties to the  Property Set  
    		$fvFolderView.PropertySet = $psPropset;  
    		#The Search filter will exclude any Search Folders  
    		$sfSearchFilter = new-object Microsoft.Exchange.WebServices.Data.SearchFilter+IsEqualTo($PR_FOLDER_TYPE,"2")  
    		$fiResult = $null  
    		#The Do loop will handle any paging that is required if there are more the 1000 folders in a mailbox  
    		do {  
    		    $fiResult = $Service.FindFolders($folderId,$sfSearchFilter,$fvFolderView)  
    		    foreach($ffFolder in $fiResult.Folders){  
    		        $foldpathval = $null  
    		        #Try to get the FolderPath Value and then covert it to a usable String   
    		        if ($ffFolder.TryGetProperty($PR_Folder_Path,[ref] $foldpathval))  
    		        {  
    		            $binarry = [Text.Encoding]::UTF8.GetBytes($foldpathval)  
    		            $hexArr = $binarry | ForEach-Object { $_.ToString("X2") }  
    		            $hexString = $hexArr -join ''  
    		            $hexString = $hexString.Replace("FEFF", "5C00")  
    		            $fpath = ConvertToString($hexString)  
    		        }  
    				$ffFolder | Add-Member -Name "FolderPath" -Value $fpath -MemberType NoteProperty
    				$Folders += $ffFolder
    		    } 
    		    $fvFolderView.Offset += $fiResult.Folders.Count
    		}while($fiResult.MoreAvailable -eq $true)  
    		return $Folders	
    	}
    }
    
    function Get-SearchFolders{
    
        param( 
        	[Parameter(Position=0, Mandatory=$true)] [string]$MailboxName,
    		[Parameter(Position=1, Mandatory=$true)] [System.Management.Automation.PSCredential]$Credentials
        )  
     	Begin
    	{
    		$service = Connect-Exchange -MailboxName $MailboxName -Credentials $Credentials
    		# Bind to the NON_IPM_ROOT Root folder  
    		$folderid= new-object Microsoft.Exchange.WebServices.Data.FolderId([Microsoft.Exchange.WebServices.Data.WellKnownFolderName]::Root,$MailboxName)   
    		$folders = Enum-SearchFolders -service $service -FolderId $folderid
    		foreach($folder in $Folders){
    			Write-Host ($folder.FolderPath)
    		}
    	}
    
    }

    the run Get-SearchFolders -MailboxName mailbox@domain.com

    Cheers
    Glen

    • Marked as answer by MikeHix Monday, November 16, 2015 4:57 PM
    Thursday, November 12, 2015 3:06 AM
  • The script you provided was able to lead us to the answer, thank you.  The reason that we couldn't find the folder for some of the users was because my folder view for the search had a size of 10, and a few users had many search folders causing the new one that we created to not be returned by my search.

    Thank you very much for your assistance Glen.

    -Michael

    Monday, November 16, 2015 4:59 PM