What would be better to sync search index data through schedule program or programmatically?

Question

Hi All,

I have gone through the documentation and came to know that We can create index and add or update data  to Search Index through rest API's this will acts as collection of documents as We can query to search Index.Off course Search Indexer crawls the data from data source and fill the data to index and then we can query to search index.

My Question is : We can create search index independently and add data to search index through rest api's this will act as data storage( similar to  table in sql), So in this case We don't require data source. Right? Please correct if i am wrong then why do we need data source in this situation.Is there any specific reason to have data source.

Another thing, There is schedule program(minimum 5 minutes interval) that indexer sync the data source to search index whenever data source updates. Here question is if frequent changes occur in data source and in every 5 mins search indexer updates the search index then will it impact the performance the query data from search index as frequent re indexing may slow down the performance.What would be better to  sync search index data through schedule program  or programmatically whenever new record is added in data source. Can you please suggest me the correct approach.

Thanks

Stick

Answer 1

Hi Stick,

You should consider leveraging the CosmosDB change feed for this purpose. You can link this to Azure Functions so that changes in Cosmos DB triggers a function which would take this feed of changes and applies it to Azure Search using the Azure Search PUSH api.  In general, the Azure Search indexer is not well suited for cases where you need to have more real time updates of content as it is a scheduled process.

As to your specific question, if there are a lot of changes happening, the indexing into Azure Search will have an impact on the performance of the search queries since all of the resources are shared, however Azure Search can index large amounts of content pretty efficiently so it may not be that much of an issue.  For this reason, it is important to leverage a change tracking mechanism in Azure Search indexer, so that it only requests the data that has been changed, rather than getting all of the data.  The only real answer to performance questions like this is to test the workload to make sure the performance meets their needs and adjust the search service tier to meet the requirements.