locked
problem with soft delete policy RRS feed

  • Question

  • Hi All,

    I have an issue with the soft delete policy. My data source is specified as

        {
          "name": "v-properties2",
          "description": "",
          "type": "azuresql",
          "credentials": {
            "connectionString": null
          },
          "container": {
            "name": "V_Properties2",
            "query": null
          },
          "dataChangeDetectionPolicy": {
            "@odata.type": "#Microsoft.Azure.Search.HighWaterMarkChangeDetectionPolicy",
            "highWaterMarkColumnName": "HighWaterMark"
          },
          "dataDeletionDetectionPolicy": {
            "@odata.type": "#Microsoft.Azure.Search.SoftDeleteColumnDeletionDetectionPolicy",
            "softDeleteColumnName": "IsDeleted",
            "softDeleteMarkerValue": "1"
          }
        }

    In my data source I have deleted and none deleted records

    select count(1) from v_properties2 where IsDeleted = 0 => 7211
    select count(1) from v_properties2 where IsDeleted = 1 => 76830

    I.e. my indexer should import 7211 records. The problem is that I keep getting the following error message:

    Indexing was stopped because the indexer quota of 10000 documents has been reached

    I.e the soft delete policy doesn't seem to work? Am I doing something wrong here?

    BR,

    Max.

    Saturday, November 28, 2015 10:53 PM

Answers

  • Hi Max, I work on Azure Search team and can hopefully help with this issue.

    This is a bit counter-intuitive, but the 10,000 document processing quota for free services applies to all documents, including those that are eventually deleted. Free search services have this quota; standard (paid) services do not.

    I would recommend the following:

    1. If possible, upgrade to a standard service to get unlimited indexer document processing quota, as well as much higher other quotas; see https://azure.microsoft.com/en-us/documentation/articles/search-limits-quotas-capacity.

    2. Set up a recurring schedule for your indexer; that way, it'll be able to make a gradual progress, one batch at a time, and should eventually process all the records in your datasource (even though you'll still see the quota error while the indexer is working through the backlog).  

    HTH,
    Eugene


    Thanks! Eugene Shvets Azure Search

    Sunday, November 29, 2015 2:48 AM
    Moderator