locked
ASA - "One item needs attention" Occasional Streaming Analytics Input Error RRS feed

  • Question

  • I have several ASA jobs that combine Event Hub data with reference data from an Azure Blob file.

    This is NOT a user/password issue like here. The jobs run fine for long periods of time.

    Once in a while they "kind-of" fail, only in that the Blob input shows the error "One item needs attention"

    Most of the data still seems to be coming in, but not those it can't reference - these are picked up by a second-part query that moves all unknown data into a separate table, for later analysis.

    I cannot paste the query without heavy redacting, however this is not an issue with the query.

    An Automation/RunJob/PowerShell script is updating the SAME lookup file each time. It is NOT creating a new one each time (ie not with yyyy/mm/dd path/pattern trickery), so just replacing the existing file.

    Could this therefore mean the ASA input "file read" is failing, because the file was locked by the PowerShell writer at the time that the ASA job needed to read it? Then it stopped trying again from that point?

    If it is a file clash issue, I'll have to rewrite, I understand that. I just want it officially confirmed that this is a genuine scenario everyone should avoid. Such a clash may cause THAT read attempt to fail, but it should surely recover and try again next time, and the error message should go away.

    Regards,
    Pete


    #PEJL
    Got any nice code? If you invest time in coding an elegant, novel or impressive answer on MSDN forums, why not copy it over to TechNet Wiki, for future generations to benefit from! You'll never get archived again, and you could win weekly awards!

    Have you got what it takes o become this month's TechNet Technical Guru? Join a long list of well known community big hitters, show your knowledge and prowess in your favoured technologies!

    Wednesday, February 17, 2016 4:36 PM

Answers

  • Hi Pete,

    Based on how I understand your question, the following may or may not apply. Please clarify if I misunderstood.

    The ASA reference data mechanism assumes that reference blobs, once produced, will not be replaced. Internally, ASA travels forward and backward in time to deal with issues such as node failures while maintaining deterministic and repeatable output. For that reason, ASA assumes that reference data updates enter the system in the form of new blobs, not updates/replacements of previously existing ones.

    Depending on the interaction of an internal replay and your replacing a blob file, almost anything could happen - from the errors you saw, to ASA ignoring the changes to reference data and continuing to join against old values, to ASA actually picking up the changes, but only applying them to some and not necessarily the right values.

    So, following the reference-data architecture as documented is really important to get correct system behavior. We are working on ways to make all this less "manual" and more robust.

    Regards,
    Clemens

    Friday, February 19, 2016 2:17 AM

All replies

  • Hi Pete,

    Based on how I understand your question, the following may or may not apply. Please clarify if I misunderstood.

    The ASA reference data mechanism assumes that reference blobs, once produced, will not be replaced. Internally, ASA travels forward and backward in time to deal with issues such as node failures while maintaining deterministic and repeatable output. For that reason, ASA assumes that reference data updates enter the system in the form of new blobs, not updates/replacements of previously existing ones.

    Depending on the interaction of an internal replay and your replacing a blob file, almost anything could happen - from the errors you saw, to ASA ignoring the changes to reference data and continuing to join against old values, to ASA actually picking up the changes, but only applying them to some and not necessarily the right values.

    So, following the reference-data architecture as documented is really important to get correct system behavior. We are working on ways to make all this less "manual" and more robust.

    Regards,
    Clemens

    Friday, February 19, 2016 2:17 AM
  • OK thanks, that makes sense. Thanks for the clarification Clemens. 

    This is a misconception I've had for a while then, and one that I think came from the description in the creation of inputs.

    It says "reference data is static or slow changing", which kind of implies the file may change:

    I think if you just changed this description it would go a long way to avoiding any confusion.

    Thanks again for the response and sorry for my delay in replying. You know how it is!

    Regards,
    Pete


    #PEJL
    Got any nice code? If you invest time in coding an elegant, novel or impressive answer on MSDN forums, why not copy it over to TechNet Wiki, for future generations to benefit from! You'll never get archived again, and you could win weekly awards!

    Have you got what it takes o become this month's TechNet Technical Guru? Join a long list of well known community big hitters, show your knowledge and prowess in your favoured technologies!

    Wednesday, February 24, 2016 2:19 PM
  • Thank you for the feedback - I can see how the description of the RD option in the UX could be misleading. I'll file a work item to improve that.

    Clemens

    Sunday, February 28, 2016 10:23 PM