none
Corrupt files comping from Stream Analytics

    Question

  • We have an Issue where the CSV files coming from Stream Analytics are corrupt. Corruption is totally random and seems to come from outside sources. We have multiple such files, all coming in at a different dates and times.

    This cannot be our fault, as Stream Analytics is unable to insert such binary strings into the CSV stream.

    Here are 3 lines from our log  CSV file:

    2016-12-21T22:06:48.0000000,DEV:123,log:123,deviceReset,DEV:123,,,,,,,,
    2016-12-21T22:31:01.0000000,DEV:123,log:123,deviceReset,DEV:123,,,,,,,,
    PAR1\FC.H spark_schemaZ\00 %SurveyCompletedId%\00\00 %ServiceRequestNumber%\00\00 %IncidentDataSource%\00\00%SurveyCompletedDateTime2016-12-21T23:31:56.0000000,DEV:123,log:123,deviceReset,DEV:123,,,,,,,,

    The corrupt part talks about some survey completed, but our app is an IoTApp and does not have surveys.

    The corrupt parts usually are before a new CSV line. Always something binary or totally unrelated to our app.



    • Edited by Lux Technology Friday, January 13, 2017 10:12 AM More details
    Friday, January 13, 2017 10:10 AM

All replies

  • Here is another interesting example:

    DEV:123,log:123,dev:123,2016-12-28T05:56:57.0000000Z,fLampMonitor,fLM2,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
    DEV:123,log:123,dev:123,2016-12-28T05:56:57.0000000Z,fBinarySensor,fBS1,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
    DEV:123,log:123,dev:123,2016-12-28T05:56:57.0000000Z,fBinarySensor,fBS2,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,
    1775968807|124350585|9350610|1|33|53769.21|0.06|0.00|N|O|1998-07-24|1998-08-01|1998-07-26|DELIVER IN PERSON|REG AIR|detect even, ironic idea
    1775968807|160272842|2772859|2|46|83114.18|0.07|0.05|N|O|1998-08-29|1998-08-11|1998-09-21|COLLECT COD|SHIP|tions cajole caref
    1775968807|105642556|5642557|3|41|61224.07|0.01|0.02|N|O|1998-05-31|1998-06-27|1998-06-01|DELIVER IN PERSON|RAIL|g the furiously ironic deposits w
    1775968807|134427444|6927458|4|50|68236.00|0.09|0.00|N|O|1998-08-05|1998-07-31|1998-08-18|COLLECT COD|FOB| furiously special
    1775968807|122039585|4539598|5|7|10629.36|0.04|0.06|N|O|1998-06-02|1998-08-15|1998-06-07|NONE|AIR|refully ironic 
    1775968807|119288862|1788874|6|7|12914.30|0.03|0.08|N|O|1998-06-16|1998-07-29|1998-06-24|TAKE BACK RETURN|RAIL|nal deposits. carefully final 
    1775968807|87900393|400402|7|2|2778.00|0.00|0.02|N|O|1998-09-02|1998-07-14|1998-09-16|TAKE BACK RETURN|TRUCK|lites. furiously final theodolites sleep
    1775968832|141888463|1888464|1|50|72218.50|0.09|0.08|N|O|1996-07-12|1996-06-13|1996-07-27|COLLECT COD|FOB|sts: slyly 
    1775968832|119385194|9385195|2|20|25464.60|0.08|0.02|N|O|1996-07-24|1996-06-04|1996-08-22|TAKE BACK RETURN|FOB|even packages integrate blithely. ex
    1775968832|106946097|9446108|3|29|32994.75|0.05|0.06|N|O|1996-07-27|1996-05-28|1996-08-12|DELIVER IN PERSON|RAIL| final accounts-- carefully ironic requests
    1775968832|43044312|3044313|4|19|23829.04|0.00|0.04|N|O|1996-06-03|1996-06-01|1996-06-15|TAKE BACK RETURN|MAIL|eas about the ironic, pendin
    1775968832|45750874|8250879|5|3|5767.77|0.03|0.01|N|O|1996-07-31|1996-07-01|1996-08-17|DELIVER IN PERSON|AIR|jole carefully along
    1775968832|134927180|2427220|6|39|46817.16|0.07|0.01|N|O|1996-05-02|1996-06-25|1996-05-14|COLLECT COD|FOB|hinly ironic pinto beans wake qui
    1775968833|64002078|6502085|1|47|55313.36|0.00|0.00|N|O|1997-11-11|1997-10-18|1997-11-20|TAKE BACK RETURN|TRUCK|efully? blithely
    1775968833|82910241|410266|2|42|52378.20|0.07|0.01|N|O|1997-08-31|1997-09-12|1997-09-02|NONE|TRUCK|gular, express requests i
    1775968833|72405490|7405505|3|17|23661.79|0.05|0.00|N|O|1997-08-12|1997-09-16|1997-09-10|TAKE BACK RETURN|SHIP|l pinto beans sleep blithely
    1775968833|119055341|1555353|4|12|15484.68|0.10|0.03|N|O|1997-10-08|1997-10-26|1997-10-29|DELIVER IN PERSON|MAIL|. fluffily regular requests wake fl
    1775968834|62838730|7838743|1|30|49967.70|0.02|0.06|N|O|1997-05-17|1997-04-24|1997-05-26|NONE|REG AIR|unusual deposits sl
    1775968834|130525146|3025160|2|43|50078.66|0.04|0.04|N|O|1997-05-02|1997-05-27|1997-05-03|COLLECT COD|AIR|ackages are packages. ironic, fin
    1775968834|136113504|1113531|3|29|43810.30|0.07|0.02|N|O|1997-04-09|1997-04-29|1997-04-29|TAKE BACK RETURN|TRUCK|en deposits haggle idly across the blith
    1775968834|59654091|4654102|4|36|37515.96|0.05|0.01|N|O|1997-04-05|1997-05-24|1997-04-15|TAKE BACK RETURN|MAIL|ven excuses. final, daring pac
    1775968834|6298835|8798836|5|1|1833.52|0.01|0.01|N|O|1997-04-29|1997-06-10|1997-05-22|COLLECT COD|REG AIR|its wake blithely above the bli
    1775968834|64513916|9513929|6|19|36607.11|0.05|0.02|N|O|1997-05-10|1997-05-20|1997-06-08|TAKE BACK RETURN|AIR|nusual accounts cajole. furi
    1775968834|85375147|375164|7|25|30447.00|0.05|0.06|N|O|1997-05-08|1997-04-21|1997-05-14|NONE|MAIL|nic foxes cajole pack
    1775968835|100079642|7579673|1|6|9699.84|0.08|0.02|A|F|1994-07-10|1994-05-05|1994-07-31|DELIVER IN PERSON|TRUCK|s nag slyly across the 
    1775968835|38541587|6041597|2|8|13013.28|0.01|0.04|A|F|1994-06-04|1994-05-08|1994-06-24|TAKE BACK RETURN|RAIL|s. furiously unusual dependencies
    1775968835|168444790|8444791|3|35|60422.95|0.08|0.00|A|F|1994-07-10|1994-05-08|1994-07-30|NONE|FOB|refully enticing deposits. furi
    1775968835|51573420|1573421|4|23|34289.55|0.10|0.03|R|F|1994-05-12|1994-06-23|1994-05-16|COLLECT COD|MAIL| the regular packages cajole always 
    1775968835|194467828|4467829|5|42|75016.20|0.03|0.08|A|F|1994-05-28|1994-05-22|1994-06-27|COLLECT COD|REG AIR|s boost quickly slyly ironic
    1775968835|20784433|8284440|6|10|15164.00|0.05|0.06|R|F|1994-05-19|1994-05-05|1994-06-15|COLLECT COD|MAIL|s cajole. even packages al
    17759DEV:123,log:123,dev:lc01028,2016-12-28T06:06:36.0000000Z,fElectricalMeter,fEM,,,,,,,,,,,,0.035,,0.872,229.46,0.179,,49.99,,,,,,,,,,,,,,,,,,,,
    DEV:123,log:123,dev:lc01028,2016-12-28T06:06:36.0000000Z,fCommunication,fC_RF,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,,0,
    
    Friday, January 13, 2017 10:53 AM
  • Hi,

    We are investigating this issue as we speak. Thanks for pointing to it. Will update this post when we have a fix deployed. Sorry for the inconvenience.

    Thanks,
    Sachin Sheth
    Program Manager, Azure Data Lake.


    Friday, January 20, 2017 4:19 PM
  • Hi,

    The fix for this issue has been deployed. If you could please try again will be super. Thanks for your patience.

    Cheers,
    Sachin Sheth.

    Friday, January 27, 2017 4:39 PM