none
[BUG] VS caches Data Lake file sets

    Question

  • I don't know if this is a known issue, but this happens to me all the time.

    I run a script that pulls data from a file set. I then delete or add files that should affect what gets pulled into the file set. If I run the script again (without debugging), the script will run, but it may miss the newly added files. Alternately, if will error out because it can't find files (cause I deleted them). However, if I run with debugging, these issues do not occur.

    When it doesn't recognize my deletions, I get an error and have simply lost some development time.

    When if does see additions, I'm left scratching my head until I realize that this issue has come up again.

    This does not repro 100%.

    Is this a known issue?

    Wednesday, September 14, 2016 5:01 PM

Answers

  • Hi,

    U-SQL collect the input file information at compilation time.  However, because the input files are not part of the build task, the build task can't know if the input files are changed and do rebuild in that case.  As a result, whenever you change the input files, you will have to build clean the project and recompile the script to run. 

    The issue is avoided in debug mode because we always rebuild the script in debug mode.  However, debug mode comes with the extra overhead of debugger attachment and symbol loading.

    Please let us know if rebuild won't fix your issue.

    Thanks a lot,

    Yu Dong

    Monday, September 19, 2016 4:05 AM

All replies

  • Hi - sorry for the inconvenience when using the VS tools. I have asked the dev owner of this feature to take a look at this and let us get back to you asap.

    Xiaoyong

    Monday, September 19, 2016 2:58 AM
  • Hi,

    U-SQL collect the input file information at compilation time.  However, because the input files are not part of the build task, the build task can't know if the input files are changed and do rebuild in that case.  As a result, whenever you change the input files, you will have to build clean the project and recompile the script to run. 

    The issue is avoided in debug mode because we always rebuild the script in debug mode.  However, debug mode comes with the extra overhead of debugger attachment and symbol loading.

    Please let us know if rebuild won't fix your issue.

    Thanks a lot,

    Yu Dong

    Monday, September 19, 2016 4:05 AM