none
Python notebook failed but the overall status of notebook activity shows as succeeded. RRS feed

  • Question

  • Hi I have daily ADF pipeline has the databricks activity calls this py notebook. This morning it has failed because of some transformation rules. But the overall notebook status and databricks activity shows as succeeded. 

    My assumption is like if the notebook has failed because of any reason, the activity in ADF should fail. Please correct me if my understanding is not right?

    Can you please review and help with this request? I am unable to upload images from ADF /DBricks .. I will do it once I get the access. Thanks 

    Wednesday, November 20, 2019 2:41 PM

Answers

  • Response from Support team:

    The notebook has not failed since you are handling the exception.

    For eg : Below commands wouldn't fail since the exception is being caught and is being swallowed.

    try: print(x) except NameError: print("Variable x is not defined") except: print("Something else went wrong")

    Below would work since the exception is being thrown and would cause notebook run failure and hence the job failure.

    try: print(x) except NameError: print("Variable x is not defined") raise Exception("Variable x is not defined") except: print("Something else went wrong")


    Thursday, December 5, 2019 8:36 PM

All replies

  • Hi

    This issue looks strange. Could you please re-run and check if you see the same behaviour?

    To share the screenshots, you can expedite verification by replying to this thread with your request to verify your account - https://social.technet.microsoft.com/Forums/en-US/dc4002e4-e3de-4b1e-9a97-3702387886cc/verify-account-42?forum=reportabug

    Thursday, November 21, 2019 6:20 AM
    Owner
  • Yes, I am able to reproduce this issue and it is same behavior

    thanks, I have replied back to above thread so that I can upload screenshots.
    Thursday, November 21, 2019 6:55 PM
  • Hello,

    Just checking in if you have had a chance to see the previous response. We need the following information to understand/investigate this issue further.

    Tuesday, November 26, 2019 8:20 AM
    Owner
  • Sorry for delay, as I was waiting on access to upload images. Here is the screenshot from Databricks

    This job from databricks has failed because the required blob is missing from Azure Container. Hence it failed. But the over status of job states "Succeeded" in 11 Sec. 

    Please let me know if you need any more details.



    • Edited by naveen.kuppili Tuesday, November 26, 2019 1:03 PM Added details
    Tuesday, November 26, 2019 12:55 PM
  • Hello,

    Thanks for sharing the screenshot of the error message.

    In order to investigate further, could you please share the python code which you are running in the notebook activity to test from our end.

    Wednesday, November 27, 2019 10:01 AM
    Owner
  • Hi Pradeep, If you are okay -  Can you please share your mail address, I will share the notebook with python scripts.
    Wednesday, November 27, 2019 4:59 PM
  • Here is code snippet, please let me know if you need any more info required.

    FolderName = 'SW'
    process_rundate = '2019-11-27'
    container = 'containerName'    
    storageAccount = 'AccountName'
    FilePath = "{}/Snapshot{}.csv".format( FolderName,process_rundate )
    inputFilePath = "wasbs://{}@{}.blob.core.windows.net/{}".format(container, storageAccount,FilePath )
    print(inputFilePath)
    df_SW = spark.read.format("com.databricks.spark.csv").options(header="true", inferschema="true").load(inputFilePath)
    df_SW.createOrReplaceTempView('vwSW')   

    Wednesday, November 27, 2019 8:04 PM
  • Hello,

    Sorry for the delayed response.

    What are the assigned permissions on the storage account?

    Note: You can read data from public storage accounts without any additional settings. To read data from a private storage account, you must configure a Shared Key or a Shared Access Signature (SAS). For leveraging credentials safely in Azure Databricks, we recommend that you follow the Secrets user guide as shown in Mount an Azure Blob storage container.

    If it is not public storage account, you can access the Azure Blob Storage using the following methods.

    • Mount Azure Blob storage containers to DBFS.
    • Access Azure Blob storage directly.
    • Access Azure Blob storage using the RDD API.

    For more details, refer “Azure Databricks – Blob Storage”.

    Hope this helps.

    ----------------------------------------------------------------------------------------

    Do click on "Mark as Answer" and Upvote on the post that helps you, this can be beneficial to other community members.

    Monday, December 2, 2019 4:40 AM
    Owner
  • Hi Pradeep, 

    I am "Contributor" role to this blob store. I have no problem reading files from blob and I use (option 2) access directly dataframe API and it is working fine.

    This issue is that when the file is not available on source (Azure blob Storage) then it fails to find the specific file which is fine. But the overall status of notebook still state as "Succeeded". Expectation is  that the status should read as "Failed" in the event of pyspark code throws exception.

    Please let me know if you have questions.


    Monday, December 2, 2019 5:44 PM
  • Hello, 

    This issue looks strange. For a deeper investigation and immediate assistance on this issue, if you have a support plan you may file a support ticket, else could you send an email to AzCommunity@Microsoft.com with your Subscription ID and thread link to this post, and I will enable a one-time free support request for your subscription.  

    Please reference this forum thread in the subject: “Python notebook failed but the overall status of notebook activity shows as succeeded”. Thank you for your persistence. 

    Tuesday, December 3, 2019 4:33 AM
    Owner
  • Yes, I am able to reproduce this issue and it is same behavior

    Ideally, if notebook fails the pipeline - notebook activity also should fail.

    For a test, try to add try catch kind of exception handling around python code and see if the activity fails if there is an exception.


    If the response helped, do "Mark as answer" and upvote it
    - Vaibhav

    Tuesday, December 3, 2019 7:19 AM
  • Thanks for checking, I will take help from my admin to raise support request as I don't have access to create one. 
    Tuesday, December 3, 2019 3:11 PM
  • Update: Support ticket has been raised.
    Wednesday, December 4, 2019 9:43 PM
  • Hello,

    Thanks for the update, once the issue is sorted out with the support, please do share the resolution, which might be beneficial to other community members reading this issue. 

    Thursday, December 5, 2019 8:26 AM
    Owner
  • Sure, I will 
    Thursday, December 5, 2019 2:12 PM
  • Response from Support team:

    The notebook has not failed since you are handling the exception.

    For eg : Below commands wouldn't fail since the exception is being caught and is being swallowed.

    try: print(x) except NameError: print("Variable x is not defined") except: print("Something else went wrong")

    Below would work since the exception is being thrown and would cause notebook run failure and hence the job failure.

    try: print(x) except NameError: print("Variable x is not defined") raise Exception("Variable x is not defined") except: print("Something else went wrong")


    Thursday, December 5, 2019 8:36 PM
  • Hi Naveen,

    Thanks for sharing the solution, which might be beneficial to other community members reading this thread. 

    Friday, December 6, 2019 4:35 AM
    Owner