none
Automated machine learning model Deployment fails RRS feed

  • Question

  • I am trying to deploy an automated machine learning model on azure portal ,but it fails no matter what I do.

    But it fails on deploy the model step.

    Friday, September 20, 2019 10:17 AM

Answers

  • The problem seems to be that the first row in their training data has null values for some columns and the system looks like all values to be present so it can figure out datatypes for all columns. We have fixed the behavior to accommodate this situation and the fix will be rolled out by first week of Oct.

    Meanwhile, you can workaround the situation by manually cleaning up first row or just move a fully filled row to the first position and retrain.

    Thanks!

     

    Thursday, September 26, 2019 4:15 AM

All replies

  • Hello,

    Could you please let us know if you are using the visual interface of automl or through the SDK? It would be great if you can post a screen shot or the logs to debug further.

    -Rohit

    Friday, September 20, 2019 1:46 PM
    Moderator
  • Hi ,I am using the Visual Interface,

    following is the log.

    2019-09-20T07:08:08,190558940+00:00 - gunicorn/run 
    2019-09-20T07:08:08,196526472+00:00 - rsyslog/run 
    2019-09-20T07:08:08,201116797+00:00 - iot-server/run 
    2019-09-20T07:08:08,202847006+00:00 - nginx/run 
    EdgeHubConnectionString and IOTEDGE_IOTHUBHOSTNAME are not set. Exiting...
    2019-09-20T07:08:08,539745720+00:00 - iot-server/finish 1 0
    2019-09-20T07:08:08,551747485+00:00 - Exit code 1 is normal. Not restarting iot-server.
    Starting gunicorn 19.6.0
    Listening at: http://127.0.0.1:31311 (14)
    Using worker: sync
    worker timeout is set to 300
    Booting worker with pid: 49
    'pattern' package not found; tag filters are not available for English
    Exception in worker process
    Traceback (most recent call last):
      File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/arbiter.py", line 557, in spawn_worker
        worker.init_process()
      File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/workers/base.py", line 126, in init_process
        self.load_wsgi()
      File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/workers/base.py", line 136, in load_wsgi
        self.wsgi = self.app.wsgi()
      File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/app/base.py", line 67, in wsgi
        self.callable = self.load()
      File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 65, in load
        return self.load_wsgiapp()
      File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
        return util.import_app(self.app_uri)
      File "/opt/miniconda/lib/python3.6/site-packages/gunicorn/util.py", line 357, in import_app
        __import__(module)
      File "/var/azureml-server/wsgi.py", line 1, in <module>
        import create_app
      File "/var/azureml-server/create_app.py", line 3, in <module>
        from app import main
      File "/var/azureml-server/app.py", line 13, in <module>
        import main as user_main
      File "/var/azureml-app/main.py", line 17, in <module>
        input_sample = pd.DataFrame(data=[{"AREA":"Arts","SPECIAL_GROUP":"Current Matric","ADMISSION_RATING":41.0,"IN_RESIDENCE":"Not In Residence","TOTAL_FIN_AID":1.0,"COURSES_ENROLLED":8.0,"PAYMENTS":-9400.0,"ENROLLED_CREDIT_POINTS":162.0,"Faculty":"Faculty of Humanities","ACADEMIC_CHARGES":100582.0,"PAYMENTS2":9400.0,"PAYMENTS_RATIO":0.0934560856,"FAM_UNIV_ATTENDANCE":"Y","USED_SCH_COMPUTERS":" ","USED_SCH_LIBRARY":" ","USED_SCH_SCIENCE_LABS":" ","GeneralSupport":"Single Parents","RACE":"BLACK","GENDER":"F","URBAN_RURAL":"URBAN","QUINTILE":4.0,"Mathematics":null,"Physical Sciences":null,"English":76.0}])
    NameError: name 'null' is not defined
    Worker exiting (pid: 49)
    Shutting down: Master
    Reason: Worker failed to boot.
    2019-09-20T07:08:16,808583902+00:00 - gunicorn/finish 3 0
    2019-09-20T07:08:16,809909909+00:00 - Exit code 3 is not normal. Killing image.
    


    Wednesday, September 25, 2019 7:50 AM
  • I am using Azure tutorials for deployment,due to some reasons I can post the link here.
    Wednesday, September 25, 2019 7:59 AM
  • The problem seems to be that the first row in their training data has null values for some columns and the system looks like all values to be present so it can figure out datatypes for all columns. We have fixed the behavior to accommodate this situation and the fix will be rolled out by first week of Oct.

    Meanwhile, you can workaround the situation by manually cleaning up first row or just move a fully filled row to the first position and retrain.

    Thanks!

     

    Thursday, September 26, 2019 4:15 AM
  • Thanks I will check and confirm
    Thursday, September 26, 2019 9:34 AM
  • hi ,

    I have fixed the file for the null values in the first column,but when I try to run the automl I get following error:

    ERROR: Run AutoML_55175b2a-94e8-42df-8d9c-5a65434336bd_4 failed with exception "[Errno 24] Too many open files"

    Any idea whats wrong?

    Friday, September 27, 2019 7:48 AM
  • Hi ,

    Is the server down ?I am not able to run any of my experiments since friday.

    Monday, September 30, 2019 6:58 AM
  • Hi,

    Is the server down still an issue? if so, can you provide more details, what exactly is the error message?

    Friday, October 4, 2019 7:11 PM