none
passing large datasets to Azure ML published service RRS feed

  • Question

  • Hi there, 

    I am new to Azure ML Studio, and as such may be trying to use it in ways it isn't design for.   Hopefully someone can guide me here.

    I am trying to develop a demand planning model for a large inventory of materials (active ingredients).

    I want to pass in a time series dataset of daily consumption going back several years for ALL active ingredients used.  I want to get back another dataset that represents the predicted consumption 8 weeks into the future for each active ingredient.   

    The intent here would be to create a pipeline that would run nightly and continuously evolve with the latest actual consumption metrics. 

    My model trims, resamples, normalizes,  and groups the data, and I then pass it to a python script module that I wrote to interatively apply ARIMA to each active ingredients individual time series, and project out 8 weeks... there are about 50+ active ingredients.

    In effect I am doing 50+ on-the-fly training and prediction iterations (actually thousands because I also have to find the best parameters for ARIMA which involves calling the ARIMA algorithm 216 times per material) , and building up a new dataset as the ultimate output.   It can take a couple of hours to run one complete cycle, but I do get adequate results.

    I have this working as an experiment in Azure ML Studio, but I now want to publish it as a web service and allow someone to pass in large datasets and wait asynchronously for the results to eventually come back (I would have a service continuously running in the background waiting for the results)

    When I published this experiment through ML studio, it didn't seem possible to pass in a dataset.   It seemed like it would only let me pass in the individual fields that make up one record from the original dataset.

    Am I approaching this the wrong way?   Any suggestions?

    -Sheldon

     




    • Edited by slyttle Friday, November 8, 2019 1:35 PM
    Thursday, November 7, 2019 7:11 PM

All replies

  • OK .. I think I have answered my own question.

    I just need to upload my CSV to blob storage and reference it in my web service call as a new AzureBlobDataReference()

     new AzureBlobDataReference()
                                 {
                                     ConnectionString = storageConnectionString,
                                     RelativeLocation = string.Format("{0}/Usage Datadatablob.file_extension", StorageContainerName)
                                 }

    The web service will eventually send me back my outputs ... one being a dataset with the predictions, and other being a set of graphs, plotting out the predictions.   The graphs will be in base64 encoding which I can easily translate.

      
    Friday, November 8, 2019 8:41 PM
  • Hi,

    Thanks for asking this question and contributing to the community. Just to add, you may consider connecting an online data source as web service input in your predictive experiment and set Blob URI as web service parameter. Then you call the web service by specifying the "Blob URI" path in "Global Parameters". You can review sample code from your web service API help page for more details on how to consume the web service. Please let me know if you have any more questions. Thanks.

    Regards,

    GiftA-MSFT.

    If a post helps to resolve your issue, please click “Mark as Answer” and/or “Vote as helpful”. By marking a post as Answered and/or Helpful, you help others find the answer faster.  Thanks.

    Friday, November 8, 2019 11:37 PM
    Moderator