locked
Azure ML Web-service - Same sample file but different predictions RRS feed

  • Question

  • Hi,

    My Azure ML Web-service is returning scored probabilities which are totally different from the ones I am getting in the ML Studio Experiment.

    Step 1: I created a basic experience with a dataset of 11,000 entries and a 70/30 randomized (but fixed) split:

    Screenshot: social.msdn.microsoft.com/Forums/getfile/732977

    Step 2: I used a Writer to retrieve the sample part from the Split (about 3,300 entries) as a .csv file (to re-use it with the Web-service)

    I carefully checked the CSV file generated by the Writer and I have exactly the same number of positive and negative entries than in the "Evaluate model" results window.

    Step 3: I saved the trained model and created the Web-service Experiment from it:

    Screenshot: social.msdn.microsoft.com/Forums/getfile/732978

    Step 4: I published the Web-service and created an external script which is sending values from sample file to the Web-service

    Issue: The scored probabilities returned by the Web-service are totally different from the ones I am getting and my F1 score is close to 7% (yes 7%!), whereas I am getting about 55% when using the ML Studio experiment with the same threshold.

    Anything I missed?

    Thank you for your help :-)

    Wednesday, October 14, 2015 12:00 AM

Answers

  • Thanks, but I still seem to get an error while trying to access these locations, do I have an incorrect address?

    http://social.msdn.microsoft.com/Forums/getfile/732977

    http://social.msdn.microsoft.com/Forums/getfile/732978

    • Marked as answer by neerajkh_MSFT Sunday, October 25, 2015 5:05 PM
    Wednesday, October 14, 2015 5:13 PM
  • Please try following:

    1) Save the right output of Normalize module: "Transformation Function"

    2) Substitute Normalize module by Apply Transformation module.

    3) Connect the saved transformation function to Apply Transformation module's left input.

    Otherwise, the Normalize will recalculate the normalization function for every row of web service input and return different results, instead of using the same normalization.

    Hope this helps,

    Roope

    • Marked as answer by neerajkh_MSFT Sunday, October 25, 2015 5:06 PM
    Thursday, October 15, 2015 5:25 PM

All replies

  • Hi Bruno,

    For some reason, I am unable to see the screenshots of your training/scoring experiments. Could you re-attach the screenshots and tell us which model you are using?

    In step-1: when you say fixed split, I assume you set the seed to be a fixed number to ensure that the split remains the same?  

    Thanks,
    Jaya.

    Wednesday, October 14, 2015 1:46 PM
  • Hi Jaya,

    Unfortunately I can't post images as this is my first post and my account is not verified yet.

    You can simply add "http://" before my screenshot links and it should work just fine :)

    For step 1, yes I am using a random seed to make sure I am always getting the same data.

    Regards,

    Wednesday, October 14, 2015 3:40 PM
  • Thanks, but I still seem to get an error while trying to access these locations, do I have an incorrect address?

    http://social.msdn.microsoft.com/Forums/getfile/732977

    http://social.msdn.microsoft.com/Forums/getfile/732978

    • Marked as answer by neerajkh_MSFT Sunday, October 25, 2015 5:05 PM
    Wednesday, October 14, 2015 5:13 PM
  • I think you're right, I can only see these two files when I am logged in (but not in private navigation mode where I am getting a 404).

    I just posted them on Imgur: imgur.com/a/fnFQs (add http://)

    Thank you

    Wednesday, October 14, 2015 5:19 PM
  • Hi Bruno,

    Thanks, now I am able to access the screenshots:

    Would it be possible to re-run your experiment without the 'Normalize Data' module and compare the results?

    Thanks,
    Jaya.

    Wednesday, October 14, 2015 6:31 PM
  • Please try following:

    1) Save the right output of Normalize module: "Transformation Function"

    2) Substitute Normalize module by Apply Transformation module.

    3) Connect the saved transformation function to Apply Transformation module's left input.

    Otherwise, the Normalize will recalculate the normalization function for every row of web service input and return different results, instead of using the same normalization.

    Hope this helps,

    Roope

    • Marked as answer by neerajkh_MSFT Sunday, October 25, 2015 5:06 PM
    Thursday, October 15, 2015 5:25 PM
  • Please try following:

    1) Save the right output of Normalize module: "Transformation Function"

    2) Substitute Normalize module by Apply Transformation module.

    3) Connect the saved transformation function to Apply Transformation module's left input.

    Otherwise, the Normalize will recalculate the normalization function for every row of web service input and return different results, instead of using the same normalization.

    Hope this helps,

    Roope

    Hi Roope,

    Thank you, I just tried this but unfortunately it did not work :(

    Screenshot: imgur.com/E3s7ezD (add http://) 

    Any other idea?

    Regards,

    Thursday, October 15, 2015 7:24 PM
  • Could you send a snapshot of output results from Studio Score vs web service Score for same data?

    Also, how are you computing the F1-score at this step: "The scored probabilities returned by the Web-service are totally different from the ones I am getting and my F1 score is close to 7% (yes 7%!)" ?

    Thanks,

    Roope

    Thursday, October 15, 2015 7:55 PM
  • Hi Bruno!

    Are you still having this issue? Please see my previous reply and let us know so we can investigate further.

    Thanks, Roope 

    Thursday, October 22, 2015 4:28 PM