locked
Unexpected results for some datasets using ARIMA RRS feed

  • Question

  • Hello,

    We are testing the viability of using the Time Series Algorithm to do some planning.  We ran multiple forecasts on historical data and did MAPE analysis against known actuals.  Many tests produced good results but a few tests produced forecasts that yielded outliers that were orders of magnitude from what we expected.  As a test we simply rounded the input set by tens which yielded a forecast much more in line with our expectations. 

    We are looking to determine what is causing the results we are seeing.  We are trying to gain confidence in the results that the ARIMA algorithm is producing and wonder if we are doing something wrong or if this is a bug.  Is there any pre-processing of the input data that we should be doing prior to feeding it to the ARIMA algorithm?

    We noticed this anomaly with ARIMA but not with ARTXP.  Using the default parameters, the blended results appear similar since the ARIMA results were so skewed.  So the only parameter set here is FORECAST_METHOD=ARIMA.  The input data is highly seasonal but setting the PERIODICITY_HINT did not change the result.  The data is 3 years (36 months) of history.

     

    Here is the forecast with the same input dataset rounded to tens using the same parameters as above.

     


    We were able to replicate this behavior on both 32-bit and 64-bit Excel 2013 DM add-in clients using 3 different instances of SSAS:
    SSAS 2008 R2 10.50.4033.0
    SSAS 2012 11.0.2100.60
    SSAS 2014 12.0.2000.8


    Here is the dataset below.  Looking forward to getting some insights from the knowledgeable experts here. 

    Regards,
    Luke Pargiter

    Month_Number,Actuals,RoundTens
    1,26326,26330
    2,33304,33300
    3,23937,23940
    4,30361,30360
    5,59608,59610
    6,115833,115830
    7,137246,137250
    8,231858,231860
    9,131603,131600
    10,167205,167210
    11,143898,143900
    12,38176,38180
    13,35601,35600
    14,33527,33530
    15,21678,21680
    16,24937,24940
    17,52682,52680
    18,113818,113820
    19,172357,172360
    20,182304,182300
    21,123104,123100
    22,138360,138360
    23,96800,96800
    24,49337,49340
    25,28422,28420
    26,31213,31210
    27,23461,23460
    28,28803,28800
    29,72233,72230
    30,131516,131520
    31,137565,137570
    32,199553,199550
    33,86884,86880
    34,142882,142880
    35,114475,114480
    36,52832,52830


    • Edited by Luke Pargiter Wednesday, April 1, 2015 1:23 PM Added pre-processing question
    Wednesday, April 1, 2015 2:38 AM

Answers

  • Interesting issue. I can reproduce the issue using the sample data that you provided.

    It looks like something wrong with the algorithm implementation. If I include only 35 rows or a dummy record #37, it works fine.

    Suggest that you open up a support incident with Microsoft CSS to report this bug.

    • Marked as answer by Luke Pargiter Thursday, April 2, 2015 10:11 PM
    Thursday, April 2, 2015 9:54 AM

All replies

  • Hi Luke,

    I am trying to involve someone more familiar with this topic for a further look at this issue. Sometime delay might be expected from the job transferring. Your patience is greatly appreciated.

    Thank you for your understanding and support.

    Regards,
    Katherine Xiong


    Katherine Xiong
    TechNet Community Support

    Thursday, April 2, 2015 1:54 AM
  • Interesting issue. I can reproduce the issue using the sample data that you provided.

    It looks like something wrong with the algorithm implementation. If I include only 35 rows or a dummy record #37, it works fine.

    Suggest that you open up a support incident with Microsoft CSS to report this bug.

    • Marked as answer by Luke Pargiter Thursday, April 2, 2015 10:11 PM
    Thursday, April 2, 2015 9:54 AM
  • I'm just wondering why you decided to use ARIMA for short-term forecast.

    Andrey V Artemyev | Saint-Petersburg, Russia

    Thursday, April 2, 2015 10:41 AM
  • Thank you for your response.  We'll open a Microsoft CSS support incident.

    Regards,

    Luke

    Thursday, April 2, 2015 8:52 PM
  • Good question.  Actually we are not using ARIMA for a short-term forecast.  We are using the default blended ARTXP/ARIMA forecast method.  We noticed this anomaly in the blended forecast and determined the results were being significantly skewed by the ARIMA component of the blend. Blending 4,000,000 and 20,000 still leave you with a result orders of magnitude different from the expected result of around 20,000. So when reporting the issue we specifically isolated the ARIMA component for attempts to reproduce the issue.
    Thursday, April 2, 2015 9:07 PM
  • Thanks for the explanation.

    I have noticed some interesting things with ARIMA too. There are 3 charts based on the same data and almost the same models. The first is MIXED (even with PREDICTION_SMOOTHING parameter set to 0.1). The second is ARIMA only, and the last is ARTXP only.

    As you can see only one of 7 series goes crazy. There is nothing special in the data of this series. Above I also set MINIMUM_SERIES_VALUE to 0 because my values here can't be negative by definition. If I remove this parameter, I will see nearly exactly the same behavior as you did.


    Andrey V Artemyev | Saint-Petersburg, Russia

    Friday, April 3, 2015 7:04 AM
  • Pls, if there is something interesting by the end of your support incident, let us know here.

    Andrey V Artemyev | Saint-Petersburg, Russia

    Friday, April 3, 2015 7:13 AM
  • Andrey, thank you for sharing your experiences and providing additional examples.  The behavior does appear very similar.  We also set the MINIMUM_SERIES_VALUE=0 for our tests but did not include that for reproducing the issue.  Our original results in our testing looked like this with parameters set as MIXED forecast method (default), PERIODICITY_HINT={12}, MINIMUM_SERIES_VALUE=0.

    For any folks interested in following it, bug feedback is submitted on Microsoft Connect here:

    https://connect.microsoft.com/SQLServer/Feedback/Details/1221449

    -- Luke

    Friday, April 3, 2015 11:11 PM