locked
Intermittent data and what to look for in WebApi RRS feed

  • Question

  • User-501297529 posted

    When testing my application I'm seeing different data set every time even for the same region(Intermittent). There is a different count in the data set. What should I look for in the API for a situation like this. Would it authentication, not sure because it happens to all users and not just one.

    Thursday, April 18, 2019 2:10 PM

All replies

  • User753101303 posted

    Hi,

    Looks like what happens when using static data. As a web app is a single application used byt multiple users if using mistakenly shared data ALL users will see just the result for the latest query done by someone...

    Double check if you are storing results using a static variable.

    Thursday, April 18, 2019 2:25 PM
  • User-501297529 posted

    Interesting. The only thing that I could find that is storing results for static variable would be this.

     public static class ResponseMessage
        {
            #region DFS-PM module
    
            public static string PM_Welcome_Msg = "PM_Welcome_Msg";
            public static string TokenGranted { get { return "Token granted."; } }
            public static string Failed { get { return "Failed"; } }
            public static string PictureNotPosted { get { return "PictureNotPosted"; } }
            public static string PictureNotAvailable { get { return "PictureNotAvailable"; } }
            public static string PictureUploadedSuccessfully { get { return "PictureUploadedSuccessfully"; } }
            public static string PictureNotUploaded { get { return "PictureNotUploaded"; } }
            public static string FailedMessage { get { return "Invalid request, check parameters."; } }
            public static string FailedToRetreiveSyncedStatus { get { return "Failed to fetch the Synced data."; } }
            public static string Success { get { return "Success"; } }
            public static string InvalidCredentials { get { return "You have not supplied valid credentials."; } }
            public static string InternalServerError { get { return "Internal Server Error."; } }
            public static string UserNotRegisteredOnWeb { get { return "User is not registered in the system."; } }
    
            public static string CustomerFound { get { return "Customers received successfully."; } }
            public static string CustomerNotFound { get { return "Customers were not found."; } }
    
            public static string UnitsFound { get { return "Units & VIN received successfully."; } }
            public static string UnitsNotFound { get { return "Units & VIN were not found for customer."; } }
    
            public static string PmtypeFound { get { return "PM types received successfully."; } }
            public static string PmtypeNotFound { get { return "PM types were found."; } }
    
            public static string ScreenTypesSuccess { get { return "Screen Types received successfully."; } }
            public static string ScreenTypesFailed { get { return "Screen Types were not found."; } }
    
            public static string UnitsRecieved { get { return "Units received successfully."; } }
            public static string UnitsFailedToRecieve { get { return "Unites were not found."; } }
    
            public static string DefaultValuesRecieved { get { return "Default values received successfully."; } }
            public static string DefaultValuesFailed { get { return "Default values were not found."; } }
    
            public static string InspectionQuestionSuccess { get { return "Inspection questions received successfully."; } }
            public static string InspectionQuestionCategories { get { return "Inspection questions categories received successfully."; } }
            public static string InspectionQuestionFailed { get { return "Inspection questions not were found."; } }
            public static string QuestionCategoriesFailed { get { return "Inspection questions not were found."; } }
    
            public static string LogCreated { get { return "Log Created successfully."; } }
            public static string LogNotCreated { get { return "Log Creation Failed."; } }
            public static string InspectionSubmitted { get { return "Inspection submitted successfully."; } }
            public static string InspectionSubmittedFailed { get { return "Inspection could not be submitted successfully."; } }
    
            public static string LocationSaved { get { return "Location saved successfully."; } }
            public static string LocationNotSaved { get { return "Location save Failed."; } }
    
    
            public static string DeltaSyncSuccess { get { return "Delta Sync fetched successfully."; } }
            public static string DeltaSyncFailed { get { return "Delta Sync fetched failed."; } }
    
    
    
            #endregion
        }

    Is this what you are talking about and if so what should I update. What is an alternative instead of using static? Do I just remove 'static'?

    Thursday, April 18, 2019 2:40 PM
  • User753101303 posted

    Yes, delete static everywhere and create a new object that will represent the response for the current web api call so that each request can have its own specific response.

    Currently your application uses a single response for ALL API calls and if multiple requests are processed "at the same time" they could contribute to the final response to other requests.

    In short static data (using a static method returning non static data is ok) are shared accross the whole application. It's a common catch when coming from desktop development where each user have its own application instance while a web application is a single application instance used by multiple users (sharing then all static data).

    Thursday, April 18, 2019 3:04 PM
  • User475983607 posted

    The static class shown is read-only so the class is not the source of the data issue.  It still can be a static variable used to hold the data or caching.

    When testing my application I'm seeing different data set every time even for the same region(Intermittent).

    What does this mean, "regions"? What is a region?

    There is a different count in the data set. What should I look for in the API for a situation like this. Would it authentication, not sure because it happens to all users and not just one.

    How are you reproducing the issue?

    Thursday, April 18, 2019 3:14 PM
  • User-501297529 posted

    The static class shown is read-only so the class is not the source of the data issue.  It still can be a static variable used to hold the data or caching.

    bootzilla

    When testing my application I'm seeing different data set every time even for the same region(Intermittent).

    What does this mean, "regions"? What is a region?

    bootzilla

    There is a different count in the data set. What should I look for in the API for a situation like this. Would it authentication, not sure because it happens to all users and not just one.

    How are you reproducing the issue?

    Region is part of the data. One of the fields/columns is called region. That's all.

    I'm reproducing the issue in the app by selecting a customer and then select a vehicle/unit no. for that customer. When selecting vehicle/unit on one of the customer it shows 4 unit numbers which is correct. I go out of the unit selection screen without selecting a unit and then select the same customer again and it may show 3 or 4. Most times I test it shows 4 but sometimes it will show 3. For example one time I did this like 20 times before it showed 3 units. That's how I'm reproducing it. It doesn't happen every time but it does happen intermittently.

    Thursday, April 18, 2019 3:24 PM
  • User-501297529 posted

    Yes, delete static everywhere and create a new object that will represent the response for the current web api call so that each request can have its own specific response.

    Currently your application uses a single response for ALL API calls and if multiple requests are processed "at the same time" they could contribute to the final response to other requests.

    In short static data (using a static method returning non static data is ok) are shared accross the whole application. It's a common catch when coming from desktop development where each user have its own application instance while a web application is a single application instance used by multiple users (sharing then all static data).

    When you say create a new object how would I do that. Can you give an example based off my code? Basically I'm not sure how to do that. I did remove static from the class ResponseMessage and from the string variables but not sure how to create new object.

    Thursday, April 18, 2019 3:44 PM
  • User475983607 posted

    bootzilla

    Region is part of the data. One of the fields/columns is called region. That's all.

    I'm reproducing the issue in the app by selecting a customer and then select a vehicle/unit no. for that customer. When selecting vehicle/unit on one of the customer it shows 4 unit numbers which is correct. I go out of the unit selection screen without selecting a unit and then select the same customer again and it may show 3 or 4. Most times I test it shows 4 but sometimes it will show 3. For example one time I did this like 20 times before it showed 3 units. That's how I'm reproducing it. It doesn't happen every time but it does happen intermittently.

    It's not possible to answer this question. 

    You have a method that should return the same results given a fixed input.   The problem is either the method logic or the inputs.  I would create a unit test that passes the same inputs many times to the method and evaluate the results.  If the results change then you might investigate the logic further.  If the results are consistent then there is a problem with unexpected inputs.

    Thursday, April 18, 2019 3:51 PM
  • User-501297529 posted

    bootzilla

    Region is part of the data. One of the fields/columns is called region. That's all.

    I'm reproducing the issue in the app by selecting a customer and then select a vehicle/unit no. for that customer. When selecting vehicle/unit on one of the customer it shows 4 unit numbers which is correct. I go out of the unit selection screen without selecting a unit and then select the same customer again and it may show 3 or 4. Most times I test it shows 4 but sometimes it will show 3. For example one time I did this like 20 times before it showed 3 units. That's how I'm reproducing it. It doesn't happen every time but it does happen intermittently.

    It's not possible to answer this question. 

    You have a method that should return the same results given a fixed input.   The problem is either the method logic or the inputs.  I would create a unit test that passes the same inputs many times to the method and evaluate the results.  If the results change then you might investigate the logic further.  If the results are consistent then there is a problem with unexpected inputs.

    I see. I think I would like to try by removing static but I'm not sure how to create a new object as Patrice suggested. That is something I need assistance on how to do that. So I'm waiting on his response.

    Thursday, April 18, 2019 4:06 PM
  • User753101303 posted

    Ah it's actual code that return just message strings and it is intended to be the same for all users ? Then it is not the issue.

    Rather than looking for static (which is a guess and can be ok if your intent is to always have the same value), have you tried to look directly at the actual "different data set for the same region" you talked about ? This code doesn't use static data ??? Show maybe the relevant part so that we can spot a possible problem?

    For now my understanding is that each user doesn't necessarily always see the expected data he is supposed to see (and maybe a kind of mix with what other users should see).

    Thursday, April 18, 2019 4:10 PM
  • User-501297529 posted

    Ah it's actual code that return just message strings and it is intended to be the same for all users ? Then it is not the issue.

    Rather than looking for static (which is a guess and can be ok if your intent is to always have the same value), have you tried to look directly at the actual "different data set for the same region" you talked about ? This code doesn't use static data ??? Show maybe the relevant part so that we can spot a possible problem?

    For now my understanding is that each user doesn't necessarily always see the expected data he is supposed to see (and maybe a kind of mix with what other users should see).

    Ok I think I know what you are getting at.

    Here is the result set

     public dynamic GetFleetUnitNumberAndVINByCustomer(RegionParam regionParam)
            {
                try
                {
                    var fleetDetails = new List<CustomerFleetUnitVIN>();
    
                    SqlConnection conn = new SqlConnection(ApplicationDbContext.ConnectionString);
                    SqlDataAdapter da = new SqlDataAdapter();
                    SqlCommand cmd = conn.CreateCommand();
                    cmd.CommandText = "GetFleetUnitNumberAndVINByCustomer";
                    da.SelectCommand = cmd;
    
                    conn.Open();
                    DataSet ds = SqlHelper.ExecuteDataset(conn, CommandType.StoredProcedure, cmd.CommandText, new SqlParameter("@REGION", regionParam.Region.ToString()));
                    conn.Close();
    
                    DataTable dt = ds.Tables[0];
                    Parallel.ForEach(dt.Rows.Cast<DataRow>(), dr =>
                    {
                        var fleetInfo = new CustomerFleetUnitVIN
                        {
                            Unit = (dr["Unit"] == DBNull.Value) ? string.Empty : Convert.ToString(dr["Unit"]),
                            VIN = (dr["VIN"] == DBNull.Value) ? string.Empty : Convert.ToString(dr["VIN"]),
                            CustomerNumber = (dr["CustomerNumber"] == DBNull.Value) ? string.Empty : Convert.ToString(dr["CustomerNumber"]).Trim()
                        };
                        fleetDetails.Add(fleetInfo);
                    });
                    return fleetDetails;
                }
                catch (Exception e)
                {
                    throw;
                }
    
            }

    Here is part of the result set that I run in Postman, keep in mind the result is normally over 260000 lines:

    Sometimes when I hit send in postman I will get less lines and when that happens and I scroll up to see some of the results i see nulls in the results. Does that make sense and can you see any problems based off my code.

    {
                "unit": "N41762",
                "vin": "2FZHAZCV14AN41762",
                "customerNumber": "700150"
            },
            {
                "unit": "N48",
                "vin": "10102",
                "customerNumber": "700015"
            },
            {
                "unit": "N50307",
                "vin": "13N1452C641520258",
                "customerNumber": "700458"
            },
            {
                "unit": "N50347",
                "vin": "13N1452C851527617",
                "customerNumber": "700461"
            },
            {
                "unit": "N50348",
                "vin": "13N1452CX51527618",
                "customerNumber": "700458"
            },
    Thursday, April 18, 2019 4:35 PM
  • User753101303 posted

    What does ExecuteDataset ? I would drop Parallel.ForEach. AFAIK List<T>.Add is not thread safe.

    You return really 260000 for a single response ???? Usually you return a page or limit the number of rows as there is no way a user could find the line he is looking for among that many rows.

    Thursday, April 18, 2019 4:45 PM
  • User-501297529 posted
    ExecuteDataset executes the command and returns the results in a new dataset with all the rows. If that is what you are asking.

    Yes when I send the results in Postman I get that many lines of results for each set.

    As for drop Parallel.ForEach. AFAIK List<T. Should I just drop it and that's it or replace it with something else and do you believe that could be the cause if the problem?

    Thursday, April 18, 2019 5:18 PM
  • User753101303 posted

    I meant showing the code for ExecuteDataset in case the issue could be there. I meant use a classic "foreach" loop rather than Parallel.ForEach.

    According to https://docs.microsoft.com/en-us/dotnet/api/system.collections.generic.list-1?view=netframework-4.8#thread-safety it is not thread safe ie updating the collection from multiple threads could cause issues.

    Yoiu have specialized concurrent collection but given what is done I doubt it would give a significant difference which as single trheaded foreach loop

    Thursday, April 18, 2019 7:02 PM
  • User-501297529 posted

    I updated to foreach but now I get an error on Get error on the last '}' and ',' after <DataRow>() says expected : or }. I changed text color to orange for both where it errors.  Do I need to change those to something else?

     foreach(dt.Rows.Cast<DataRow>(), dr =>
                    {
                        var fleetInfo = new CustomerFleetUnitVIN
                        {
                            Unit = (dr["Unit"] == DBNull.Value) ? string.Empty : Convert.ToString(dr["Unit"]),
                            VIN = (dr["VIN"] == DBNull.Value) ? string.Empty : Convert.ToString(dr["VIN"]),
                            CustomerNumber = (dr["CustomerNumber"] == DBNull.Value) ? string.Empty : Convert.ToString(dr["CustomerNumber"]).Trim()
                        };
                        fleetDetails.Add(fleetInfo);
                    });

    foreach(dt.Rows.Cast<DataRow>(), dr =>

    { var fleetInfo = new CustomerFleetUnitVIN

    { Unit = (dr["Unit"] == DBNull.Value) ? string.Empty : Convert.ToString(dr["Unit"]),

    VIN = (dr["VIN"] == DBNull.Value) ? string.Empty : Convert.ToString(dr["VIN"]),

    CustomerNumber = (dr["CustomerNumber"] == DBNull.Value) ? string.Empty : Convert.ToString(dr["CustomerNumber"]).Trim() }; fleetDetails.Add(fleetInfo); });

    Monday, April 22, 2019 2:31 PM
  • User475983607 posted

    The foreach syntax is incorrect.

    foreach(var dr in dt.Rows)
    {
    	var fleetInfo = new CustomerFleetUnitVIN
    	{
    		Unit = (dr["Unit"] == DBNull.Value) ? string.Empty : Convert.ToString(dr["Unit"]),
    		VIN = (dr["VIN"] == DBNull.Value) ? string.Empty : Convert.ToString(dr["VIN"]),
    		CustomerNumber = (dr["CustomerNumber"] == DBNull.Value) ? string.Empty : Convert.ToString(dr["CustomerNumber"]).Trim()
    	};
    	fleetDetails.Add(fleetInfo);
    }

    C# programming guide.

    https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/arrays/using-foreach-with-arrays

    https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/keywords/foreach-in

    Monday, April 22, 2019 2:45 PM
  • User-501297529 posted

    The foreach syntax is incorrect.

    foreach(var dr in dt.Rows)
    {
    	var fleetInfo = new CustomerFleetUnitVIN
    	{
    		Unit = (dr["Unit"] == DBNull.Value) ? string.Empty : Convert.ToString(dr["Unit"]),
    		VIN = (dr["VIN"] == DBNull.Value) ? string.Empty : Convert.ToString(dr["VIN"]),
    		CustomerNumber = (dr["CustomerNumber"] == DBNull.Value) ? string.Empty : Convert.ToString(dr["CustomerNumber"]).Trim()
    	};
    	fleetDetails.Add(fleetInfo);
    });

    C# programming guide.

    https://docs.microsoft.com/en-us/dotnet/csharp/programming-guide/arrays/using-foreach-with-arrays

    https://docs.microsoft.com/en-us/dotnet/csharp/language-reference/keywords/foreach-in

    This doesn't work either. I get errors on all the datareaders 'Cannot apply indexing with [] to an expression of type 'object'

    Monday, April 22, 2019 2:53 PM
  • User475983607 posted

    This doesn't work either. I get errors on all the datareaders 'Cannot apply indexing with [] to an expression of type 'object'

    Try using DataRow rather than var.

    foreach(DataRow dr in dt.Rows)
    {
        var fleetInfo = new CustomerFleetUnitVIN
        {
            Unit = (dr["Unit"] == DBNull.Value) ? string.Empty : Convert.ToString(dr["Unit"]),
            VIN = (dr["VIN"] == DBNull.Value) ? string.Empty : Convert.ToString(dr["VIN"]),
            CustomerNumber = (dr["CustomerNumber"] == DBNull.Value) ? string.Empty : Convert.ToString(dr["CustomerNumber"]).Trim()
        };
        fleetDetails.Add(fleetInfo);
    }

    Monday, April 22, 2019 3:50 PM