none
HttpClient Rest Api Paging RRS feed

  • Question

  • So, I have an API that I am calling to get back records.  The format is below in json format:

    "query": "budgets",
        "meta": {
            "href": "https://api2.e-builder.net/api/v2/budgets",
            "offset": 0,
            "limit": 3000,
            "size": 3000,
            "totalRecords": 15016
        },
        "records": [
            {
                "budgetId": "3e435eda-0fa6-49d3-afb3-c541b881711b",
                "projectId": "df9a22d6-72e4-4250-9dfb-9b423c9dfae2",
                "projectName": "*e-Builder Implementation Project",
                "budgetControl": "Uncontrolled",
                "description": "",
                "status": "Draft",
                "submittedForApprovalById": null,
                "submittedForApprovalBy": null,
                "submittedForApprovalDate": null,
                "approvedById": null,
                "approvedBy": null,
                "approvedByDate": null,
                "createdById": "e708077e-e570-4049-8bf5-09015427c494",
                "createdBy": "Avi Levin",
                "dateCreated": "2014-05-21T20:28:47",
                "lastModifiedById": "e708077e-e570-4049-8bf5-09015427c494",
                "lastModifiedBy": "Avi Levin",
                "lastModifiedDate": "2014-05-21T20:28:47",
                "costControlTolerancePercent": null
            },

    I can get the first 3000 records as stated for the limit in pagination.  What I am not understanding is how to create a call back so I can keep going on to the next 3000 records and so forth.  I am pretty new to API's so I am trying really hard to get this call back working.  Here is my C# code:

    GetToken p = new GetToken();
                p.ProcessToken();
                var token = p.AuthToken.access_token;

                var serializer = new DataContractJsonSerializer(typeof(ebuilder));
                client.DefaultRequestHeaders.Accept.Clear();
                client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/x-www-form-urlencoded"));
                client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", token);
                // client.DefaultRequestHeaders.Add("User-Agent", ".NET Foundation Repository Reporter");

                var streamTask = client.GetStreamAsync("https://api2.e-builder.net/api/v2/budgets");
                var repositories = serializer.ReadObject(await streamTask) as ebuilder;

                DateConverter dateConverter = new DateConverter();

                foreach (var record in repositories.records)
                {
                    var date = dateConverter.DateConvert(record.dateCreated);
                    using (var context = new EBuilderContext())
                    {
                        var budgets = new Budgets()
                        {
                            BudgetId = record.budgetId,
                            ProjectId = record.projectId,
                            ProjectName = record.projectName,
                            BudgetControl = record.budgetControl,
                            Description = record.description,
                            Status = record.status,
                            SubmittedForApprovalById = record.submittedForApprovalById,
                            SubmittedForApprovalBy = record.submittedForApprovalBy,
                            SubmittedForApprovalDate = dateConverter.DateConvert(record.submittedForApprovalDate),
                            ApprovedById = record.approvedById,
                            ApprovedBy = record.approvedBy,
                            ApprovedByDate = dateConverter.DateConvert(record.approvedByDate),
                            CreatedById = record.createdById,
                            CreatedBy = record.createdBy,
                            DateCreated = dateConverter.DateConvert(record.dateCreated),
                            LastModifiedById = record.lastModifiedById,
                            LastModifiedBy = record.lastModifiedBy,
                            LastModifiedDate = dateConverter.DateConvert(record.lastModifiedDate),
                            CostControlTolerancePercent = record.costControlTolerancePercent
                        };
                        context.Budgets.Add(budgets);
                        context.SaveChanges();
                    }
                }

    I would imagine that I would need to use some type of a recursive function but not sure how to build that in to an async method.  Any help would be greatly appreciated.


    Thursday, December 6, 2018 3:21 PM

All replies

  • REST APIs do it differently so you'll have to refer to the APIs documentation. They will all discuss how they do paging. Some allow you to pass query parameters (e.g. $pageIndex=1&$pageSize=100). Others require you add an HTTP header to the request. For GET requests (the most common time you'll need paging) query parameters are generally used. My gut instinct given your API and URL is that you need to pass the offset/limit as query parameters. So update your URL to use the correct parameter names as defined by the APIs documentation.

    Note, you're specifying a content type that isn't JSON. If the API returns JSON (which they pretty much all do) then you should be using the JSON content type instead. If you're code is working now then I wouldn't change it but you should look into it at a later date.


    Michael Taylor http://www.michaeltaylorp3.net

    Thursday, December 6, 2018 4:06 PM
    Moderator
  • I am still a little bit lost here.  So I would have to put some type of logic that hits the offset/limt?  I am still not sure what to do at this point.

    Thursday, December 6, 2018 4:45 PM
  • Yes. We'll assume the paging options are part of the query string. You need to adjust your call to the API to be in a loop. From the GetStreamAsync call and down is in the loop. Each time through the loop you'll need to adjust the URL to include the correct values for the paging parameters. Then get the data using GetStreamAsync and enumerate through the results adding them to your existing data.

    Each API is a little different so you'll have to play around with the paging and how to tell when you're done but based upon your sample data it looks like they are using offset/limit. In this model the offset is how many items to skip and limit is how many you want. The API can return less than this if desired. On return offset and limit are simply the values you specified but size indicates how many records were returned.

    So you would start with an offset of 0 and a limit of however many you want (or don't set it). Each time through the loop offset needs to get changed to offset += size (from the current request). So given your example offset = 0 for the first pass, offset = 3000 for the second, etc. Note that you need to use the size they give you and not assume it is some fixed value. An API can return a partial set of data irrelevant of your limit value.

    To know when you're done you can do a couple of different things. Perhaps the easiest thing to do is simply keep calling until you get back a size of 0. Most APIs will return nothing when you run out. Alternatively you could keep track of how many you've already read and compare that to the total records they are returning in the metadata. 


    Michael Taylor http://www.michaeltaylorp3.net

    Thursday, December 6, 2018 4:54 PM
    Moderator