locked
Database Read (Break up of Data) RRS feed

  • Question

  • User-2017823710 posted

    Hi,

    I have the following components:

    UI Page:

    WebService:

    SQLServerDB:

    I am calling a web method in my web service which in turn calls a SP which return 30k rows. Unfortunately there is a 7MB limit on the data being passed from the WebService to the UI.

    I would like to split the data 30k into chunks of 5k and send to UI. Is there a way to accomplish this? Any other ideas?

    Thanks,

    Sid

     

     

     

     

     

    Monday, September 9, 2013 11:32 AM

Answers

  • User-821857111 posted

    You can use Row_Number to page the data from the database and provide paging links in the UI. That way you will only retrieve 20 rows (for example) at a time. If the user really wants to read all 30,000 rows of data, they can go through the paging links.

    • Marked as answer by Anonymous Thursday, October 7, 2021 12:00 AM
    Monday, September 9, 2013 2:42 PM
  • User697462465 posted

    Hi sid,

    According to your description, my understanding is that you would like to page the data.

    If so, please try to refer to the following code:

    First of all the big magic here is the SQL Query which makes it easy to handle the amount of data to be fetch from database.  I used the NorthWind database's Orders  table which has over 800 records. Here Index and Size

    are two integer variables which are comes from querystring .

    • Index is the current page number
    • Size is the no of records to be shown on a page

     

    select * from (SELECT  ROW_NUMBER() OVER (ORDER BY CustomerID asc) as row,* FROM Orders) 
      tblTemp WHERE row between (" + index + " - 1) * " + size + 
      " + 1 and " + index + "*" + size + " ";
                SQL += " select COUNT(*) from Orders 
    

    First time when the page loads it has no querystring

    values so the default Index

    is 1 and Size

    is 10. You can latter change the page size, there are options for 10 records per page and 15 records per page. I tried to explain all necessary points within the code. <!--?xml:namespace prefix = "o" ns = "urn:schemas-microsoft-com:office:office" /--><o:p></o:p>

     

    void BindGrid(int size, int index)
    {
        //page url--get from Web.config file--------change it as your page URL
        string url = ConfigurationManager.AppSettings["URL"].ToString();
        //string templates for links
        string link = "<a  href='" + url + 
          "?Index=##Index##&amp;Size=##Size##'><span class='page-numbers'>##Text##</span></a>";
        string link_pre = "<a href='" + url + 
          "?Index=##Index##&amp;Size=##Size##'><span class='page-numbers prev'>##Text##</span></a>";
        string link_next = "<a href='" + url + 
          "?Index=##Index##&amp;Size=##Size##'><span class='page-numbers next'>##Text##</span></a>";
        try
        {
            //the connectionstring from Web.config file--------change it as per your database settings
            String ConnStr = ConfigurationManager.ConnectionStrings["myConnection"].ToString();
            //the sql query with paging logics
            //here table name is "Orders" change this name as your requirement.
            //"CustomerID" is the column by which you can sort the records, change it as per your requirement
            String SQL = @"select * from (SELECT  ROW_NUMBER() OVER (ORDER BY CustomerID asc) as row,* FROM Orders) tblTemp
                        WHERE row between (" + index + " - 1) * " + size + 
                        " + 1 and " + index + "*" + size + " ";
            SQL += " select COUNT(*) from Orders";
            //fetching data from database suing SqlDataAdapter Fill method to bind the Gridview
            SqlDataAdapter ad = new SqlDataAdapter(SQL, ConnStr);
            DataSet ds = new DataSet();
            ad.Fill(ds);
            //bind the grid with the ist data table---------remember that this dataset consist of two data tables
            this.gvPaging.DataSource = ds.Tables[0];
            this.gvPaging.DataBind();
            ////////get the n number of record///////////
            Double n = Convert.ToDouble(Convert.ToInt32(ds.Tables[1].Rows[0][0]) / size);
            /////////setting page numbers with links
            if (index != 1)
                lblpre.Text = link_pre.Replace("##Size##", size.ToString()).Replace(
                  "##Index##", (index - 1).ToString()).Replace("##Text##", "prev");
            else
                lblpre.Text = "<span class='page-numbers prev'>prev</span>";
            if (index != Convert.ToInt32(n))
                lblnext.Text = link_next.Replace("##Size##", size.ToString()).Replace(
                  "##Index##", (index + 1).ToString()).Replace("##Text##", "next");
            else
                lblnext.Text = "<span class='page-numbers next'>next</span>";
            //generate dynamic paging 
            int start;
            if (index <= 5) start = 1;
            else start = index - 4;
            for (int i = start; i < start + 7; i++)
            {
                if (i > n) continue;
                //create dynamic HyperLinks 
                HyperLink lnk = new HyperLink();
     
                lnk.ID = "lnk_" + i.ToString();
                if (i == index)//current page
                {
                    lnk.CssClass = "page-numbers current";
                    lnk.Text = i.ToString();
                }
                else
                {
                    lnk.Text = i.ToString();
                    lnk.CssClass = "page-numbers";
                    lnk.NavigateUrl = url + "?Index=" + i + "&Size=" + size + "";
                }
                //add links to page
                this.pl.Controls.Add(lnk);
            }
            //------------------------------------------------------------------
            //set up the ist page and the last page
            if (n > 7)
            {
                if (index <= Convert.ToInt32(n / 2))
                {
                    lblLast.Visible = true;
                    lblIst.Visible = false;
                    lblLast.Text = link.Replace("##Index##", n.ToString()).Replace(
                      "##Size##", size.ToString()).Replace("##Text##", n.ToString());
                    spDot2.Visible = true;
                    spDot1.Visible = false;
                }
                else
                {
                    lblLast.Visible = false;
                    lblIst.Visible = true;
                    lblIst.Text = link.Replace("##Index##", (n - n + 1).ToString()).Replace(
                      "##Size##", size.ToString()).Replace("##Text##", (n - n + 1).ToString());
                    spDot2.Visible = false;
                    spDot1.Visible = true;
                }
            }
        }
        catch (Exception ee)
        {
            //catch the exception
        }
    } 
    

    In the above method you have to change the Config settings like the Connectionstring 

    and page url which can be edited in the web.config file.<!--?xml:namespace prefix = "o" ns = "urn:schemas-microsoft-com:office:office" /--><o:p></o:p>

    On the aspx page there are six  labels:<o:p></o:p>

    1. previous page link
    2. ist page link <o:p></o:p>
    3. dot before ist page link <o:p></o:p>
    4. next page link<o:p></o:p>
    5. last page link <o:p></o:p>
    6. dot before ist page link<o:p></o:p>

    and a PlaceHolder

     for other page numbers <o:p></o:p>

    <div class="pager fl">
        <asp:Label runat="server" ID="lblpre"></asp:Label>
        <asp:Label runat="server" ID="lblIst" Visible="false"></asp:Label>
        <asp:Label runat="server" ID="spDot1" CssClass="page-numbers prev" Visible="false"
            Text="..."></asp:Label>
        <asp:PlaceHolder ID="pl" runat="server"></asp:PlaceHolder>
        <asp:Label runat="server" ID="spDot2" Visible="false" CssClass="page-numbers prev"
            Text="..."></asp:Label>
        <asp:Label runat="server" ID="lblLast" Visible="false"></asp:Label>
        <asp:Label runat="server" ID="lblnext"></asp:Label>
    </div>
    <div class="pager f2">
        <a id="A1" href="javascript:void(0);" runat="server" class="page-numbers">10</a>&nbsp;
        <a href="javascript:void(0);" id="A2" runat="server" class="page-numbers">15</a><span
            class="page-numbers desc">per page</span>
    </div>
    
    HyperLinks
     

    are dynamically created and set their Links and css and add in the placeholder at page load.  <!--?xml:namespace prefix = "o" ns = "urn:schemas-microsoft-com:office:office" /--><o:p></o:p>

    Another important point is to replace the Table name in the sql Query and the column name in the order by statement.   <o:p></o:p>

    To change the page size there are two options(html anchor tags) on the right side of the page. You can set 10 records per page or 15 records per page. Using JQuery.

     I capture the click events of these two html anchors and set the currently set page size  in a HiddenField

     and  reload the current page by passing Index and Size in querystring.<o:p></o:p>

    jquery code:<o:p></o:p>

    <input type="hidden" runat="server" id="hdSize" />
    <script type="text/javascript">
        $(document).ready(function () {
            $('#A1').click(function () {
               $('#hdSize').val('10');
                Loadpage();
            });
            $('#A2').click(function () {
                $('#hdSize').val('15');
                Loadpage();
            });
        });
        function Loadpage() {
            try {
                //reload page using new page size
                var url = $(location).attr('href');
                var index = "Index=1";
                if (url.indexOf('Index') > 0) {
                    index = url.split('?')[1];
                    index = index.split('&')[0];
                }
                url = url.split('?')[0] + "?" + index + "&Size=" + $('#hdSize').val() + "";
                window.location.href = url;
            } catch (e) {
                alert(e);
            }
        }
    </script>
    

    CSS: CSS plays an important role in this paging. Although all works done but these all looks very poor untill a good css effect is given. <!--?xml:namespace prefix = "o" ns = "urn:schemas-microsoft-com:office:office" /--><o:p></o:p>

    td
    {
        color: #55ff00;
    }
    .text
    {
        color: #74aacc;
    }
     
    .fl
    {
        float: left;
    }
    .f2
    {
        float: right;
        padding-right: 200px;
    }
    .pager
    {
        margin-top: 20px;
        margin-bottom: 20px;
    }
    .page-numbers
    {
        font-family: 'Helvetica Neue' ,Helvetica,Arial,sans-serif;
        border: 1px solid #CCC;
        color: #808185;
        display: block;
        float: left;
        font-size: 130%;
        margin-right: 3px;
        padding: 4px 4px 3px;
        text-decoration: none;
    }
    .page-numbers.current
    {
        background-color: #808185;
        border: 1px solid #808185;
        color: white;
        font-weight: bold;
    }
    .page-numbers.next, .page-numbers.prev
    {
        border: 1px solid white;
    }
    .page-numbers.desc
    {
        border: none;
        margin-bottom: 10px;
    }

    Hope it can help you.

    Best Regards,
    Terry Guo


    • Marked as answer by Anonymous Thursday, October 7, 2021 12:00 AM
    Friday, September 13, 2013 9:25 AM

All replies

  • User-821857111 posted

    You can use Row_Number to page the data from the database and provide paging links in the UI. That way you will only retrieve 20 rows (for example) at a time. If the user really wants to read all 30,000 rows of data, they can go through the paging links.

    • Marked as answer by Anonymous Thursday, October 7, 2021 12:00 AM
    Monday, September 9, 2013 2:42 PM
  • User-2017823710 posted

    Hi,

    Thanks for the reply. Yes I incorporated the Row_Count logic in my SP. I was wondering if there was any example for the grid to incorporate the paging?

    Thanks,

    Sid

     

    Tuesday, September 10, 2013 9:53 AM
  • User-821857111 posted

    Put this in Google (or Bing) 'gridview efficient paging row_number'

    Tuesday, September 10, 2013 3:14 PM
  • User697462465 posted

    Hi sid,

    According to your description, my understanding is that you would like to page the data.

    If so, please try to refer to the following code:

    First of all the big magic here is the SQL Query which makes it easy to handle the amount of data to be fetch from database.  I used the NorthWind database's Orders  table which has over 800 records. Here Index and Size

    are two integer variables which are comes from querystring .

    • Index is the current page number
    • Size is the no of records to be shown on a page

     

    select * from (SELECT  ROW_NUMBER() OVER (ORDER BY CustomerID asc) as row,* FROM Orders) 
      tblTemp WHERE row between (" + index + " - 1) * " + size + 
      " + 1 and " + index + "*" + size + " ";
                SQL += " select COUNT(*) from Orders 
    

    First time when the page loads it has no querystring

    values so the default Index

    is 1 and Size

    is 10. You can latter change the page size, there are options for 10 records per page and 15 records per page. I tried to explain all necessary points within the code. <!--?xml:namespace prefix = "o" ns = "urn:schemas-microsoft-com:office:office" /--><o:p></o:p>

     

    void BindGrid(int size, int index)
    {
        //page url--get from Web.config file--------change it as your page URL
        string url = ConfigurationManager.AppSettings["URL"].ToString();
        //string templates for links
        string link = "<a  href='" + url + 
          "?Index=##Index##&amp;Size=##Size##'><span class='page-numbers'>##Text##</span></a>";
        string link_pre = "<a href='" + url + 
          "?Index=##Index##&amp;Size=##Size##'><span class='page-numbers prev'>##Text##</span></a>";
        string link_next = "<a href='" + url + 
          "?Index=##Index##&amp;Size=##Size##'><span class='page-numbers next'>##Text##</span></a>";
        try
        {
            //the connectionstring from Web.config file--------change it as per your database settings
            String ConnStr = ConfigurationManager.ConnectionStrings["myConnection"].ToString();
            //the sql query with paging logics
            //here table name is "Orders" change this name as your requirement.
            //"CustomerID" is the column by which you can sort the records, change it as per your requirement
            String SQL = @"select * from (SELECT  ROW_NUMBER() OVER (ORDER BY CustomerID asc) as row,* FROM Orders) tblTemp
                        WHERE row between (" + index + " - 1) * " + size + 
                        " + 1 and " + index + "*" + size + " ";
            SQL += " select COUNT(*) from Orders";
            //fetching data from database suing SqlDataAdapter Fill method to bind the Gridview
            SqlDataAdapter ad = new SqlDataAdapter(SQL, ConnStr);
            DataSet ds = new DataSet();
            ad.Fill(ds);
            //bind the grid with the ist data table---------remember that this dataset consist of two data tables
            this.gvPaging.DataSource = ds.Tables[0];
            this.gvPaging.DataBind();
            ////////get the n number of record///////////
            Double n = Convert.ToDouble(Convert.ToInt32(ds.Tables[1].Rows[0][0]) / size);
            /////////setting page numbers with links
            if (index != 1)
                lblpre.Text = link_pre.Replace("##Size##", size.ToString()).Replace(
                  "##Index##", (index - 1).ToString()).Replace("##Text##", "prev");
            else
                lblpre.Text = "<span class='page-numbers prev'>prev</span>";
            if (index != Convert.ToInt32(n))
                lblnext.Text = link_next.Replace("##Size##", size.ToString()).Replace(
                  "##Index##", (index + 1).ToString()).Replace("##Text##", "next");
            else
                lblnext.Text = "<span class='page-numbers next'>next</span>";
            //generate dynamic paging 
            int start;
            if (index <= 5) start = 1;
            else start = index - 4;
            for (int i = start; i < start + 7; i++)
            {
                if (i > n) continue;
                //create dynamic HyperLinks 
                HyperLink lnk = new HyperLink();
     
                lnk.ID = "lnk_" + i.ToString();
                if (i == index)//current page
                {
                    lnk.CssClass = "page-numbers current";
                    lnk.Text = i.ToString();
                }
                else
                {
                    lnk.Text = i.ToString();
                    lnk.CssClass = "page-numbers";
                    lnk.NavigateUrl = url + "?Index=" + i + "&Size=" + size + "";
                }
                //add links to page
                this.pl.Controls.Add(lnk);
            }
            //------------------------------------------------------------------
            //set up the ist page and the last page
            if (n > 7)
            {
                if (index <= Convert.ToInt32(n / 2))
                {
                    lblLast.Visible = true;
                    lblIst.Visible = false;
                    lblLast.Text = link.Replace("##Index##", n.ToString()).Replace(
                      "##Size##", size.ToString()).Replace("##Text##", n.ToString());
                    spDot2.Visible = true;
                    spDot1.Visible = false;
                }
                else
                {
                    lblLast.Visible = false;
                    lblIst.Visible = true;
                    lblIst.Text = link.Replace("##Index##", (n - n + 1).ToString()).Replace(
                      "##Size##", size.ToString()).Replace("##Text##", (n - n + 1).ToString());
                    spDot2.Visible = false;
                    spDot1.Visible = true;
                }
            }
        }
        catch (Exception ee)
        {
            //catch the exception
        }
    } 
    

    In the above method you have to change the Config settings like the Connectionstring 

    and page url which can be edited in the web.config file.<!--?xml:namespace prefix = "o" ns = "urn:schemas-microsoft-com:office:office" /--><o:p></o:p>

    On the aspx page there are six  labels:<o:p></o:p>

    1. previous page link
    2. ist page link <o:p></o:p>
    3. dot before ist page link <o:p></o:p>
    4. next page link<o:p></o:p>
    5. last page link <o:p></o:p>
    6. dot before ist page link<o:p></o:p>

    and a PlaceHolder

     for other page numbers <o:p></o:p>

    <div class="pager fl">
        <asp:Label runat="server" ID="lblpre"></asp:Label>
        <asp:Label runat="server" ID="lblIst" Visible="false"></asp:Label>
        <asp:Label runat="server" ID="spDot1" CssClass="page-numbers prev" Visible="false"
            Text="..."></asp:Label>
        <asp:PlaceHolder ID="pl" runat="server"></asp:PlaceHolder>
        <asp:Label runat="server" ID="spDot2" Visible="false" CssClass="page-numbers prev"
            Text="..."></asp:Label>
        <asp:Label runat="server" ID="lblLast" Visible="false"></asp:Label>
        <asp:Label runat="server" ID="lblnext"></asp:Label>
    </div>
    <div class="pager f2">
        <a id="A1" href="javascript:void(0);" runat="server" class="page-numbers">10</a>&nbsp;
        <a href="javascript:void(0);" id="A2" runat="server" class="page-numbers">15</a><span
            class="page-numbers desc">per page</span>
    </div>
    
    HyperLinks
     

    are dynamically created and set their Links and css and add in the placeholder at page load.  <!--?xml:namespace prefix = "o" ns = "urn:schemas-microsoft-com:office:office" /--><o:p></o:p>

    Another important point is to replace the Table name in the sql Query and the column name in the order by statement.   <o:p></o:p>

    To change the page size there are two options(html anchor tags) on the right side of the page. You can set 10 records per page or 15 records per page. Using JQuery.

     I capture the click events of these two html anchors and set the currently set page size  in a HiddenField

     and  reload the current page by passing Index and Size in querystring.<o:p></o:p>

    jquery code:<o:p></o:p>

    <input type="hidden" runat="server" id="hdSize" />
    <script type="text/javascript">
        $(document).ready(function () {
            $('#A1').click(function () {
               $('#hdSize').val('10');
                Loadpage();
            });
            $('#A2').click(function () {
                $('#hdSize').val('15');
                Loadpage();
            });
        });
        function Loadpage() {
            try {
                //reload page using new page size
                var url = $(location).attr('href');
                var index = "Index=1";
                if (url.indexOf('Index') > 0) {
                    index = url.split('?')[1];
                    index = index.split('&')[0];
                }
                url = url.split('?')[0] + "?" + index + "&Size=" + $('#hdSize').val() + "";
                window.location.href = url;
            } catch (e) {
                alert(e);
            }
        }
    </script>
    

    CSS: CSS plays an important role in this paging. Although all works done but these all looks very poor untill a good css effect is given. <!--?xml:namespace prefix = "o" ns = "urn:schemas-microsoft-com:office:office" /--><o:p></o:p>

    td
    {
        color: #55ff00;
    }
    .text
    {
        color: #74aacc;
    }
     
    .fl
    {
        float: left;
    }
    .f2
    {
        float: right;
        padding-right: 200px;
    }
    .pager
    {
        margin-top: 20px;
        margin-bottom: 20px;
    }
    .page-numbers
    {
        font-family: 'Helvetica Neue' ,Helvetica,Arial,sans-serif;
        border: 1px solid #CCC;
        color: #808185;
        display: block;
        float: left;
        font-size: 130%;
        margin-right: 3px;
        padding: 4px 4px 3px;
        text-decoration: none;
    }
    .page-numbers.current
    {
        background-color: #808185;
        border: 1px solid #808185;
        color: white;
        font-weight: bold;
    }
    .page-numbers.next, .page-numbers.prev
    {
        border: 1px solid white;
    }
    .page-numbers.desc
    {
        border: none;
        margin-bottom: 10px;
    }

    Hope it can help you.

    Best Regards,
    Terry Guo


    • Marked as answer by Anonymous Thursday, October 7, 2021 12:00 AM
    Friday, September 13, 2013 9:25 AM
  • User-2017823710 posted

    Thanks for your help Terry. I have incorporated using the same. I appreciate your help.

     

    Regards,

    Siddharth

    Monday, September 16, 2013 12:10 PM
  • User-2017823710 posted

    Thanks Mike for your help.

     

    Regards,

    Sid

    Monday, September 16, 2013 12:11 PM