locked
Memory keeps increasing when showing realtime data RRS feed

  • Question

  • User-1360795023 posted

    Hello, 

    I'm working on a mvc application to display realtime data (using IIS server), I have to work with EF and signalR (which I'm pretty new to this two), the software works perfectly except for one part where I have to reload the data from SQL Server every seconds (whenever the DB is modified) up to 70 rows can be modified (but usually about 40). 
    It works but when i check on the IIS worker process, the memory usage keeps growing whithout releasing it no matter what I tried... It use JS to reload this part and it seems that the problem (even to me it doesn't seem like the right way to do it) but I can't find another way around.
    I already tried to force the connection/disconnection, the GC seems to do his work, I know I can work with IIS server to recycle the pool application and limit the memory usage but it doesn't fix the issue. I know the issue come from the reload on the partial view, but I'm blocked. 
    Hope you guys can help.

    Model (The model is pretty basic only with some calculation like)

    public decimal TRS_EQ
            {
                get 
                {
                    if (pcs_theoriques_eq != " " && pcs_theoriques_eq != "" && pcs_theoriques_eq != null)
                    {
                        long theoEq = Convert.ToInt64(pcs_theoriques_eq);
                        if (pcs_bonnes_eq != null && pcs_bonnes_eq != " " && pcs_bonnes_eq != "" && theoEq != 0)
                        {
                            return Convert.ToDecimal(pcs_bonnes_eq) / theoEq * 100;
                        }
                    }
                    return 0.0m;
                }
            }

    Controller

    private IQueryable<TempsReel> collection;
    public ActionResult IndexPartial()
            {
                Response.Cache.SetNoStore();
                Response.Cache.SetNoServerCaching();
                using (var db = new SignalRTestContext())
                {
                    collection = from g in db.TempsReels select g;
                    collection = collection.OrderBy(g => g.groupe_machine);
                    ViewBag.NotifierEntity = db.GetNotifierEntity<TempsReel>(collection).ToJson();
                    ViewBag.IdMachine = Variables.idMachine;
                    ViewBag.Count = Variables.ListCatArrets.Count();
                    ViewBag.NomCategorie = Variables.ListCatArrets.Select(c => c.arretcatName).ToList();
                    ViewBag.Color = Variables.ListCatArrets.Select(c => c.arretcatColor).ToList();
                    collection = ArretTempsReel.OrganiseListParGroupe(collection);
                    GC.Collect();
                    db.Dispose();
                    return PartialView(collection.ToList());
                }               
            }

    View (reloading only partial view)

    <div id="tbTempsReelMachines">   
        @{ Html.RenderPartial("IndexPartial", Model); }
    </div>
    
    @section scripts 
    {
        <script src="~/Scripts/jquery.signalR-2.4.0.min.js"></script>
        <script src="~/signalr/hubs"></script>
        <script type="text/javascript">
            var signalRHubInitialized = false;
            $(function ()
            {
                InitializeSignalRHubStore();
            });
    
            function InitializeSignalRHubStore()
            {
                if (signalRHubInitialized)
                    return;
                try
                {
                    var clientHub = $.connection.tempsReelArretHub;
    
                    clientHub.client.broadcastMessage = function (message)
                    {
                        if (message === "Refresh")
                        {
                            ReloadIndexPartial();
                        }                  
                    };
                    $.connection.hub.start().done(function ()
                    {
                        clientHub.server.initialize($("#NotifierEntity").val());
                        signalRHubInitialized = true;
                    });
    
                } catch (err) {
                    signalRHubInitialized = false;
                }
            };
    
            function ReloadIndexPartial()
            {
    
                $.post('@(Url.Action("IndexPartial",
                "TempsReels", null, Request.Url.Scheme))')
                    .done(function (response)
                    {
                        $("#tbTempsReelMachines").html(response)     
                        if (!signalRHubInitialized)
                            InitializeSignalRHubStore();     
                    });
                $("#tbTempsReelMachines").end();
            };
        </script>
    }

    ViewModel (to organize the data)

    public class ArretTempsReel
        {
            public static IQueryable<TempsReel> OrganiseListParGroupe(IQueryable<TempsReel> InputList)
            {
                List<string> Groupe = InputList.Select(g => g.groupe_machine).Distinct().ToList();
                List<TempsReel> L = new List<TempsReel>();
                foreach (var item in Groupe)
                {
                    var l = InputList.Where(f => f.groupe_machine == item).ToList();
                    L.AddRange(l);
                    TempsReel SP = new TempsReel();
                    SP = SommePartielle(l);
                    L.Add(SP);
                }
                return L.AsQueryable<TempsReel>();
            }
            public static TempsReel SommePartielle(List<TempsReel> InputList)
            {
                TempsReelClone TC = new TempsReelClone();
                try
                {
                    foreach (var item in InputList)
                    {
                        TC.pcs_bonnes_eq += (item.pcs_bonnes_eq != null) ? Convert.ToInt64(item.pcs_bonnes_eq) : 0;
                        TC.pcs_bonnes_j += (item.pcs_bonnes_j != null) ? Convert.ToInt64(item.pcs_bonnes_j) : 0;
                        TC.pcs_bonnes_of += (item.pcs_bonnes_of != null) ? Convert.ToInt64(item.pcs_bonnes_of) : 0;
                        TC.pcs_mauvaises_eq += (item.pcs_mauvaises_eq != null) ? Convert.ToInt64(item.pcs_mauvaises_eq) : 0;
                        TC.pcs_mauvaises_j += (item.pcs_mauvaises_j != null) ? Convert.ToInt64(item.pcs_mauvaises_j) : 0;
                        TC.pcs_mauvaises_of += (item.pcs_mauvaises_of != null) ? Convert.ToInt64(item.pcs_mauvaises_of) : 0;
                        TC.pcs_theoriques_eq += (item.pcs_theoriques_eq != null) ? Convert.ToInt64(item.pcs_theoriques_eq) : 0;
                        TC.pcs_theoriques_j += (item.pcs_theoriques_j != null) ? Convert.ToInt64(item.pcs_theoriques_j) : 0;
                        TC.pcs_theoriques_of += (item.pcs_theoriques_of != null) ? Convert.ToInt64(item.pcs_theoriques_of) : 0;
                    }
                    return new TempsReel
                    {
                        pcs_bonnes_eq = TC.pcs_bonnes_eq.ToString(),
                        pcs_bonnes_j = TC.pcs_bonnes_j.ToString(),
                        pcs_bonnes_of = TC.pcs_bonnes_of.ToString(),
                        pcs_mauvaises_eq = TC.pcs_mauvaises_eq.ToString(),
                        pcs_mauvaises_j = TC.pcs_mauvaises_j.ToString(),
                        pcs_mauvaises_of = TC.pcs_mauvaises_of.ToString(),
                        pcs_theoriques_eq = TC.pcs_theoriques_eq.ToString(),
                        pcs_theoriques_j = TC.pcs_theoriques_j.ToString(),
                        pcs_theoriques_of = TC.pcs_theoriques_of.ToString(),
                        nom_machine = "TOTAL GROUPE",
                        categorie = "",
                        nom_arret = ""
    
                    };
                }
                catch (Exception)
                {
                    return null;
                }
            }

    Hub

    public class TempsReelHub : Hub
        {
            internal NotifierEntity NotifierEntity { get; private set; }
    
            public void DispatchToClient()
            {
                Clients.All.broadcastMessage("Refresh");
            }
    
            public void Initialize(String value)
            {
                NotifierEntity = NotifierEntity.FromJson(value);
                if (NotifierEntity == null)
                    return;
                Action<String> dispatcher = (t) => { DispatchToClient(); };
                PushSqlDependency.Instance(NotifierEntity, dispatcher);
            }
        }

    Notifier for SQL change

    public class SqlDependencyRegister
        {
            public event SqlNotificationEventHandler SqlNotification;
    
            readonly NotifierEntity notificationEntity;
    
            internal SqlDependencyRegister(NotifierEntity notificationEntity)
            {
                this.notificationEntity = notificationEntity;
                RegisterForNotifications();
            }
    
            [System.Diagnostics.CodeAnalysis.SuppressMessage("Microsoft.Security",
            "CA2100:Review SQL queries for security vulnerabilities")]
            void RegisterForNotifications()
            {
                using (var sqlConnection = new SqlConnection(notificationEntity.SqlConnectionString))
                {
                    using (var sqlCommand = new SqlCommand(notificationEntity.SqlQuery, sqlConnection))
                    {
                        foreach (var sqlParameter in notificationEntity.SqlParameters)
                            sqlCommand.Parameters.Add(sqlParameter);
                        sqlCommand.Notification = null;
                        var sqlDependency = new SqlDependency(sqlCommand);
                        sqlDependency.OnChange += OnSqlDependencyChange;
                        if (sqlConnection.State == ConnectionState.Closed)
                        {
                            sqlConnection.Open();
                        }
                        sqlCommand.ExecuteNonQuery();
                        //sqlCommand.Parameters.Clear();
                        sqlConnection.Close();
                    }
                }
            }
            void OnSqlDependencyChange(object sender, SqlNotificationEventArgs e)
            {
                SqlNotification?.Invoke(sender, e);
                RegisterForNotifications();
            }
        }

    Monday, May 20, 2019 7:10 AM

All replies

  • User765422875 posted

    There is NO REASON to call db.Dispose(). Your context is within a using block and the compiler automatically creates a try/finally block and calls dispose in the finally block.

    As a general rule DO NOT use GC.Collect(). I'm guessing calling that is the cause of your performance issues. GC.Collect forces a collection by the garbage collector. Which means pausing all the threads of the program so that the garbage collector can verify which objects are no longer referenced and claim the unused memory.

    This, of course, CAN have an impact on performance. Bottom line, it's not a good practice to use it and you should only do it when you have a really good reason to do it.

    Monday, May 20, 2019 2:33 PM
  • User61956409 posted

    Hi Eddy00,

    Welcome to ASP.NET forums.

    Eddy00

    the memory usage keeps growing whithout releasing it no matter what I tried

    As far as I know, SignalR retains 1000 messages in memory per hub per connection by default, if large messages are exchanged between server and clients, which might cause memory issues.

    You can try to reduce the size of SignalR message(s) and reduce the value of DefaultMessageBufferSize to reduce the amount of server memory used. For detailed information, please check this link:

    https://docs.microsoft.com/en-us/aspnet/signalr/overview/performance/signalr-performance#tuning-your-signalr-server-for-performance

    With Regards,

    Fei Han

    Tuesday, May 21, 2019 1:29 AM
  • User-1360795023 posted

    Thx for the reply deepalgorithm,

    I was using db.Dispose() en GC.collect() to test out if the issue come from the threads (the issue doesn't come from these two because I already have increased data before I try to use them). At first, I was looking to clear the unused memory and I found the GC but it didn't affect the performance, the issue come from antoher part of the program unfortunaly.

    Tuesday, May 21, 2019 8:42 AM
  • User-1360795023 posted

    You can try to reduce the size of SignalR message(s) and reduce the value of DefaultMessageBufferSize to reduce the amount of server memory used.

    I didn't think to put it but there is already a DefaultMessageBufferSize that I lock to 500, it makes it slower to grow but it doesn't stop the data consumption... The thing is I have an another view (with the same classes and methods) where I check for a new entry in the DB and it works perfectly even if I send mutiple entries in a row.


    @model IEnumerable<SignalRWebApp.Models.TempsReelArret>
    @Html.Hidden("NotifierEntity", (object)ViewBag.NotifierEntity)
    <br />
    @{
        SignalRWebApp.Models.TempsReelArret elt =
        Model.Where(q => q.id_machine == ViewBag.IdMachine).FirstOrDefault();
    }
    @{
        SignalRWebApp.ViewModels.NomCouleurErreur item = new SignalRWebApp.ViewModels.NomCouleurErreur { CouleurErreur = "#000000", NomErreur = "" };
        if (elt != null)
        {
        var Lst = ViewBag.ListeCouleursArret as List<SignalRWebApp.ViewModels.NomCouleurErreur>
        ;
        item = Lst.Where(q => q.NomErreur == elt.categorie).FirstOrDefault();
        }
    }
    @if (elt != null)
        {
        if (item == null)
        {
        <label style="font-family:Consolas; font-size:30px; vertical-align:middle; text-align:center; background-color:white ; width:950px; height:60px; align-self:center;">
            @SignalRWebApp.Resources.Models.TempsReelArret.None
        </label>
        }
        else
        {
        <label style="font-family:Consolas; font-size:30px; vertical-align:middle; text-align:center; background-color:@item.CouleurErreur ; width:950px; height:60px; align-self:center;">
            @elt.categorie / @elt.nom_arret
        </label>
        }
    }

    Tuesday, May 21, 2019 8:55 AM
  • User765422875 posted

    There are other improvements you can make to your code.

    In  SqlDependencyRegister you do not need to call sqlConnection.Close() as you wrapped the connection in a using. Also, I would use async versions of the ADO.NET calls:

    OpenAsync

    ExecuteNonQueryAsync()

    The theoretical benefit of using async is that your application will use fewer threads, therefore consuming less memory and CPU resources on thread overhead. This leaves more resources available for your application, improving the performance and scalability of the application.

    For applications that receive a lot of simultaneous requests, the benefit can be significant. Not sure if this will help, but it's an easy thing to change. Apart from that, you should profile your SQL query and make sure that the correct indexes are in place.

    Tuesday, May 21, 2019 7:46 PM
  • User-1360795023 posted

    I would use async versions of the ADO.NET

    I though about it, wich in fact I already use for the other part of the project to show when there's a change in the DB, but the thing is I am no longer using DbSet to work with but IQueryable and when I try to async I got : 

    The source IQueryable doesn't implement IDbAsyncEnumerable<TempsReel>. Only sources that implement IDbAsyncEnumerable can be used for Entity Framework asynchronous operations. For more details see http://go.microsoft.com/fwlink/?LinkId=287068.

    I found some explanation with Unit test and mock but I didn't fully understand the process to solve the problem..


    I check if there's a stop (when a machine is running) in the DB with this one in async :

    public async Task<ActionResult> Index(int id)
            {
                // ***********************************
                IQueryable<NomCouleurErreur> query = (from c in db.Categories select new NomCouleurErreur { NomErreur = c.arretcatName, CouleurErreur = c.arretcatColor });
                Variables.ListCouleursArret = new System.Collections.Generic.List<NomCouleurErreur>();
                foreach (NomCouleurErreur item in query.ToList())
                {
                    Variables.ListCouleursArret.Add(item);
                }
                // ************************************
                DbSet<TempsReelArret> collection = db.TempsReelArrets;
                ViewBag.NotifierEntity = db.GetNotifierEntity<TempsReelArret>(collection).ToJson();
                ViewBag.IdMachine = id;
                Variables.idMachine = id;
                ViewBag.ListeCouleursArret = Variables.ListCouleursArret;
                Variables.TempPageNumber = 1;
                return View(await collection.ToListAsync());
            }
    
    public async Task<ActionResult> IndexPartial()
            {
                DbSet<TempsReelArret> collection = db.TempsReelArrets;
                ViewBag.NotifierEntity = db.GetNotifierEntity<TempsReelArret>(collection).ToJson();
                ViewBag.IdMachine = Variables.idMachine;
                ViewBag.ListeCouleursArret = Variables.ListCouleursArret;
                return PartialView(await collection.ToListAsync());
            }

    Wednesday, May 22, 2019 2:48 PM
  • User765422875 posted

    There is an extension you can use (see below). But why are you using IQueryable? Your usage of it doesn't make sense as you convert it to a list right away in the loop with:

    query.ToList();

    https://docs.microsoft.com/en-us/dotnet/api/microsoft.entityframeworkcore.entityframeworkqueryableextensions.tolistasync?view=efcore-2.1

    Either use IQueryable or not. Not sure why you need it. Just the async version of ToList().

    Wednesday, May 22, 2019 3:15 PM
  • User-1360795023 posted

    The usage of the IQueryable (I'm not the one who work on this part of the project so I'm trying to work with) is to make some calculations with certains data and to injects it back in group order (each row show the real time data and calculations from all machines). So when we convert int the loop, it has everything we need.

    Where the calculations are send as queryable :

    public static IQueryable<TempsReel> OrganiseListParGroupe(IQueryable<TempsReel> InputList)
            {
                List<string> Groupe = InputList.Select(g => g.groupe_machine).Distinct().ToList();
                List<TempsReel> L = new List<TempsReel>();
                foreach (string item in Groupe)
                {
                    List<TempsReel> l = InputList.Where(f => f.groupe_machine == item).ToList();
                    L.AddRange(l);
                    TempsReel SP = new TempsReel();
                    SP = SommePartielle(l);
                    L.Add(SP);
                }
                return L.AsQueryable<TempsReel>();
            }        
    

    A part of the calculation made in the Model :

    public decimal TRS
            {
                get
                {
                    if (pcs_theoriques_eq != " " && pcs_theoriques_eq != "" && pcs_theoriques_eq != null)
                    {
                        long theoEq = Convert.ToInt64(pcs_theoriques_eq);
                        if (pcs_bonnes_eq != null && pcs_bonnes_eq != " " && pcs_bonnes_eq != "" && theoEq != 0)
                        {
                            return Convert.ToDecimal(pcs_bonnes_eq) / theoEq * 100;
                        }
                    }
                    return 0.0m;
                }
            }

    I get the error when I try to use ToListAsync() (with the EF Core and not the Entity) :  The source IQueryable doesn't implement IAsyncEnumerable but I found another way around with the Task usage, it seems to slow the issue but not stopping it..

    public async Task<ActionResult> IndexPartial()
            {
                using (SignalRTestContext db = new SignalRTestContext())
                {
                    IQueryable<TempsReel> collection = db.TempsReels.OrderBy(c => c.groupe_machine); 
                    ViewBag.NotifierEntity = db.GetNotifierEntity<TempsReel>(collection).ToJson();
                    ViewBag.IdMachine = Variables.idMachine;
                    ViewBag.Count = Variables.ListCatArrets.Count();
                    ViewBag.NomCategorie = Variables.ListCatArrets.Select(c => c.arretcatName).ToList();
                    ViewBag.Color = Variables.ListCatArrets.Select(c => c.arretcatColor).ToList();
                    collection = ArretTempsReel.OrganiseListParGroupe(collection);
                    Task<List<TempsReel>> result = Task.FromResult(collection.ToList());
                    return PartialView(await result);
                }               
            } 

    Thursday, May 23, 2019 8:29 AM
  • User765422875 posted

    I see a few more things that look questionable.

    Is this an EF call? If so, why isn't it async?

    ViewBag.NotifierEntity = db.GetNotifierEntity<TempsReel>(collection).ToJson();

    Why are you using Task.FromResult?

    Task<List<TempsReel>> result = Task.FromResult(collection.ToList());

    Task.FromResult doesn't actually create or runs a task but it just wraps the returned result in a task object.

    What is the point of using it? Your method is async. Just use async all the way.

    Hope all of this helps. Good luck.

    Thursday, May 23, 2019 4:14 PM
  • User-1360795023 posted

    It seems that's we use this viewbag to keep in track with the notifier (when to reload the data) but for this one I'm a bit lost myself

    public static NotifierEntity GetNotifierEntity<TEntity>
            (this DbContext dbContext, IQueryable iQueryable) where TEntity : EntityBase
            {
                ObjectQuery objectQuery = dbContext.GetObjectQuery<TEntity>(iQueryable);
                return new NotifierEntity()
                {
                    SqlQuery = objectQuery.ToTraceString(),
                    SqlConnectionString = Variables.ConnectionString,
                    SqlParameters = objectQuery.SqlParameters()
                };
            }
    
            public static ObjectQuery GetObjectQuery<TEntity>
            (this DbContext dbContext, IQueryable query) where TEntity : EntityBase
            {
                if (query is ObjectQuery)
                    return query as ObjectQuery;
    
                if (dbContext == null)
                    throw new ArgumentException("dbContext cannot be null");
                ObjectSet<TEntity> objectSet = dbContext.UnderlyingContext().CreateObjectSet<TEntity>();
                IQueryProvider iQueryProvider = ((IQueryable)objectSet).Provider;
    
                // Use the provider and expression to create the ObjectQuery.
                return (ObjectQuery)iQueryProvider.CreateQuery(query.Expression);
            }

    Why are you using Task.FromResult?

    As I said, when I try to go with ToListAsync(), I have an error (The source IQueryable doesn't implement IAsyncEnumerable) so i find a way to make it functionnal, not the best option but I didn't find another as simple.

    Friday, May 24, 2019 9:53 AM
  • User765422875 posted

    Right, I see it. Another option is to write an Entity Framework extension that safely converts the IQueryable source into a list asynchronously.

    public static class EfExtensions
    {
      public static Task<List<TSource>> ToListAsyncSafe<TSource>(
        this IQueryable<TSource> source)
      {
        if (source == null)
          throw new ArgumentNullException(nameof(source));
        if (!(source is IAsyncEnumerable<TSource>))
          return Task.FromResult(source.ToList());
        return source.ToListAsync();
      }
    }

    Usage (sample)

    var result = await EfExtensions.ToListAsyncSafe<TempsReel>(collection.AsQueryable());
                    return PartialView(await result);

    I've used this extension before, but you will have to test it out with your code.

    Please mark the question(s) as answered if I've helped you out. Good luck.

    Friday, May 24, 2019 11:49 AM