none
async sockets & windows 2008

    Question


  • Hi

    I'm about to kill myself over windows stability issues. I'm using windows CE 5.0 client side, which proved a total disaster and now I'm facing win2008 hell.

    Following code works fine, except on win2008:

            private void Produce()
            {
                while (running)
                {
                    try
                    {
                        Socket client = tcpl.Accept();
                        int wi = writeind[trdid];
                        client.SendTimeout = 2000 + wi * 100;
                     //   Debug.WriteLine("SOCK " + client.SendTimeout);
                        queue[trdid][wi] = client;
                        if (++wi == queuelength)
                            wi = 0;
                        writeind[trdid] = wi;
                        arearr[trdid].Set();
    
                        if (++trdid == numthreads)
                            trdid = 0;
                    }
                    catch (SocketException)
                    {
                        Thread.Sleep(100);
                    }
                }
            }
    
            private void Consume(object obj)
            {
                int trdnum = (int)obj;
                AutoResetEvent are = arearr[trdnum];
                Socket[] tcpq = queue[trdnum];
                int ri = 0;
                int riasync = 0;
    
                while (running)
                {
                    are.WaitOne(1000);
    
                    while (ri != writeind[trdnum])
                    {
                        Socket sock = tcpq[ri];
                        try
                        {
                       //     Debug.WriteLine("REQUEST RECEIVED");
                            sock.ReceiveTimeout = 2000;
                            sock.SendTimeout = 2000;
                            try
                            {
                                if (false)//true)//sock.Poll(10000, SelectMode.SelectRead))
                                {
                               //     Debug.WriteLine("SYNC");
                                    ConsumeSocket(sock, trdnum, data[trdnum]);
                                }
                                else
                                {
                                    // async receive!
                                //    Debug.WriteLine("ASYNC");
                                    AsyncObject ao = new AsyncObject();
                                    ao.tid = trdnum;
                                    ao.sock = sock;
                                    sock.BeginReceive(ao.buf, 0, ao.buf.Length, SocketFlags.None, AsyncReceive, ao);
                                }
                            }
                            catch (SocketException se)
                            {
                                Debug.WriteLine("caught exception " + se.Message);
                            }
                        }
                        catch (Exception ex)
                        {
                            Debug.WriteLine("Exception occured in socketlistener: " + ex.Message);
                            try
                            {
                                sock.Close();
                            }
                            catch (Exception) { }
                        }
                        if (++ri == queuelength)
                            ri = 0;
                    }
                }
            }
    
            public class AsyncObject
            { 
                public byte[] buf = new byte[1000];
                public Socket sock;
                public List<byte> fullbuf = new List<byte>();
                public int contentlength = -1;
                public int bodypos = -1;
                public int tid;
            }

    So I'm doing a synchronous accept and an async receive, which is how I believe a multithreaded socket server should be implemented. It works ok on win7 (64) but on *** win2008 (r2 64), for some blackboxed reason it's giving 100% cpu whenever beginreceive is called multiple times (*I think*) on -different- incoming sockets. The application keeps working btw, it's just eating away one of my (2) cores. The sync routine is working without issues, but obviously is slower/not scalable.

    I found some other posts from people with this exact same issue but no acceptable solutions. Is it at all possible to implement an async socket server on windows 2008, cause I'll do it, whatever mountain I have to climb/demons to slay.

    Now seriously, I'm losing all credibility here. People are depending on this application, it's their daily hell and it shouldn't go 100%cpu on stuff that's just supposed to work.

    I can accept/respect the 'we bugged it on purpose' answer, pm me if this is the case. Thanks!

    Monday, December 16, 2013 9:02 PM

Answers

  • "There's the weird thing, they're executing fine as far as I can tell, if I do a 'break all' it just hits some sleep method someplace else."

    The sleep is in the produced thread, what are the consumer threads doing?

    "If you're interested in the circular buffer thing, here's one of my prettiest classes implementing it:"

    Hmm, did you omit code from Enqueue? As is now enqueue will keep spinning when the queue gets full. I don't know where you are using this queue so I don't know if this could be a problem or not.

    "No, since ri is not reset to 0 it won't loop through the queue from start. It'll simply continue where it left off, consuming only newly added sockets."

    Ugh, I got it wrong again. Sort of. What happens if there are no new sockets? It looks to me that you won't call Poll/BeginReceive again on existing sockets unless a new client connects. Sounds weird.

    • Marked as answer by exstud Wednesday, December 18, 2013 7:09 PM
    Wednesday, December 18, 2013 7:22 AM
    Moderator

All replies

  • "So I'm doing a synchronous accept and an async receive, which is how I believe a multithreaded socket server should be implemented."

    I believe not. Well, yes, you can use BeginReceive if you want to do an async receive but that's pretty much the only thing that your code appears to do right. Beyond that, hmm, just no.

    • It looks to me that you're using multiple threads to call BeginReceive on sockets that are placed on a queue by Produce. It doesn't make a lot of sense to do that, you should have used BeginAccept which in turn will have invoked the callback on a thread pool thread.
    • What's doing on when the "queue" gets full seems just wrong. You don't seem to increase the queue length, you're just overwriting existing sockets without even closing them
    • When there's at least one socket in the queue the 2 Consume loops will spin like crazy because there's nothing to stop them. The event is already set and BeginReceive returns instantly, after all that's why it's called "begin".
    • The idea of using something like are.WaitOne(1000) in code that's supposed to be efficient is dubious to begin with. You're simply delaying the receive from the first client by 1 second, that's simply wasted time.
    • When there's an error you close the socket but you don't remove it from the queue, next time you'll try calling BeginReceive on a closed socket. That will throw another exception and you'll close the sockets again. And so on until the dead socket gets replaced by Produce

    I would suggest you to take a look at the Async Server Socket sample in MSDN and try to follow that pattern. It's a reasonable simple and efficient way to deal with async sockets.

    Tuesday, December 17, 2013 10:03 AM
    Moderator
  • - The code is supposed to call 'Poll' to check whether it makes sense to handle the call synchronously (which proved faster in some apachebench testing I did). If Poll fails it falls back to async handling and that's why beginreceive is called there, indeed from multiple consumer threads.

    - It's a circular queue, I didn't list the socket disposing (which uses another queue) just to keep it simple and because it has nothing to do with the issue I'm having. I can show you the full thing though if you think it would get us anywhere.

    - it's an autoresetevent, it resets itself so there's no spinning (changed it to manualreseteventslim actually after I got my 3.5 version)

    - Are you confusing WaitOne with Sleep? WaitOne immediately returns when set is called from another thread right?

    - As long as the consumers can keep up with the producer and the circular queue is long enough, sockets will not be handled/disposed of twice

    Thanks for the link and looking into this, but there's really nothing wrong with my code, it's EXACTLY what I want it to be (actually it could use some kernel caching but oh well...). I just checked on win 2012, it's working fine there as well. 

    The issue is either win2008 specific or it has smth to do with the VMware it's running in. I don't know yet.



    • Edited by exstud Tuesday, December 17, 2013 6:38 PM
    Tuesday, December 17, 2013 6:26 PM
  • "it's an autoresetevent, it resets itself so there's no spinning (changed it to manualreseteventslim actually after I got my 3.5 version)"

    Ugh, you're right, I missed that. Let's see how this works with such an event:

    • a consumer thread starts and waits on the event
    • the producer thread accepts a client and sets the event
    • the consumer thread wakes up and tries to receive data from all the sockets in the queue
    • the consumer thread is done with receiving and goes back to waiting on the event

    At this point I'm baffled. If no client tries to connect to the server then it will take 1 second before the consumer thread wakes up again to do another receive round. Is this really what you want?

    Maybe this is exactly what you want but it's anything but intuitive and given that parts of the code are missing the best I can do is guesswork

    And there are some issues that I missed in my previous post.

    OK, a client comes in and you don't wait 1 second. That means you go again through the socket queue and call BeginReceive again, on each socket. Even if a previous BeginReceive call might not have completed yet. I have no idea what would be the point in calling BeginReceive multiple times.

    And the ri variable is not reset to 0 after the inner loop completes. That means that next time you enter the inner loop the ri index will outside of the queue and you'll cause a bunch of null reference exceptions while trying to read from null sockets.

    "The issue is either win2008 specific or it has smth to do with the VMware it's running in. I don't know yet."

    The best thing would be to simply attach the debugger and see what each thread is doing. I'd exclude the auto reset even as being the issue. That would imply that a consumer thread gets stuck inside the inner while loop but the only way that could happen is if writeind[trdnum] is >= queuelength. But from the code you posted that doesn't seem to be possible.

    "which proved faster in some apachebench testing I did"

    One possible reason why Poll may be faster is the inefficient way you're using BeginReceive. You keep allocating AsyncObjects objects for every BeginReceive call. That includes the receive buffer so you're basically allocating ~1 kbyte for each receive, that will probably stress the GC unnecessarily. And the fact that BeginReceive has to pin the buffer doesn't help.

    Tuesday, December 17, 2013 7:41 PM
    Moderator
  • >> so you're basically allocating ~1 kbyte for each receive

    So it doesn't do exactly what I want :), but it's also a very small change to move the asyncobjects to thread local memory and reuse them. Besides, the async routine is only used for the slower connections, the majority of calls should go sync. I just fixed the sloppy exception handling as well...

    >> The best thing would be to simply attach the debugger and see what each thread is doing

    There's the weird thing, they're executing fine as far as I can tell, if I do a 'break all' it just hits some sleep method someplace else. I'd conclude then MY threads stay cold when idle (no incoming calls), but the program is still eating my cpu, and only on 2008!

    >> OK, a client comes in and you don't wait 1 second. That means you go again through the socket queue and call BeginReceive again, on each socket

    No, since ri is not reset to 0 it won't loop through the queue from start. It'll simply continue where it left off, consuming only newly added sockets. Beginreceive is called exactly once for each socket, but from multiple consumer threads.

    >>  If no client tries to connect to the server then it will take 1 second before the consumer thread wakes up again to do another receive round.

    It's acceptable imo, it wakes up each second, sees ri == writeind[trdnum] and goes back to waiting...

    I did the apachebenching on win vista a few years back and can't remember the exact details, but I did a fair optimization round for both (minimalistic) implementations and my conclusion was sync beats async in performance, at least on a quad core intel. (another conclusion was IIS7 beats just about evth else, probably only because of its kernel caching)

    If you're interested in the circular buffer thing, here's one of my prettiest classes implementing it:

       public class Kueue<T> where T : class
        {
            private T[] arr;
            private volatile int ind; // acquire and release semantics, hopefully on all platforms...
    
            private int m_Capacity;
            public int Capacity { get { return m_Capacity; } }
    
            public Kueue()
                : this(1024)
            { }
    
            public Kueue(int capacity)
            {
                this.m_Capacity = capacity;
                arr = new T[capacity];
            }
    
    
            public void Enqueue(T val)
            {
                if (val == null)
                    return;
                //Thread.MemoryBarrier();
                int i = ind;
                while (true)
                {
                    while (arr[i] != null)
                    {
                        i = (i + 1) % arr.Length;
                    }
    
                    if (Interlocked.CompareExchange<T>(ref arr[i], val, null) == null)
                    {
                        return;
                    }
                }
            }
            public  void AddRange(IEnumerable<T> arr)
            {
                foreach (T tmp in arr)
                    Enqueue(tmp);
            }
    
            public List<T> DequeueList()
            {
                List<T> ret = new List<T>();
    
                T t;
                while (TryDequeue(out t))
                {
                    ret.Add(t);
                }
    
    
                return ret;
            }
    
            public bool TryDequeue(out T result)
            {
                while (true)
                {
                    //Thread.MemoryBarrier();
                    int i = ind;
                    if (arr[i] == null)
                    {
                        result = null;
                        return false;
                    }
                    else
                    {
                        if (Interlocked.CompareExchange(ref ind, (i + 1) % arr.Length, i) == i)
                        {
                            T ret = arr[i];
                            arr[i] = null;
                            result = ret;
                            return true;
                        }
                    }
                }
            }
        }
    

    Tuesday, December 17, 2013 10:49 PM
  • "There's the weird thing, they're executing fine as far as I can tell, if I do a 'break all' it just hits some sleep method someplace else."

    The sleep is in the produced thread, what are the consumer threads doing?

    "If you're interested in the circular buffer thing, here's one of my prettiest classes implementing it:"

    Hmm, did you omit code from Enqueue? As is now enqueue will keep spinning when the queue gets full. I don't know where you are using this queue so I don't know if this could be a problem or not.

    "No, since ri is not reset to 0 it won't loop through the queue from start. It'll simply continue where it left off, consuming only newly added sockets."

    Ugh, I got it wrong again. Sort of. What happens if there are no new sockets? It looks to me that you won't call Poll/BeginReceive again on existing sockets unless a new client connects. Sounds weird.

    • Marked as answer by exstud Wednesday, December 18, 2013 7:09 PM
    Wednesday, December 18, 2013 7:22 AM
    Moderator
  • >> It looks to me that you won't call Poll/BeginReceive again

    That put me on the right track to solve this. Here's what the asyncreceive looked like:

    public void AsyncReceive(IAsyncResult result) { // Console.WriteLine("AsyncReceive"); try { AsyncObject ao = (AsyncObject)result.AsyncState; SocketError se; int read = ao.sock.EndReceive(result);//, out se); // if (se != SocketError.Success) // read = 0; if (read > 0) { for (int i = 0; i < read; i++) ao.fullbuf.Add(ao.buf[i]); if (ao.contentlength < 0) { int indcl = ao.buf.FastIndexOf(patcontentlength, jtpatcontentlength); if (indcl >= 0) { int i = 0; for (; i < 10 && i + indcl + patcontentlength.Length < ao.buf.Length && Char.IsNumber((char)ao.buf[i + indcl + patcontentlength.Length]); i++) { } if (i < 10) { ao.contentlength = Convert.ToInt32(Encoding.UTF8.GetString(ao.buf, indcl + patcontentlength.Length, i)); } } } if (ao.contentlength >= 0) { if (ao.bodypos < 0) { ao.bodypos = ao.buf.FastIndexOf(patdoublelinebreak, jtpatdoublelinebreak); } if (ao.bodypos > 0 && ao.fullbuf.Count == ao.bodypos + 4 + ao.contentlength) { AsyncDone(ao); } else { ao.sock.BeginReceive(ao.buf, 0, ao.buf.Length, SocketFlags.None, AsyncReceive, ao); } } else { ao.bodypos = ao.buf.FastIndexOf(patdoublelinebreak, jtpatdoublelinebreak); if (ao.bodypos >= 0) { AsyncDone(ao); } else // no content-length or body, read again... { ao.sock.BeginReceive(ao.buf, 0, ao.buf.Length, SocketFlags.None, AsyncReceive, ao); } } // ao.sock.BeginReceive(ao.buf, 0, ao.buf.Length, SocketFlags.None, AsyncReceive, ao); } else { if (ao.fullbuf.Count > 0) AsyncDone(ao); else ao.sock.BeginReceive(ao.buf, 0, ao.buf.Length, SocketFlags.None, AsyncReceive, ao); } } catch (SocketException se) { Debug.WriteLine("async socketexception " + se.Message); } }

    For some reason it was looping the last (unnecessary) Beginreceive call, so it was (successfully) reading 0 bytes all the time.

    >> Hmm, did you omit code from Enqueue?

    Nope, it assumes the queue will never overflow, which is an assumption I made in the socket server as well (one which I'm now reconsidering thanks to your input). I could've used this class for the socket server instead of the circular arrays, but the CAS is only necessary when multiple writers/'producers' are involved. I'm using it in other places/projects(.net3.5) though and haven't had any issues with it, yet :).

    Thanks for your time and comments, it really helped getting through this one.


    • Edited by exstud Wednesday, December 18, 2013 8:08 PM
    Wednesday, December 18, 2013 6:20 PM