none
Benchmarking ASMX vs. WCF

    Question

  • Hi,

     

    Great to see StockTrader 2.0 is out, i downloaded and installed it and it seems pretty good.

     

    One thing I would really like to see in the benchmarking is how WCF compares with ASMX. Most of the projects I work with and customers I talk to are already using the Microsoft platform. The issues we have are not convincing them to use Microsoft, but convincing them to make the switch from ASMX to WCF for services. There is quite a bit of resistance to this, as the team usually knows ASMX, and using WCF requires learning the tectnology, and the risk of adding to the development budget.

     

    It would be great to see how ASMX compares to WCF in the stock trader application, and the advantages and disadvantages of each. Development time, architecture, maintainability, hosting could be considered as well as performance and cost per transaction.

     

    Regards,

     

    Alan

     

     

    Wednesday, June 04, 2008 9:25 AM

All replies

  • This is a good request....I will try to augment....FYI I have run those tests; in almost all cases ASMX perf is about the same as WCF perf (or within 10%, with ASMX up to 10% faster than an IIS-hosted WCF service, depending on the number of cores in the server)...as long as the host is an IIS host for the WCF services;  if a self-host for WCF (console, windows, or NT service); you will likely see WCF a bit better especially on servers with 4 cores or less.

     

    based on my observations with StockTrader benchmarks, which does include and ASMX endpoint (you can use Capacity Planner tool to test for yourself, if you want to compare ASMX to WCF (either IIS-hosted or self-hosted the tool is designed to do just that)...

     

    BUT......while this shows raw performance is about equivalent, and not the reason to choose to upgrade to WCF...the REAL resons to change from ASMX to WCF are the following (and some of them DO have to do with performance):

     

    1)  WCF is Microsoft's strategic communication technology for .NET.  ASMX is supported; but is not the future.

    2)  ASMX only supports SOAP 1.1/1.2, and only works for communication with http based services, hosted in IIS

    3)  WCF consolidates this basic SOAP support, but adds a ton of new capabilities (yes, more choices does mean a bit more learning, but hopefully apps like StockTrader, and samples in Win SDK help a lot here):

      • Support for WS-* (in fact, the most advanced support I believe in the industry for the latest published standards)...this means ability to use message security for services where intermediaries might be involved; secure conversation support; ability to use X509 certs (service and client) for encryption and authentication; more complete support for windows auth; card space support, support for WS-Reliable Messaging, WS-AtomicTransactions, etc....basically, support for latest greatest WS-* standards that do interop with non-MS platforms for more advanced SOA setups requiring better security, reliability...ASMX does not support WS-* like this.
      • Ability to use many different protocols/encoding stds, beyond text-xml and MTOM.  For example, using with net.tcp and binary encoding...a pure replacement for previous .NET Binary Remoting...which, for .NET based clients talking to .NET based services (typically within intranet); provides a pure remoting architecture reducing serialization costs and network bandwidth for .NET to .NET services....
      • Ability to host non-http endpoints (like tcp mentioned above) at same time as hosting http/SOAP or http/WS-* endpoints.....so host can listen on many different types of endpoints at same time, letting .NET clients connect up through net.tcp/binary; but still letting Java/PHP/Whatever clients connect up through http/SOAP or http/WS-*.
      • Support for building and consuming REST-based services (with .NET 3.5); POX/JSON etc all now supported; so its not JUST about SOAP and WS-*.....
      • Much finer control over entire net stack, including protocol settings, connection pooling, being able to build custom bindings, etc.

    I think there is a bit of a learning curve...well, I know there is, I went through it myself... BUT, if you are starting with just an ASMX SOAP set of services, its pretty easy to create the same service (or client) in WCF; becuase its just basicHttpBinding; notations are almost the same, etc.  At that point, learning curve gets steeper for more advanced scenarios with WCF (like message security, WS-* and interop; but these things are *possible* and designed into WCF to support; while with ASMX you are at a dead end.....

     

    -Greg

     

     

     

    Wednesday, June 04, 2008 8:36 PM
  • Hi,

     

    Thans for that Greg, some great points.

     

    One thing we are also keen to look at is hosting in WAS in IIS7. Have you looked at this, or do you plan to. I have heard reports (though not tested it myself) that IIS7 and WAS will be a big boost to performance for WCF services.

     

    Regards,

     

    Alan

    Thursday, June 05, 2008 8:26 AM
  • STockTrader elements hosted in IIS all use WAS hosting when installed on IIS7; for example, node endpoint is set to a net.tcp/binary endpoint....so I have done a lot with WAS hosting....but yet to run a real benchmark.  The key is net.tcp/binary can dramatically lower serialization costs (CPU) and network bandwidth....so its nice to have the option yet still host in an IIS worker process, with activation.....

     

    The WSTest benchmark + benchmarking tool included in the download could be used to do such a test, pretty quickly...I will be doing one and publishing, sometime this summer..

     

    -Greg

     

    Thursday, June 05, 2008 4:41 PM
  • A recent test I did (albeit in IIS 5.1) suggested that my ASMX service was about 22% faster than the same code exposed as WCF (wsHttp with no security). I would love to find out that I am somehow wrong, but .....
    Thursday, March 05, 2009 6:33 PM
  • They should be roughly equivalent, according to my latest tests; however this assumes:

     

    1)  if you are hosting within IIS, you should be running on IIS 6 and preferrably newer IIS 7 release (available on Windows Server 2008).

    2)  You should not compare a wsHttp binding in WCF to an ASMX service; ASMX does not even support wsHttp binding; its merely a SOAP 1.1/1.2 binding with limited/no support for WS-* (such as WS-Security).  So the more valid comparison is a WCF basicHttpBinding with no security.

    3)  If you are self-hosting your http-based WCF Service (not an option with ASMX); you would likely see the self-hosted WCF implementation over a basicHttpBinding be up to 30% faster still than when hosting WCF service within IIS.  However, I do believe IIS-hosting the service is the better way to go, given auto-start nature of worker processes; additonal management and security IIS adds to the table.

    4)  I would not expect to see an IIS-hosted WCF service significantly outperform ASMX even with a basicHttpBinding (should be about the same); but keep in mind:

        -WCF replaces ASMX as MS strategic web service stack and programming model.
        -Adds significant capabilities beyond anything ASMX can do:
            -wsHttpBinding; ws-* support; net.tcp binding; federation bindings for federated security; net.msmq binding; net.pipes binding, etc. 
      -Ability to host multiple endpoints on different protocols at same time
      -WAS hosting of non-http protocols within IIS 7 (net.tcp; msmq, etc)
      -just to name a few----there is just a ton of programming flexibility with WCF that you do not get with ASMX.


    -Greg


    Greg Leake, Microsoft
    Friday, March 06, 2009 12:50 AM
  • They should be roughly equivalent, according to my latest tests; however this assumes:

     

    1)  if you are hosting within IIS, you should be running on IIS 6 and preferrably newer IIS 7 release (available on Windows Server 2008).

    2)  You should not compare a wsHttp binding in WCF to an ASMX service; ASMX does not even support wsHttp binding; its merely a SOAP 1.1/1.2 binding with limited/no support for WS-* (such as WS-Security).  So the more valid comparison is a WCF basicHttpBinding with no security.

    3)  If you are self-hosting your http-based WCF Service (not an option with ASMX); you would likely see the self-hosted WCF implementation over a basicHttpBinding be up to 30% faster still than when hosting WCF service within IIS.  However, I do believe IIS-hosting the service is the better way to go, given auto-start nature of worker processes; additonal management and security IIS adds to the table.

    4)  I would not expect to see an IIS-hosted WCF service significantly outperform ASMX even with a basicHttpBinding (should be about the same); but keep in mind:

        -WCF replaces ASMX as MS strategic web service stack and programming model.
        -Adds significant capabilities beyond anything ASMX can do:
            -wsHttpBinding; ws-* support; net.tcp binding; federation bindings for federated security; net.msmq binding; net.pipes binding, etc. 
      -Ability to host multiple endpoints on different protocols at same time
      -WAS hosting of non-http protocols within IIS 7 (net.tcp; msmq, etc)
      -just to name a few----there is just a ton of programming flexibility with WCF that you do not get with ASMX.


    -Greg


    Greg Leake, Microsoft

    Hello there, I am in the process of comparing performance between asmx and http/tcpip wcf service (self hosted in a windows service) all without security.  Would really love to see the 30% performance gain you mentioned but unfortunately the difference is minimal and sometimes asmx is faster.  Could i be doing anything wrong?  I was really looking foward to see a huge difference but the result i am seeing is quite disappointing.  Thanks in advance for any info.

    Wednesday, April 01, 2009 5:24 PM
  • Are you testing a self-hosted WCF implementation, or an IIS-hosted?  Self-hosted (over http) is typically quite a bit faster than IIS-hosted services when using WCF (perhaps 25-30%).  However, as pointed out above, I believe IIS-hosting is the better option anyway.

    I am sure relative results might vary based on message size, document style, etc.  I know results also can vary based on number of CPUs/cores in the system.  But, with that said, I just did some quick tests on .NET StockTrader Web Services using the Capacity Planner tool.  For this workload, at least, on a single quad-core system (3.0 GHz); I got the following results (.NET 3.5/SP1; windows server 2008/IIS7):

    WCF-iis-hosted
    6,952 requests/sec

    WCF Self-host
    8,228 requests/sec

    asmx-iis
    6,460 requests/sec

    Again, this is for a specific workload (StockTrader); with a specific data contract and service contract.  Also, the differences in the results are a bit muted given not all the work in this test is Web Service/serialization; there is business logic and DB interactions on every request---but this would be more typical of real-world anyway.

    -Greg


    Greg Leake, Microsoft
    Thursday, April 02, 2009 3:19 AM
  • Hello Greg,

    The service is self- hosted in a windows service exposed as a TCPIP endpoint.  We have not upgraded to windows 2008/iis7 yet, i have only tested on XP(iis5) and Windows server 2003/IIS 6 on a single dual core system:

    WCF (TCPIP) - Self Hosted in windows service (XP and Win2003)
    vs
    Asmx - iis 6 (XP and Win2003)

    You may be wondering why I am comparing asmx to WCF (TCPIP), to me WCF-TCPIP should beat asmx and I should see benchmark results similar to what you have in your post (looks really good by the way), but then again that was not what i was seeing, asmx sometimes beat wcf(tcpip) which I am very curious to find out why.  Maybe i am just not testing it the right way.  The way I am testing it does not involve any DB interactions, basically from the client, an integer parameter is sent to the service, the service then base on the number passed in, returns x number of an employee (name/address/etc which is defined as a datacontract) objects to the client.  So i have the option to increase or reduce the message size, but the size does not seem to make much difference as far as i can see.

    So I guess, the more CPU, the bigger the difference?  But I should still see a diff with a dual core, you think?

    Thank you so much for your response!!
    Friday, April 03, 2009 4:49 PM
  • The data above in my post is actually just for http.  ASMX of course is not able to do tcp/ip with .NET binary encoding.  WCF is;

    If you are using tcp/ip with binary encoding and comparing to http/asmx; you are right, tcp/ip/wcf should blow asmx away given both the transport protocol and the fact the encoding is binary, not xml, so lower serialization cost.  I find StockTrader services over tcp/ip; which can be tested using the capacity planner in the stocktrader kit; about 4x better throughput than wcf-http. 

    However, tcp is also a bit tricker to work with, given its more stateful nature.  Some things to watch out for that might throw off your perf numbers:

    1)  On client side; make sure to call channel.open or client.open depending on whether you are using Client<T> or ChannelFactory<T>; before making calls.
    2)  On client side; make sure you call channel.close/client.close when done using
    3)  However--look at the WSTest or StockTrader client in the Capacity Planner Agent Host; you will notice on the client single shared channel is used across all running threads; this reduces CPU consumption on clients letting fewer clients drive the server to saturation; yet the client/channel logic is still thread-safe.
    4)  Make sure you are driving the server to 100% CPU saturation in all test cases. Until you hit this, the server is not saturated and can produce more tps with more client load assuming no external bottleneck (clients might be an artificial bottleneck; network; database; etc).
    5)  By default a tcp binding uses .NET Binary encoding; but you might make sure this is the case and no override is there.
    6)  Are the security settings the same between asmx and WCF in the test?  Does no good for a comparison to have either transport or message level security set for WCF; but no security for ASMX.....ASMX is also not even capable of message-level security.


    I am curious why you are getting the results you are getting; you might try setting up a simple test with the .NET Capacity Planner (either stocktrader services or WSTest pure web service benchmark) and you will see tcp/ip endpoints way outperform wcf-http endpoints and/or ASMX.

    -Greg


    Greg Leake, Microsoft
    Friday, April 03, 2009 5:15 PM
  • Should have mentioned I am very new to wcf, this is my first WCF project.  Quite frankly I am not familiar with using ChannelFactory and the only way I know how to call a wcf service from my client is to add a service reference, and call the methods, as such:

    myServiceReference.
    ContractClient myTCPService = new myServiceReference.ContractClient("epTcp");

    myServiceReference.

    Employee[] employees = myTCPService.GetEmployee(rowCount);

     

    So I am not doing any channel.open or close, could that be why?! 

    And yes the security is turned off for the WCF sservice while doing my performance test.  Can you tell me where to look to see if the tcp binding is using binary?  Cant find it in the configuration.

    Thanks again for your response, very much appreicated.

    Friday, April 03, 2009 6:16 PM
  • Using ChannelFactory and channels created from the ChannelFactory is an alternative to using the generated client class; but most users use the generated client class which is fine. 

    First, I would try the following:

    myServiceReference.ContractClient myTCPService = new myServiceReference.ContractClient("epTcp");

    myTCPService.Open();

     

    Employee[] employees = myTCPService.GetEmployee(rowCount);

    myTCPService.Close();



    In your client logic.  A tcp binding by default will use .NET binary encoding; so you should be ok there.  You might try the following binding on host and client side; but mostly make sure when you record a peak tps rate for asmx and wcf/tcp-ip that you truly have pushed the server to 100% (or near) CPU utilization measured with perfmon.:

    <binding name="My_TcpBinding" closeTimeout="00:01:00" openTimeout="00:01:00" receiveTimeout="00:30:00" sendTimeout="00:1:00" transactionFlow="false" transferMode="Buffered" transactionProtocol="OleTransactions" hostNameComparisonMode="StrongWildcard" listenBacklog="500" maxBufferPoolSize="4194304" maxBufferSize="524288" maxConnections="500" maxReceivedMessageSize="524288">

    <

     

    readerQuotas maxDepth="512" maxStringContentLength="262144" maxArrayLength="262144" maxBytesPerRead="32768" maxNameTableCharCount="262144"/>

    <

     

    reliableSession ordered="true" inactivityTimeout="00:30:00" enabled="false"/>

    <

     

    security mode="None">

    <

     

    transport clientCredentialType="None" protectionLevel="EncryptAndSign"/>

    <

     

    message clientCredentialType="Windows"/>

    </

     

    security>

    </

     

    binding>



    Greg Leake, Microsoft
    Friday, April 03, 2009 6:58 PM
  • Added the open and close, still same result, asmx is faster half the times.

    But, i didnt do what you suggested , to push the server to 100%, because not sure how (sorry not much benchmarking experience here).  Can i ask how do i set that up?  Another thing i didnt mention, the rowcount (the integer parameter sent from client to the service) will break the service if i increase it to a higher number so I am only able to test up to 1000000 rows but anything higher i will get a "The socket connection as aborted.  This could be caused by an error processing your message or a receive timeout being exceeded by the remote host" error.  In both client/host configuration, i have modified the timeout and messageSize values to be 10 minutes and 2147483647.  So anyways, I havent been able to test with as many rows as i wanted.  I know that is not really what you meant by pushing the server to 100% but in any case, would love to learn how if you do not mind sending me some instructions. 

    Thanks so much Greg.

    Friday, April 03, 2009 7:43 PM
  • It would be helpful to understand your testbed setup....is one machine driving load, against a remote service host?

    It looks like you are testing just the response time, possibly from a single request to a service op that returns a potentiall huge serialized object?

    Its important to know whether you have separated client from service via a network.  If you are just measuring response time for a single request; keep in mind you are testing a combination of service and client performance (both have serialization work to do).  In general, a better way to test, in my opinion is to try to get at the root question:

    ---->How many requests per second (throughput) can a given machine/host workload support?

    in finding this answer, you are typically also answering the question:

    ---->How many concurrent users can my setup support?

    In general, given proper threading, a service with a lower response time will take up less CPU cycles per request (in and out faster); and hence support more concurrent users and higher # of requests per second.  An exception to this rule, however, would be if the software stack does not spawn enough threads to get CPUs up to 100% utilization, for example (but neither ASMX or WCF should have this issue on a typical hardware setup).

    The tests I publish do not measure client-side performance; rather are designed to measure service host (ASMX or WCF or Java, whatever) performance.  This is done by having multiple remote client machines, each running many many client threads, hammer the server up to, hopefully, 100% CPU utilization (at which point response times will *elbow* up, and additional users simply add to response times, and cannot produce any more responses per second (additional throughput).

    If you are measuring everything from a single client calling a single service op once, and that service op creates a huge serialized message response; you are not capturing the answers to the questions above; and in fact are comparing the two stacks in terms of serialization of a huge message (not lots of concurrent requests); and you want to make sure the message size you are testing is mapping to some reasonable estimate of avg message size for a real business workload you might be building.

    The tests I quote above show a peak TPS (requests per second rate) with clients having a 0.1 second think time between each request.  If a 'real' client (as oppsoed to bench client) actually has a 1 second delay between requests; that means the hardware/software setup on host side can actually support 10x the number of concurrent clients--giving you a decent estimate of how much capacity a given setup (hardware and software stack) provides.

    -Greg


    Greg Leake, Microsoft
    Tuesday, April 14, 2009 5:35 PM
  • Yes you guessed it right, I am testing from just one machine connecting to the service located on a remote machine, and yes just a single request.

    Thanks for the detail of the test setup, I realize mine compared to yours is quite different :)  Guess I do not have all the hardware required run a real benchmark test.  So if I am limited to just one client machine and one remote machine, will it be a better test (compared to my current setup) if i keep sending requests from the client to the remote until the server is up to 100% CPU utilization?

    An update to my test, I have tried instead of passing a dataset, pass a dataContract and finally I can see a difference between wcf vs asmx.  So my question for you is, does your wcf service used in the benchmark test uses dataContract serialization only?  If so, is it safe to assume wcf is only faster than asmx if dataContract serialization is used (over xmlSerialization)?

    Thanks again Greg, really appreciate your time!!
    Tuesday, April 21, 2009 5:58 PM