none
HCK test matrix: Do we really need that many machines?? RRS feed

  • Question

  • HCK Certification, for me, requires that I test our device on all the supported operating systems.

    Let's say, for example, that I want to complete an HCK certification run as fast as feasible, with as few servers as possible.  My hardware device fits inside the server, and I have more than one version of the device, each with a different Sub Vendor ID, as listed in Hardware Manager.

    So, if I have three slightly different hardware devices, and I plan to test on the following OS versions:

    Server 2008 R2, 2012 R1, 2012 R2

    to test all these simultaneously requires 3 x 3 = 9 machines.

    My test matrix is a lot bigger than that, currently over 40, and that's a lot of machines for certification!

    Each unique device ID, tested against each OS version, and in my case, multiple OEMs, makes it a bit overwhelming.  One solution is to buy a lot of machines.  Another fix is to get enough machines to test against Server 2012 R2, and then when that is finished, re-provision the machines to test against Server 2012 R1, and repeat for Server 2008 R2, Windows 8.0, 8.1…  You get the idea.

    I looked into the distributed testing, but I guess that just shortens an 8 hour certification test run at the cost of more machines.

    I may be overlooking something...

    Can I complete certification in a more compact manner?  Can it cover the entire matrix?  Can it be fast?  Can I test multiple devices in one server at the same time?

    Any input would be helpful.  Thanks!

    Thursday, January 2, 2014 11:56 PM

All replies

  • Do the "slightly, different hardware devices" use the same driver? If so, consider using Device Families to associate dissimilar hardware for distributed testing. See http://msdn.microsoft.com/en-us/library/windows/hardware/dn260295.aspx and http://msdn.microsoft.com/en-us/library/windows/hardware/jj123816.aspx for further information.

    John -- This posting is provided "AS IS" with no warranties, and confers no rights.

    Friday, January 3, 2014 6:08 PM
  • John,

    Thank you.  I'll explain a little better:

    Let's say that I make video cards.  The Hardware ID consists of four parts:  Vendor ID, Device ID, Subvendor ID, and SubDevice ID.

    Let's say my Vendor ID is 101A,

    Device IDs are 101B and 102B, depending on whether it is a basic or advanced video card.

    SubVendor IDs are 101C, 102C, 103C, and 104C, depending on whether it is resold as OEM(NiftyVideo), OEM(SUPERGraphics), OEM(ColorPro), or OEM(HDMIMaster, Inc.).

    and SubDevice ID is 101D, 102C, or 103B, depending on the model that the OEM partner sells (more RAM or faster chip)

    My hardware ID, for example, might be PCI\VEN_101A&DEV_101B&SUBSYS_101D101C&REV_01.

    So, I have:

    1 Vendor ID,

    2 Device IDs, 

    4 Subvendor IDs,

    3 Subdevice IDs.

    All my video cards use the same driver binary, but for each OEM, there is a unique brand of that driver.

    As I currently understand Hardware Certification, for each supported Windows OS, we must obtain full a full passing HCK run for each unique Vendor ID, Device ID, and Subvendor ID.  So 1x2x4 = 8 unique card types x 6 operating systems (let's say: Win8.1, 8, 7, Server 2012R2, 2012R1, and 2008R2) = 48 unique full certification runs that must be performed in order to certify all my video cards.  Is that true?

    Sounds like distributed testing may help me reduce the number of machines I need to use during a certification run (I think).  How would I want to group my cards (by vendor ID, device ID, and/or sub vendor ID) and how many machines would I need to complete a valid certification run in one pass?  Please feel free to include the example, I'm sure it will help a lot.

    Thanks!

    Tuesday, January 7, 2014 12:22 AM