none
NVMe Passthrough for vendor specific commands returns error 1: ERROR_INVALID_FUNCTION RRS feed

  • Question

  • Hi

    I have closely followed the example on how to send a NVME Vendor Specific pass-through commands to the device, however i can't figure out why the ERROR_INVALID_FUNCTION code gets returned every single time.

    Im working with an NVME device which is developed in house (our own SSD/NVME controller chip) which has been tested and works under linux with no issues.

    I have also verified, that the device is able to send the Command Effects Log with all the commands that is supports, here is the output of this log queried under Linux environment:

    fpga@host:[~/WORK/nvme-cli/nvme-cli]: sudo ./nvme effects-log /dev/nvme0
    Admin Command Set
    ACS0     [Delete I/O Submission Queue     ] 00000001
    ACS1     [Create I/O Submission Queue     ] 00000001
    ACS2     [Get Log Page                    ] 00000001
    ACS4     [Delete I/O Completion Queue     ] 00000001
    ACS5     [Create I/O Completion Queue     ] 00000001
    ACS6     [Identify                        ] 00000001
    ACS8     [Abort                           ] 00000001
    ACS9     [Set Features                    ] 00000001
    ACS10    [Get Features                    ] 00000001
    ACS12    [Asynchronous Event Request      ] 00000001
    ACS16    [Firmware Commit                 ] 00000001
    ACS17    [Firmware Image Download         ] 00000001
    ACS128   [Format NVM                      ] 00000001
    ACS129   [Security Send                   ] 00000001
    ACS130   [Security Receive                ] 00000001
    ACS192   [Unknown                         ] 00000001
    ACS193   [Unknown                         ] 00000001
    ACS194   [Unknown                         ] 00000001
    ACS195   [Unknown                         ] 00000001
    ACS196   [Unknown                         ] 00000001
    ACS197   [Unknown                         ] 00000001
    ACS198   [Unknown                         ] 00000001
    ACS199   [Unknown                         ] 00000001
    ACS200   [Unknown                         ] 00000001

    NVM Command Set
    IOCS0    [Flush                           ] 00000001
    IOCS1    [Write                           ] 00000001
    IOCS2    [Read                            ] 00000001
    IOCS4    [Write Uncorrectable             ] 00000001
    IOCS5    [Compare                         ] 00000001
    IOCS8    [Write Zeroes                    ] 00000001
    IOCS9    [Dataset Management              ] 00000001
    IOCS13   [Reservation Register            ] 00000001
    IOCS14   [Reservation Report              ] 00000001
    IOCS17   [Reservation Acquire             ] 00000001
    IOCS21   [Reservation Release             ] 00000001
    IOCS131  [Unknown                         ] 00000001
    IOCS132  [Unknown                         ] 00000001
    IOCS133  [Unknown                         ] 00000001
    IOCS134  [Unknown                         ] 00000001

    I have based the test code i have for windows on the example from https://docs.microsoft.com/en-us/windows/desktop/FileIO/working-with-nvme-devices#pass-through-mechanism

    I'm able to send a standard IDENTITY command from the same example with no issues and getting a proper response.

    However, when i try to send a Vendor Specific Admin command with any command code from 0xC1-0xC6 i  get the error 1 (ERROR_INVALID_FUNCTION)

    Here is the code i use to send a Vendor Specific Admin command:

    BOOL nvme_vendor_admin_cmd(HANDLE hDevice)
    {
    
    	BOOL    result = 0;
    	PVOID   buffer = NULL;
    	ULONG   bufferLength = 0;
    	ULONG   returnedLength = 0;
    	PSTORAGE_PROTOCOL_COMMAND protocolCommand = NULL;
    	PNVME_COMMAND command = NULL;
    
    	//
    	// Allocate buffer for use.
    	//
    
    	bufferLength = FIELD_OFFSET(STORAGE_PROTOCOL_COMMAND, Command) +
    		STORAGE_PROTOCOL_COMMAND_LENGTH_NVME +  sizeof(NVME_ERROR_INFO_LOG) + NVME_MAX_LOG_SIZE;
    
    	buffer = malloc(bufferLength);
    
            if(!buffer)
    	  return FALSE;
    	//
    	// Initialize query data structure to get Identify Controller Data.
    	//
    	ZeroMemory(buffer, bufferLength);
    
    	protocolCommand = (PSTORAGE_PROTOCOL_COMMAND)buffer;
    	command = (PNVME_COMMAND)protocolCommand->Command;
    
    	protocolCommand->Version = STORAGE_PROTOCOL_STRUCTURE_VERSION;
    	protocolCommand->Length = sizeof(STORAGE_PROTOCOL_COMMAND);
    	protocolCommand->ProtocolType = ProtocolTypeNvme;
    	protocolCommand->Flags = STORAGE_PROTOCOL_COMMAND_FLAG_ADAPTER_REQUEST;
    	protocolCommand->CommandLength = STORAGE_PROTOCOL_COMMAND_LENGTH_NVME;
    	protocolCommand->ErrorInfoLength = sizeof(NVME_ERROR_INFO_LOG);
    	protocolCommand->DataFromDeviceTransferLength = NVME_MAX_LOG_SIZE;
    	protocolCommand->TimeOutValue = 10;
    	protocolCommand->ErrorInfoOffset = FIELD_OFFSET(STORAGE_PROTOCOL_COMMAND, Command) + STORAGE_PROTOCOL_COMMAND_LENGTH_NVME;
    	protocolCommand->DataToDeviceBufferOffset = protocolCommand->ErrorInfoOffset + protocolCommand->ErrorInfoLength;
    	protocolCommand->CommandSpecific = STORAGE_PROTOCOL_SPECIFIC_NVME_ADMIN_COMMAND;
    
    	command->CDW0.OPC = 0xC3; // opcode;
    	command->u.GENERAL.CDW10 = 256; 
    	command->u.GENERAL.CDW11 = 0;
    	command->u.GENERAL.CDW12 = 0;
    	command->u.GENERAL.CDW13 = 0;
    	command->u.GENERAL.CDW14 = 0;
    
    	//
    	// Send request down.
    	//
    	result = DeviceIoControl(hDevice,
    		IOCTL_STORAGE_PROTOCOL_COMMAND,
    		buffer,
    		bufferLength,
    		buffer,
    		bufferLength,
    		&returnedLength,
    		NULL
    	);
    
    	if (FALSE == result)
    	{
    		DWORD dw = GetLastError();
    		std::cout << "LastErrorCode: " << dw << "\n";
    		goto exit;
    	}
    	else
    	{
    		std::cout << "NVME Vendor Admin Command Success: " << dw << "\n";
    	}
    
    exit:
    
    	if (buffer)
    		free(buffer);
    
    	return result;
    
    }

    Please let me know what am i doing wrong since i need to get this to work with our device under windows.

    Any help is greatly appreciated



    Thursday, May 9, 2019 2:02 PM

Answers

All replies

  • Anyone from Microsoft care to comment? its been over 2 weeks and not a single reply :(
    Thursday, May 23, 2019 4:20 PM
  • Hello,

    A dev from the feature team reviewed this and statest that the code looks fine and asked does the vendor specific commands send down has corresponding command effects data from command effects log? If not, storport will block any command to the device as mentioned in https://docs.microsoft.com/en-us/windows/desktop/FileIO/working-with-nvme-devices#pass-through-mechanism

    Important:

    StoreNVMe.sys and Storeport.sys will block any command to a device if it is not described in the Command Effects Log. 

    Monday, June 10, 2019 7:55 PM
    Moderator
  • Ok so it looks like the pass-through started to work however we are facing another slew of issues.

    Sending the Vendor Specific commands work only if there is no data to be written to the NVMe device, what i mean by that is that we can send the Vendor Specific commands but as soon as we try to send a command that also required a PRP transfer from host to Device, command fails in 2 different ways.

    Here is the example code:

    DWORD data_len = 1024; 
    DWORD buffer_length = 0;
    DWORD data_buffer_offset = 0;
    int32_t                      retval = 0;
    PSTORAGE_PROTOCOL_COMMAND    p_protocol_cmd = NULL;
    PNVME_COMMAND                p_nvme_cmd = NULL;
    BOOL                         b_retval = FALSE
    
    buffer_length = FIELD_OFFSET(STORAGE_PROTOCOL_COMMAND, Command) +
                        STORAGE_PROTOCOL_COMMAND_LENGTH_NVME + sizeof(NVME_ERROR_INFO_LOG) + data_len;
    
    
    p_buffer = VirtualAlloc(NULL, buffer_length, MEM_COMMIT | MEM_RESERVE, PAGE_READWRITE);
      if (!p_buffer)
         return -1;
    
    ZeroMemory(p_buffer, buffer_length);
    
    p_protocol_cmd = (PSTORAGE_PROTOCOL_COMMAND)p_buffer;
        p_nvme_cmd     = (PNVME_COMMAND)p_protocol_cmd->Command;
    
    p_protocol_cmd->Version = STORAGE_PROTOCOL_STRUCTURE_VERSION;
    p_protocol_cmd->Length = sizeof(STORAGE_PROTOCOL_COMMAND);
    p_protocol_cmd->ProtocolType = ProtocolTypeNvme;
    p_protocol_cmd->Flags = STORAGE_PROTOCOL_COMMAND_FLAG_ADAPTER_REQUEST;
    p_protocol_cmd->CommandLength = STORAGE_PROTOCOL_COMMAND_LENGTH_NVME;
    p_protocol_cmd->ErrorInfoLength = sizeof(NVME_ERROR_INFO_LOG);
    p_protocol_cmd->TimeOutValue = 10;
    p_protocol_cmd->ErrorInfoOffset = FIELD_OFFSET(STORAGE_PROTOCOL_COMMAND, Command) + STORAGE_PROTOCOL_COMMAND_LENGTH_NVME;
        
    data_buffer_offset = p_protocol_cmd->ErrorInfoOffset + p_protocol_cmd->ErrorInfoLength;
    
    
    p_protocol_cmd->DataToDeviceBufferOffset = data_buffer_offset;
    p_protocol_cmd->DataToDeviceTransferLength = data_len;
    memcpy(&(p_buffer[data_buffer_offset]), p_user_data, data_len);
    
    p_nvme_cmd->CDW0.OPC = 0x83;
    p_nvme_cmd->u.GENERAL.CDW12 = <custom vendor value>;
    p_nvme_cmd->u.GENERAL.CDW13 = <custom vendor value>;
    p_nvme_cmd->u.GENERAL.CDW14 = <custom vendor value>;
    p_nvme_cmd->u.GENERAL.CDW15 = <custom vendor value>;
    p_protocol_cmd->CommandSpecific = STORAGE_PROTOCOL_SPECIFIC_NVME_NVM_COMMAND;
    /p_nvme_cmd->NSID = 1;
    
    b_retval = DeviceIoControl(
                                    device_handle,
                                    IOCTL_STORAGE_PROTOCOL_COMMAND,
                                    p_buffer,
                                    buffer_length,
                                    p_buffer,
                                    buffer_length,
                                    &returned_data_len,
                                    NULL
                                 );
    
    if (FALSE == b_retval) 
    {
      DWORD error = GetLastError();
      NVME_ERROR_INFO_LOG* ptrLog = (NVME_ERROR_INFO_LOG*)(p_buffer + FIELD_OFFSET(STORAGE_PROTOCOL_COMMAND, Command) + STORAGE_PROTOCOL_COMMAND_LENGTH_NVME);
    }
    
    

    In this case, the command always fails with ErrorCode=1117, IO Error.

    What we have tried:

    1. Setting AVSCC and NVSCC values in the identity Struct returned from the Device (in the device firmware obviously) to 1 and 0 in the Identity command trying to indicate compliance or not to standard NVMe Vendor Commands.

    2. Setting NSID (namespace ID in DW0 of the p_nvme_command) to 0, 1 and 0xFFFFFFFF indicating all the possibilities

    3. As per NVME spec, setting the p_nvme_cmd->u.GENERAL.CDW10 to 256 indicating a need to transfer 256 DWORDS which is precisely 1024 bytes we are trying to transfer.

    None of this worked, we always get an IO_ERRROR error code 1117.

    Next, we have discovered that commenting out the following line allows the command to go through

    p_protocol_cmd->Flags = STORAGE_PROTOCOL_COMMAND_FLAG_ADAPTER_REQUEST;

    however, even though command goes through, no PRP data transfers are done, even though data length and data offset are set in STORAGE_PROTOCOL_COMMAND struct.

    We also tried every possible combination mentioned above as well with no luck, no data is ever transferred period.

    Device has one namespace, so NSID should be 1 for all non-admin commands.

    Everything works fine with the OFA driver, but we want to get this working with standard windows driver since this is important to our customers.

    Please let me know what can be the issue, and how to possible resolve it, this is very important for us

    Thursday, June 27, 2019 3:01 PM
  • Hello,

    A dev from the feature team reviewed this and statest that the code looks fine and asked does the vendor specific commands send down has corresponding command effects data from command effects log? If not, storport will block any command to the device as mentioned in https://docs.microsoft.com/en-us/windows/desktop/FileIO/working-with-nvme-devices#pass-through-mechanism

    Important:

    StoreNVMe.sys and Storeport.sys will block any command to a device if it is not described in the Command Effects Log. 

    Hi Cymon,

    can you please pass the information i have posted below to your Dev team, its extremely important for us to get this working sooner than later.

    thank you

    Thursday, June 27, 2019 3:04 PM