locked
How to filter and parse the live capture data. RRS feed

  • Question

  • hi all

    I am a C# programmer, My scenario is UDP communication between two machines, machine one will sent to the specified port on machine two, for example, send data to 192.168.1.5 : 20000. So, my purpose is only to get port 20000 data.

    I am following the examples in the help file, Yes,  CaptureCallbackDelegate had been called, I get the data. I had added the filter,and there are no fields added. However, all data had been captured.  The filter did not work.

    public long Start(uint _adapterIndex)
    {
      ret = NetmonAPI.NmOpenCaptureEngine(out myCaptureEngine);
      // Initialize the parser engine and return a frame parser.
      myFrameParser = MyLoadNPL();
      uint CapSize;
      ret = NetmonAPI.NmCreateCaptureFile("temp_capfiltlive.cap",
                    uint.MaxValue, NmCaptureFileFlag.WrapAround,
                    out capFile, out CapSize);
      // Configure the adapters callback and pass capture handle as context value.
                ret = NetmonAPI.NmConfigAdapter(myCaptureEngine,
                    adapterIndex, _captureCallbackDelegate, capFile,
                    NmCaptureCallbackExitMode.DiscardRemainFrames);
    }

    private IntPtr MyLoadNPL()
    {
        uint ret = NetmonAPI.NmLoadNplParser(null,
                    NmNplParserLoadingOption.NmAppendRegisteredNplSets,
                    _parserCallBack, IntPtr.Zero, out myNplParser);           
        ret = NetmonAPI.NmCreateFrameParserConfiguration(myNplParser,
                   MyParserBuild, IntPtr.Zero, out myFrameParserConfig);               
        //filter
        ret = NetmonAPI.NmAddFilter(myFrameParserConfig,
                   "Tcp.Port == 20000 or Udp.Port == 20000",
                   out myHTTPFilterID);
        ret = NetmonAPI.NmCreateFrameParser(myFrameParserConfig, out myFrameParser,
                  NmFrameParserOptimizeOption.ParserOptimizeNone);  
        return myNplParser;
    }

    Q1: How to make the filter "Tcp.Port == 20000 or Udp.Port == 20000" work? what am I doing wrong or something missed?  if the filter works, only port 2000 data can be captured? means only matched data can be captured and call CaptureCallbackDelegate .

    Q2: Can I parse the data in CaptureCallbackDelegate without add it to the frame. And how?

    Q3: Maybe the data must be parsed through NetmonAPI.NmGetFieldValueString. I must call NetmonAPI.NmAddFrame(mycapFile, hRawFrame), and then parse data from mycapFile.

          3.1 if the program runs for a long time, the frame data will be replaced, how can I know and get the new add data?

          3.2 how can I parse the data which are send by machine one? Maybe through the field? which field can I use?


    Best Regards Ethan
    Tuesday, September 27, 2011 2:54 PM

Answers

  • For your previous questions:

    1. The callback will receive all frames, regardless of any filters you've added using NmAddFilter.  The only way to filter the frames in teh driver is using the Driver Filtering Functions (see examples and reference is help under API Functions.)

    2. The frames are buffered for you, so in most low load situations you should not have to worry about dropping frames.  However in high load there is always a possiblity.  At some point network trafic can overwhelm the ability of the machine to capture frames reliably.

    3. You can't use High Perf filters in the API in exactly the same way.  You can manually specify the high perf parser set and use fully qualified paths.  But we still buffer the frames and that particular funcionallity is not availalbe from the UI.  Also offset pattern matches using BLOB will also work (mentioned in the Capturing at Bootup blog and High Perf Filtering blog).  However, as I described in #1, we have a way to perform more flexible filtering, though it is more complex to setup, but gives you even more flexibility and speed.

    4. I think your last two replies are all related.  I believe you want to get the data in the UDP payload.  The payloads don't really have an associated field because sometimes they are consumed by other layers.  Instead, you can get the offset and size of the UDP payload and use NmGetParitalRawFrame to grab the payload manually.  I have a blog that describes how to do this for TCP.  You should be able to use the same method to do this for UDP.

    Thanks,

    Paul

    • Marked as answer by Ethan xia Friday, November 18, 2011 2:14 PM
    Wednesday, October 5, 2011 5:59 PM
  • appreciate your help.

    it had been fixed.

    thanks a lot


    Best Regards Ethan
    • Marked as answer by Paul E Long Tuesday, July 16, 2013 4:19 PM
    Friday, November 18, 2011 2:14 PM

All replies

  • When you add a filter to the configuration, you are not actually applying a filter.  You are only building a filter that you could use as you evaluated a parsed frame.  This allows you to create many different filters in your parser, but only apply the one you want.  You evaluate the filter using NmEvaluateFilter against a parsed frame.

    You can parse the frame in the callback by taking the raw frame and using NmParseFrame using the frame parser you created in MyLoadNPL.  Just keep in mind that if you have a lot of data, and you can't keep up, frames could be dropped.  If that's the case, saving to a file might be better.  You might also consider the high performance filtering the API provides.

     

    NmGetFieldValueString is one way to get parsed data.  This call requires a parsed frame which you get by calling NmParseFrame.  NmParseFrame requires a raw frame which can come from the callback or a file.

    Let me know if you still have more questions.

    Thanks,

    Paul

    Tuesday, September 27, 2011 5:59 PM
  • Hi Paul
     Appreciate your quickly response.
     I had tested with your suggestion. Yes, it works. In call back function, I get the raw frame, use the NmParseFrame to get hParsedFrame, then it is succeed to filter data with function NmEvaluateFilter.
       There are still serials questions, please offer you insight.
    1. The filer works when I call NmEvaluateFiler. Means, the captureCallBackDelegate will always be called without filter. In callback we filter data with NmEvaluateFilter. Does system work like this way?
    2. If I don’t use AddFrame, the capture file will never enlarge? Means, I can ignore the capture file, and the frames will never be dropped?
    3. Yes, I will have a lot of data. There is quick response requirement, maybe save data to the file is not an appropriate way. How can I use the high performance filtering with API? It will be perfect if there is an example.
    4. If there is no field added to the frame, does the NgGetFieldValue* still work? If not, where is the data fields list? In my stance, I need remote pc IP and the binary data. How can I get them?

    Thanks!


    Best Regards Ethan
    Saturday, October 1, 2011 2:15 PM
  • Hi Paul

        I try to get data from parsed frame though NmGetFieldValueString. The number 160 always be returned. There are somthing wrong?

        In MyLoadNPL, two fields have been succeed to add.

    ret = NetmonAPI.NmAddField(myFrameParserConfig, "Blob", out blob);
    ret = NetmonAPI.NmAddField(myFrameParserConfig, "TCP.DstPort", out myHTTPFieldID);

    blob =0 and myHTTPFieldID = 1.

         In call back function,

         ret = NetmonAPI.NmParseFrame(myFrameParser,
                    hRawFrame, FrameCount,
                    NmFrameParsingOption.None,
                    out phParsedFrame, out phInsertedRawFrame);

         The phParsedFrame has been succed to get from hRawFram.

         And then,

         ret = NetmonAPI.NmEvaluateFilter(phParsedFrame, myHTTPFilterID, out passed);

         It succed to know if the data passed filter.

         However, when I try to get data though field, ERROR.

         if (passed)
                {
                    unsafe
                    {
                        char[] charBuffer = new char[bufferLength];
                        fixed (char* pchar = charBuffer)
                        {
                           ret = NetmonAPI.NmGetFieldValueString(phParsedFrame, bufferLength, bufferLength, pchar);
                            ret = NetmonAPI.NmGetFieldValueString(phParsedFrame, myHTTPFieldID, bufferLength, pchar);
                        }
                    }
                }

    ret is 160, means ERROR_BAD_ARGUMENTS. bufferLength is catched from hRawFrame through NmGetRawFrameLength.

    My question is how to get data? or I did something wrong? Please give me a hand!


    Best Regards Ethan
    Sunday, October 2, 2011 2:55 PM
  • Hi Paul

        Refer the NMTopUsers, I tried to use NmGetParsedFieldInfo, then NmGetFieldValue*.

        However, I still get nothing of UDP data because parsedField.FieldBitLength always equal zero. 

        How to get Binary data from the remoting PC? 

        Please give me some suggestion.


    Best Regards Ethan

    • Edited by Ethan xia Wednesday, October 5, 2011 1:47 PM
    Tuesday, October 4, 2011 2:49 PM
  • For your previous questions:

    1. The callback will receive all frames, regardless of any filters you've added using NmAddFilter.  The only way to filter the frames in teh driver is using the Driver Filtering Functions (see examples and reference is help under API Functions.)

    2. The frames are buffered for you, so in most low load situations you should not have to worry about dropping frames.  However in high load there is always a possiblity.  At some point network trafic can overwhelm the ability of the machine to capture frames reliably.

    3. You can't use High Perf filters in the API in exactly the same way.  You can manually specify the high perf parser set and use fully qualified paths.  But we still buffer the frames and that particular funcionallity is not availalbe from the UI.  Also offset pattern matches using BLOB will also work (mentioned in the Capturing at Bootup blog and High Perf Filtering blog).  However, as I described in #1, we have a way to perform more flexible filtering, though it is more complex to setup, but gives you even more flexibility and speed.

    4. I think your last two replies are all related.  I believe you want to get the data in the UDP payload.  The payloads don't really have an associated field because sometimes they are consumed by other layers.  Instead, you can get the offset and size of the UDP payload and use NmGetParitalRawFrame to grab the payload manually.  I have a blog that describes how to do this for TCP.  You should be able to use the same method to do this for UDP.

    Thanks,

    Paul

    • Marked as answer by Ethan xia Friday, November 18, 2011 2:14 PM
    Wednesday, October 5, 2011 5:59 PM
  • Appreciate your suggestions.

    It is success to filter data use "(Frame.Ethernet.Ipv4.UDP.Port == {0} OR Frame.Ethernet.IPv4.TCP.Port == {1})".

    There is still one more question. After run for a while, following function has been called.

    ret = NetmonAPI.NmParseFrame(myFrameParser,
                        hRawFrame, FrameCount,
                        NmFrameParsingOption.None,
                        out phParsedFrame, out phInsertedRawFrame);

    An error appeared, its number is 1283. The upper limit of a resource has been reached.

    Maybe it is better way to save data into the file and read it from file. And you had mentioned it on 27/9.

    How to save the data to the file? I had used NetmonAPI.NmAddFrame(capFile, hRawFrame), but, I believe there is nothing had been written on my disk because I cannot find the capFile named temp_capfiltlive.cap.

    What is the way to read data?

    Thanks a lot!


    Best Regards Ethan
    Thursday, October 6, 2011 3:01 PM
  • The error is most likely because you are not closing all of your handles.  Make sure to close all handles for parser, raw frames and any other objects.  There's a limit deifned by the API of around 1000 for raw and parser handles.  So once that limit is reached, we'll return this error.

    Yes, using NmCreateCaptureFile, NmAddFrame and NmCloseHandle should allow you to write the capture file.  If the file is not there, then perhaps it doens't have any frames OR it was never closed properly.

    Paul

    Thursday, October 6, 2011 3:24 PM
  • appreciate your help.

    it had been fixed.

    thanks a lot


    Best Regards Ethan
    • Marked as answer by Paul E Long Tuesday, July 16, 2013 4:19 PM
    Friday, November 18, 2011 2:14 PM
  • Hi Ethan

    Please tell me how you fixed this, I have been sitting with this same issue for a while now.


    MCTS

    Wednesday, July 3, 2013 3:01 PM