Replace("\u0000", System.String.Empty) will remove previous part of network message

Question

filepacket := (!filepacket).Replace("\u0000", System.String.Empty)

with above statement
816868213</ApplVerID>
      <ApplExtID>1003915839</ApplExtID>
      <CstmApplVerID>1211382902</CstmApplVerID>
      <SenderCompID>481139364</SenderCompID>
without above statement
<FIXML>
  <FIXMLMessage>
    <Header>
      <StandardHeaderID>469630316</StandardHeaderID>
      <BodyLength>97039197</BodyLength>
      <SecureDataLen>149055299</SecureDataLen>
      <MsgSeqNum>968228384</MsgSeqNum>
      <XmlDataLen>405485468</XmlDataLen>
      <HopGrpID>2093342714</HopGrpID>
      <SendingTime>168721401</SendingTime>
      <OrigSendingTime>705433440</OrigSendingTime>
      <BeginString>77979790</BeginString>
      <MsgType>1226209223</MsgType>
      <ApplVerID>1728469215</ApplVerID>
      <ApplExtID>1133849752</ApplExtID>

After remove it, missing part is recovered, however, sometimes it still miss

Answer 1

There's not enough context here to tell what is going on, and why there might even be (presumably spurious, because you want to remove them) NUL characters in your received XML in the first place.  Were they there in what you sent?

Assuming that this is in an application related to your previous questions about encrypting and decrypting data on the wire, I would suspect that there is something not quite right about your implementation of the transfer process at one end or the other, so what you are re-assembling doesn't quite match what you meant to send. 

Debugging crypto is fun because what you have in the middle is expected to resemble random noise; but you could try breaking the problem into parts by having an identity transform for the cryptographic transformation to make sure all the data -> bytes on the wire -> data is doing what you expect in the easy case; and then the plaintext bytes -> encrypted -> plaintext separately to make sure that what goes in comes out; before putting the two together.

Answer 2

i share it in skydrive, after unzip, run server first and then run client

press connect and then press send in client

https://skydrive.live.com/redir.aspx?cid=6d52b235d9f88afb&resid=6D52B235D9F88AFB!173&parid=6D52B235D9F88AFB!137

this is version of no encrypting, i try to do no encrypting first for building the protocol

sometimes it has the complete message, sometimes it lost the part of message,

originally remove null for parsing message directly,

now save it first and then parse the file, however, it has missing part

Answer 3

As a first step to looking at the source you provided, I found many systematic errors in amongst the 68 messages (I ignore the two dealing with the fact that I don't have the Excel interop installed) that the first test compile threw up.  Even though this is F#, you should still hew to the first of the Ten Commandments for C Programmers (where in this context "lint" translates to running with warning level 4 and warnings as errors)

There are a lot of

                current_state = ref OrdState.PartiallyFilled

with warnings

This expression should have type 'unit', but has type 'bool'. If assigning to a property use the syntax 'obj.Prop <- expr'.

and in this block (lines 1113-1117 of the server, repeated as 534-538 of the client)

    wStream.AsyncRead(buffer, 0, buffer.Length)
    wStream.FlushFinalBlock()
    mStream.GetBuffer()
    nStream.Write(buffer, 0, buffer.Length)
    nStream.Flush()

we have

Program.fs(1113,49): warning FS0020: This expression should have type 'unit', but has type 'Async<int>'. Use 'ignore' to discard the result of the expression, or 'let' to bind the result to a name.
Program.fs(1115,49): warning FS0020: This expression should have type 'unit', but has type 'byte []'. Use 'ignore' to discard the result of the expression, or 'let' to bind the result to a name.

The second of these looks potentially interesting, if you had ever meant to do anything with the stream buffer.

The next obvious bug in waiting is here

                                                    while(s.Read(buffer, 0, buffer.Length) <> 0) do                                           
                                                        nStream.Write(buffer, 0, buffer.Length)
                                                        nStream.Flush()

where you may write more than you actually read at each pass (what if the returned value from Read() is between 0 and the buffer length?) -- this may explain the spurious NUL characters, since you are recreating the buffer as an array full of zeroes at each pass.  I suspect that in the past you added the

                                                        buffer <- Array.zeroCreate 1024

at the end of the loop because you kept receiving extra garbage mixed into the message and making this change sort of made it go away, only now you had to deal with these NUL characters out of (apparently) nowhere.

This sort of brittle behaviour leads me to suspect that there are a lot more lurking issues buried in the clutter than this fairly superficial review has turned up.

The code base could benefit from significant refactoring -- putting common code into a shared source file, extracting repeated operations into little functions, being functional (turning the long imperative initialization of StandardHeader_dict and friends into a fold over the sequence of header element names would be more compact and expressive); from having some unit tests to stand as specifications for what the simple elements of the code are supposed to do; and from using a version control system instead of parking code by commenting it out.

Answer 4

1.
After tried
comment out the warning from ref current state and whole OrderStateMachine

and

changed to

while(s.Read(buffer, 0, buffer.Length) > 0) do  //  do not understand what problem with this

let mutable buffer : byte array = new byte[1024]  // can not compile
let mutable buffer : byte array = 0 // still sometimes have missing part of message
let mutable buffer : byte array = null // still sometimes have missing part of message

tried twice, first time is complete message, second time is missing part message

finally back to

buffer <- Array.zeroCreate 1024 // as i do not know method to new an byte array with length 1024

debug to run client and break at

nStream.Write(buffer, 0, buffer.Length)

only run step by step then server can receive a complete message, tried twice, all successful

the problem occur when not run step by step in debug mode

After tried

while(s.Read(buffer, 0, buffer.Length) > 0) do
                                                       
	let mutable i = buffer.Length - 1; 
	while buffer.[i] = Convert.ToByte(0) do
		i <- i-1 
	let mutable bar = Array.zeroCreate (i+1)
	Array.Copy(buffer, bar, i+1); 
                                                        
	nStream.Write(bar, 0, bar.Length)
	nStream.Flush()
	buffer <- Array.zeroCreate 1024

also have missing message

Answer 5

The lines with = rather than <- are doing nothing as they stand, so there should be no effect in commenting them out; but you may find that you really wanted to set those values.

>only run step by step then server can receive a complete message, tried twice, all successful

>the problem occur when not run step by step in debug mode

When you are stepping through manually, that gives plenty of time for the data to be sent across the network, and so you always get a full buffer until the last one (unless the message is an exact number of kilobytes). When running through at speed, you will be pulling data from the network faster than it's arriving, only reading partially full buffers, and because you are writing the whole buffer each time, will be getting spurious NUL characters from beyond the point where you read data.

For the code moving data between streams, you want to do something like this

let (+>) (input : Stream) (output : Stream) =
  let buffer = Array.zeroCreate<byte> 1024
  let rec move (input : Stream) (output : Stream) =
    let got = input.Read(buffer, 0, buffer.Length)
    if got > 0 then
        output.Write(buffer, 0, got)
        move input output
  move input output

which creates a buffer, then recursively reads as  much as possible from the input, then if it got anything, writes exactly what it read into the output and no more, before going round for more.

As a general point, in a functional language, the appearance of imperative constructs like while loops should be a flag to you to say "there is a better way to write this using this language" -- otherwise you might just as well use C# instead and not have to fight the language.

With the +> stream transfer operator thus defined, you can now simply write

s +> nStream

to replace the block I quoted (and the 3 other instances of that loop, two of them commented out, with variable names appropriately changed).

Answer 6

After tried to use this function, got the same missing message problem

i uploaded the latest version and it can be view and edited, i have never use this link to edit, maybe it can be uploaded

https://skydrive.live.com/redir.aspx?cid=6d52b235d9f88afb&resid=6D52B235D9F88AFB!174&parid=6D52B235D9F88AFB!137&authkey=!AAoStdLKToYtStc

does you mean that it run output.Write(buffer, 0, got) without finish input.Read(buffer, 0, buffer.Length)

i am trying to use AsynRead, and AsynWrite, so far got many compile error, incomplete structure

                                                    async {
                                                        let! nbytes = s.AsyncRead(buffer,0,buffer.Length)
                                                        //let count = 
                                                            //nbytes
                                                            //|> Array.fold (fun acc x -> if (x=space) then acc + 1 else acc) 0
                                                        while(nbytes > 0) do
                                                            //let encrypted_buffer = EncryptBytes(aesCSP, buffer)
                                                            //wStream.Write(buffer, 0, buffer.Length)
                                                            //wStream.Flush()
                                                            
                                                            let mutable i = buffer.Length - 1; 
                                                            while buffer.[i] = Convert.ToByte(0) do
                                                                i <- i-1 // now foo[i] is the last non-zero byte 
                                                            let mutable bar = Array.zeroCreate (i+1)
                                                            Array.Copy(buffer, bar, i+1); 
                                                            
                                                            nStream.Write(bar, 0, bar.Length)
                                                            //nStream.Write(bar, 0, bar.Length)
                                                            nStream.Flush()
                                                            buffer <- Array.zeroCreate 1024
                                                            let! nbytes = s.AsyncRead(buffer,0,buffer.Length)
                                                    }

Answer 7

I didn't promise that that was the only problem -- just that it was the first related one that I saw that would end up with you having spurious NUL bytes in the stream you're processing.

That async block could be simplified by not working out by hand the value you had been given in nbytes; I'm also dubious about the assumption that nbytes here is working like a mutable, rather than being a redeclaration in the inner scope.

Perhaps you want something along these lines for that clause

let (++>) (input : Stream) (output : Stream) =
  async {
    let buffer = Array.zeroCreate<byte> 1024
    let rec copy (input : Stream) (output : Stream) =
      async { let! nbytes = input.AsyncRead(buffer, 0, buffer.Length)
      if nbytes > 0 then
        do! output.AsyncWrite(buffer, 0, nbytes)
        do copy input output
        } |> Async.StartImmediate        
    do copy input output
  } |> Async.StartImmediate  

again only writing what you know you've read, and using tail recursion rather than iteration.

Answer 8

After using this code, having to remove s.Close() at the end of this thread function due to close closed file error

then run it, the received file is blank file