none
StreamReader over WCF problem

    Question

  • Hi there,
    This one has really got me. Spent all day reading forums on this type of exception but still cannot find a solution:

    {"There was an error while trying to serialize parameter http://tempuri.org/:class1. The InnerException message was 'Type 'System.Text.UTF8Encoding+UTF8Decoder' with data contract name 'UTF8Encoding.UTF8Decoder:http://schemas.datacontract.org/2004/07/System.Text' is not expected. Add any types not known statically to the list of known types - for example, by using the KnownTypeAttribute attribute or by adding them to the list of known types passed to DataContractSerializer.'.  Please see InnerException for more details."}

    I've created a WCF Service which runs on our LAN. It requires a StreamReader object in which to perform some file processing duties. However i'm unable to pass my object from the client because the service doesn't seem to know about encoding types. I've tried all sorts of variants. I know that you have to set attributes to set unknown types and believe me i've set virtually every combination of attribute possible, but it refuses to 'know' about this type, even though it knows enough to tell me what it doesn't know????

    I am completely stuck. I've even tried passing a FileStream instead but I can't then convert back into a a StreamReader.

    All I need is to implement a single method:

    public void Process(StreamReader reader)

    If I pass a simple type such as an int there's no problem. So it is just more 'complex' types I have a problem with.

    Am I on the wrong track with this? Is there a better way of moving small files over WCF? I have to process a variety of files including CSV, TXT, XLS etc.

    Thanks.

    Neil
    Friday, August 07, 2009 1:50 PM

Answers

  • In order to pass a stream to a WCF method, the Stream parameter must be the only parameter in the operation (or in the message body). That is to take advantage of the streamed transfer mode (where not all message is transferred at once), see the example below. If you don't use the streamed transfer mode, then the whole message (i.e., the file) is buffered in the client prior to being transferred over to the server, so if this is the case, then you can change your operation to take byte[] inputs, something like
        long TestMethod(byte[] file1, byte[] file2, byte[] file3)
    And when calling it:
        client.TestMethod(File.ReadAllBytes("c:\\testfile1.csv"), File.ReadAllBytes("c:\\testfile2.xml"), File.ReadAllBytes("c:\\testfile3.xsd"));

    This example shows a "file" transfer using the streamed transfer mode. Notice that the file can be really huge, and the memory usage will remain (fairly) flat in the client and the server.

        public class Post_7b9e9cd9_49f1_498c_85c5_fc869dddae87
        {
            [ServiceContract]
            public interface ITest
            {
                [OperationContract]
                long TestMethod(Stream file);
            }
            public class Service : ITest
            {
                public long TestMethod(Stream file)
                {
                    long result = 0;
                    while (file.ReadByte() >= 0) result++;
                    return result;
                }
            }
            static Binding GetBinding()
            {
                return new BasicHttpBinding()
                {
                    TransferMode = TransferMode.Streamed,
                    MaxReceivedMessageSize = long.MaxValue,
                };
            }
            class MyFakeStream : Stream
            {
                long length;
                public MyFakeStream(long length)
                {
                    this.length = length;
                }
                public override bool CanRead
                {
                    get { return true; }
                }
                public override bool CanSeek
                {
                    get { return false; }
                }
                public override bool CanWrite
                {
                    get { return false; }
                }
                public override void Flush() { }
                public override long Length
                {
                    get { throw new NotSupportedException(); }
                }
                public override long Position
                {
                    get { throw new NotSupportedException(); }
                    set { throw new NotSupportedException(); }
                }
                public override int Read(byte[] buffer, int offset, int count)
                {
                    long toReturn = Math.Min(this.length, count);
                    this.length -= toReturn;
                    return (int)toReturn;
                }
                public override long Seek(long offset, SeekOrigin origin)
                {
                    throw new NotSupportedException();
                }
                public override void SetLength(long value)
                {
                    throw new NotSupportedException();
                }
                public override void Write(byte[] buffer, int offset, int count)
                {
                    throw new NotSupportedException();
                }
            }
            public static void Test()
            {
                string baseAddress = "http://" + Environment.MachineName + ":8000/Service";
                ServiceHost host = new ServiceHost(typeof(Service), new Uri(baseAddress));
                ServiceEndpoint endpoint = host.AddServiceEndpoint(typeof(ITest), GetBinding(), "");
                host.Open();
                Console.WriteLine("Host opened");
    
                ChannelFactory<ITest> factory = new ChannelFactory<ITest>(GetBinding(), new EndpointAddress(baseAddress));
                ITest proxy = factory.CreateChannel();
    
                Stream myFile = new MyFakeStream(1234567L);
                Console.WriteLine("Size of the \"file\": {0}", proxy.TestMethod(myFile));
    
                ((IClientChannel)proxy).Close();
                factory.Close();
                host.Close();
            }
        }
    
    • Marked as answer by Neil Dobson Sunday, August 09, 2009 4:11 AM
    Saturday, August 08, 2009 4:21 PM

All replies

  • Hello Neil,

    The best way to transfer files over WCF is to use the Stream class as a parameter to the operation. Stream is treated ina special way, so you can pass any of its subclasses (i.e., FileStream) and it should just work.

    And with the Stream you can convert it back on the other side to a StreamReader (new StreamReader(Stream)).
    Friday, August 07, 2009 3:33 PM
  • Hello Carlos,
    Thankyou for your assistance. I initially tried a Stream - in fact one of the sublasses, a FileStream. And the method did not throw an exception when I passed a FileStream. However trying to convert the Stream to a StreamReader in the service is throwing exceptions.

    I am developing an ETL application which accepts a multitute of different file types and loads them into our backend SQL Server. These files can be anything from XML, XLS to proprietary files with no headers and fixed column widths. To accompany the import file is an XML mapping file which tells the processor how to handle the import file. In addition if the import file is XML then a schema should also be included. So for any given file type I will need to transfer between 2 and 3 files as a collection.

    So I created a method to pass three sesparate FileStream objects and return the length of the first one.

    [OperationContract]
    long TestMethod(FileStream stream1, FileStream stream2, FileStream stream3);

    public long TestMethod(FileStream stream1, FileStream stream2, FileStream stream3)
    {
        return stream1.Length;
    }

    And in my client:

    ServiceReference1.Service1Client client = new WindowsFormsApplication1.ServiceReference1.Service1Client();
    FileStream stream1 = new FileStream("C:\\TestFile1.csv", FileMode.Open);
    FileStream stream2 = new FileStream("C:\\TestFile2.xml", FileMode.Open);
    FileStream stream3 = new FileStream("C:\\TestFile3.xsd", FileMode.Open);
    long streamLen = client.TestMethod(stream1, stream2, stream3);
    MessageBox.Show(streamLen.ToString());

    However when I run the app an exception is thrown which internally is a NullReferenceException. I have checked and each of the streams are open and locally return me properties including file length etc. And my service is running on the same PC.

    I think the reason I could not initially convert back to a StreamReader, like you mentioned, because the streams are not being transferred correctly. However I am struggling to debug the service and can only rely on return values to help me out.

    Any other suggestions?

    Thanks.

    Neil
    Friday, August 07, 2009 11:53 PM
  • Well i've spent 3 full days on this without any luck. I read in various places that you can only transfer single streams at once so I created a method to do this. And after attempting to convert the stream to a StreamReader or even write the file to disk I noticed that only 255 bytes were being transferred. After more research I tried adjusting the binding values including MaxReceivedMessageSize. This increased the transferred file size to a maximum of 4.24kb. No matter what else I adjust I can't transfer a file > 4.24kb. I tried buffered and streamed modes with no difference.

    I think i've spent too long on this. Its clearly not fit for purpose so i'll create a custom TCP messaging app to do the job instead.

    Neil
    Saturday, August 08, 2009 8:01 AM
  • In order to pass a stream to a WCF method, the Stream parameter must be the only parameter in the operation (or in the message body). That is to take advantage of the streamed transfer mode (where not all message is transferred at once), see the example below. If you don't use the streamed transfer mode, then the whole message (i.e., the file) is buffered in the client prior to being transferred over to the server, so if this is the case, then you can change your operation to take byte[] inputs, something like
        long TestMethod(byte[] file1, byte[] file2, byte[] file3)
    And when calling it:
        client.TestMethod(File.ReadAllBytes("c:\\testfile1.csv"), File.ReadAllBytes("c:\\testfile2.xml"), File.ReadAllBytes("c:\\testfile3.xsd"));

    This example shows a "file" transfer using the streamed transfer mode. Notice that the file can be really huge, and the memory usage will remain (fairly) flat in the client and the server.

        public class Post_7b9e9cd9_49f1_498c_85c5_fc869dddae87
        {
            [ServiceContract]
            public interface ITest
            {
                [OperationContract]
                long TestMethod(Stream file);
            }
            public class Service : ITest
            {
                public long TestMethod(Stream file)
                {
                    long result = 0;
                    while (file.ReadByte() >= 0) result++;
                    return result;
                }
            }
            static Binding GetBinding()
            {
                return new BasicHttpBinding()
                {
                    TransferMode = TransferMode.Streamed,
                    MaxReceivedMessageSize = long.MaxValue,
                };
            }
            class MyFakeStream : Stream
            {
                long length;
                public MyFakeStream(long length)
                {
                    this.length = length;
                }
                public override bool CanRead
                {
                    get { return true; }
                }
                public override bool CanSeek
                {
                    get { return false; }
                }
                public override bool CanWrite
                {
                    get { return false; }
                }
                public override void Flush() { }
                public override long Length
                {
                    get { throw new NotSupportedException(); }
                }
                public override long Position
                {
                    get { throw new NotSupportedException(); }
                    set { throw new NotSupportedException(); }
                }
                public override int Read(byte[] buffer, int offset, int count)
                {
                    long toReturn = Math.Min(this.length, count);
                    this.length -= toReturn;
                    return (int)toReturn;
                }
                public override long Seek(long offset, SeekOrigin origin)
                {
                    throw new NotSupportedException();
                }
                public override void SetLength(long value)
                {
                    throw new NotSupportedException();
                }
                public override void Write(byte[] buffer, int offset, int count)
                {
                    throw new NotSupportedException();
                }
            }
            public static void Test()
            {
                string baseAddress = "http://" + Environment.MachineName + ":8000/Service";
                ServiceHost host = new ServiceHost(typeof(Service), new Uri(baseAddress));
                ServiceEndpoint endpoint = host.AddServiceEndpoint(typeof(ITest), GetBinding(), "");
                host.Open();
                Console.WriteLine("Host opened");
    
                ChannelFactory<ITest> factory = new ChannelFactory<ITest>(GetBinding(), new EndpointAddress(baseAddress));
                ITest proxy = factory.CreateChannel();
    
                Stream myFile = new MyFakeStream(1234567L);
                Console.WriteLine("Size of the \"file\": {0}", proxy.TestMethod(myFile));
    
                ((IClientChannel)proxy).Close();
                factory.Close();
                host.Close();
            }
        }
    
    • Marked as answer by Neil Dobson Sunday, August 09, 2009 4:11 AM
    Saturday, August 08, 2009 4:21 PM
  • Hi Carlos,

    Thankyou for your help, all is working now.

    Two things I got wrong: Firstly, trying to transfer more than one stream at once, and secondly, assuming other persons submitted code works properly. The latter caused me grief as I used someone's method to save a stream to disk. And this had a bug in it which would occasionally only save a single buffer of data, which was why I was getting only 255 bytes.

    In the end the above, and the link below enabled me to get my app working.

    http://csharp-codesamples.com/2009/02/data-transfer-using-self-hosted-wcf-service/

    Thanks,

    Neil
    Sunday, August 09, 2009 4:12 AM