none
Problem to read char RRS feed

  • Question

  • Hi,
    How to resolve

    System.ArgumentException was unhandled
      HResult=-2147024809
      Message=The output char buffer is too small to contain the decoded characters, encoding 'Unicode (UTF-8)' fallback 'System.Text.DecoderReplacementFallback'.
    Parameter name: chars
      ParamName=chars
      Source=mscorlib
      StackTrace:
           at System.Text.Encoding.ThrowCharsOverflow()
           at System.Text.Encoding.ThrowCharsOverflow(DecoderNLS decoder, Boolean nothingDecoded)
           at System.Text.UTF8Encoding.GetChars(Byte* bytes, Int32 byteCount, Char* chars, Int32 charCount, DecoderNLS baseDecoder)
           at System.Text.DecoderNLS.GetChars(Byte* bytes, Int32 byteCount, Char* chars, Int32 charCount, Boolean flush)
           at System.Text.DecoderNLS.GetChars(Byte[] bytes, Int32 byteIndex, Int32 byteCount, Char[] chars, Int32 charIndex, Boolean flush)
           at System.Text.DecoderNLS.GetChars(Byte[] bytes, Int32 byteIndex, Int32 byteCount, Char[] chars, Int32 charIndex)
           at System.IO.BinaryReader.InternalReadOneChar()
           at System.IO.BinaryReader.Read()
           at System.IO.BinaryReader.ReadChar()
           at ConsoleApplication2.Program.Main(String[] args) in C:\dp11\Program.cs:line 61
           at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
           at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
           at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
           at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
           at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
           at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
           at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
           at System.Threading.ThreadHelper.ThreadStart()
      InnerException: 


    to last line of below codes

                    while (reader.BaseStream.Position != reader.BaseStream.Length)
                    {
                        Rec1 item = new Rec1();
                        item.Name = reader.ReadString();
                        item.c0 = reader.ReadChar().ToString().ToCharArray();



    and here is how c0 has been defined

            public char[] c0;



    Many Thanks & Best Regards, Hua Min


    Friday, March 15, 2019 10:07 AM

All replies

  • Hello,

    Please read the following thread.


    Please remember to mark the replies as answers if they help and unmarked them if they provide no help, this will help others who are looking for solutions to the same or similar problem. Contact via my Twitter (Karen Payne) or Facebook (Karen Payne) via my MSDN profile but will not answer coding question on either.

    NuGet BaseConnectionLibrary for database connections.

    StackOverFlow
    profile for Karen Payne on Stack Exchange

    Friday, March 15, 2019 10:35 AM
    Moderator
  • What, exactly, is the format of this file?  Do you really have a character string, followed by a newline, followed by one byte?

    Unlike C and C++, C# makes a very sharp distinction between "char" and "byte".  If you ask it to read a char, then it had better be a valid UTF-8 encoded character.  That means you can't use it to read an arbitrary byte, because many UTF-8 characters are 2 or 3 bytes long.  If you really just one one byte, then you need to use ReadByte instead of ReadChar.


    Tim Roberts | Driver MVP Emeritus | Providenza & Boekelheide, Inc.

    Friday, March 15, 2019 10:36 PM
  • Hi

    Thank you for posting here.

    Based on your description, you want to develop two c# applications that you needed to complete.

    You could try the following code.

      static void Main(string[] args)
            {
                FileStream fs = new FileStream(@"D:\abc.txt", FileMode.Open, FileAccess.Read);
                using (var reader = new BinaryReader(fs))
                {
                        Rec1 item = new Rec1();
                        item.Name = reader.ReadString();
                        reader.BaseStream.Position = 0;
                        item.c0 = reader.ReadChars((int)(reader.BaseStream.Length - reader.BaseStream.Position));
                }
            }

    Result:

    Best Regards,

    Jack



    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.


    Monday, March 18, 2019 6:11 AM
    Moderator
  • You lack an indexer in the array when assigning value?

     item.c0[currentposition] = reader.ReadChar().ToString().ToCharArray();

    • Edited by ThisNewbie Monday, March 18, 2019 6:18 PM
    Monday, March 18, 2019 6:16 PM
  • Hi Jack,
    I further encounter

    System.OutOfMemoryException was unhandled
      HResult=-2147024882
      Message=Exception of type 'System.OutOfMemoryException' was thrown.
      StackTrace:
           at System.IO.BinaryReader.ReadChars(Int32 count)
           at ConsoleApplication2.Program.Main(String[] args) in C:\dp11\Program.cs:line 63
           at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
           at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
           at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
           at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
           at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
           at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
           at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
           at System.Threading.ThreadHelper.ThreadStart()
      InnerException: 



    due to below line

                        item.c0 = reader.ReadChars((int)(reader.BaseStream.Length - reader.BaseStream.Position));



    Many Thanks & Best Regards, Hua Min

    Tuesday, March 19, 2019 9:59 AM
  • Hi

    Thanks for the feedback.

    I could not reproduce your problem. I think it may be related to your binary file. You could try the following complete code to solve your problem better.

    class Program
        {
            static void Main(string[] args)
            {
                WriteFile();
                FileStream fs = new FileStream(@"D:\abc.txt", FileMode.Open, FileAccess.Read);
    
    
                using (var reader = new BinaryReader(fs))
                {
                        Rec1 item = new Rec1();
                        item.Name = reader.ReadString();
                        reader.BaseStream.Position = 0;
                        item.c0 = reader.ReadChars((int)(reader.BaseStream.Length - reader.BaseStream.Position));
                }
            }
    
    
            static void WriteFile()     
            {
                FileStream fs = new FileStream(@"D:\abc.txt",FileMode.OpenOrCreate);
                BinaryWriter w = new BinaryWriter(fs);
                char[] chars = { 'm', 's', 'd', 'n' };
     
                w.Write("hello");              
                w.Write("test");       
                w.Write(123);               
                w.Write(true);                 
                w.Write(chars);       
    
                w.Close();
                fs.Close();
            }
    
        }
        public class Rec1
        {
            public string Name { get; set; }
            public char[] c0 { get; set; }
    
        }

    Result:








    Best Regards,

    Jack


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Wednesday, March 20, 2019 2:51 AM
    Moderator
  • Hi Jack,
    I read the binary file that is created by below struct, and have got the current Out of Memory issue.

        public struct Rec1
        {
            public int Id;
            [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 20)]
            public string Name;
            public char[] c0;
            public char[] c1;
            public char[] c2;
            public char[] c3;
            public char[] c4;
            public char[] c5;
            public char[] c6;
            public char[] c7;
            public char[] c8;
            public string s0;
            public string s1;
            public string s2;
            public string s3;
            public string s4;
            public byte[] b0;
            public byte[] b1;
            public byte[] b2;
            public byte[] b3;
            public byte[] b4;
            public byte[] b5;
        }



    what is the good way to read binary file having above struct?

    Many Thanks & Best Regards, Hua Min

    Sunday, March 24, 2019 8:06 AM
  • Hi

    Thanks for the feedback.

    Could you provide the related code about creating the binary file?

    Best Regards,

    Jack


    MSDN Community Support
    Please remember to click "Mark as Answer" the responses that resolved your issue, and to click "Unmark as Answer" if not. This can be beneficial to other community members reading this thread. If you have any compliments or complaints to MSDN Support, feel free to contact MSDNFSF@microsoft.com.

    Monday, March 25, 2019 6:04 AM
    Moderator
  • Hi,

    I created the binary file using such codes

    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    using System.IO;
    using System.Runtime.Serialization.Formatters.Binary;
    using System.Runtime.Serialization;
    using System.Text.RegularExpressions;
    using System.Globalization;
    using System.Threading.Tasks;
    using System.Diagnostics; // for using Stopwatch
    //using System.Windows.Forms;
    using System.Runtime.InteropServices;
    
    namespace ConsoleApplication2
    {
        [StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode, Pack = 1)]
        [Serializable]
        public struct Rec1
        {
            public int Id;
            [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 20)]
            public string Name;
            public char[] c0;
            public char[] c1;
            public char[] c2;
            public char[] c3;
            public char[] c4;
            public char[] c5;
            public char[] c6;
            public char[] c7;
            public char[] c8;
            public string s0;
            public string s1;
            public string s2;
            public string s3;
            public string s4;
            public byte[] b0;
            public byte[] b1;
            public byte[] b2;
            public byte[] b3;
            public byte[] b4;
            public byte[] b5;
        }
        class Program
        {
            static void Main(string[] args)
            {
                //long StartingTime = Stopwatch.GetTimestamp();
                var stopwatch = new Stopwatch();
                stopwatch.Start();
    
                List<Rec1> data = new List<Rec1>();
    
                for (int i = 0; i < 1000000; ++i)
                    data.Add(new Rec1 { Id = i, Name = i.ToString().Trim(), s1 = "汕尾中国银行aaaaaaaaaaaaaaaaabbbbbbbbbbbccccccccccccccccdddddddddddddddddddddddeeeeeeeeeeeeeeeeeeeeeeeeeeeefffffffffffffffffffffffffggggggggggggggggghhhhhhhhhhhhhhh" + i.ToString().Trim(), s2 = "abbbbbbbbbbbcccccccccccccccc" });
    
                string filename = "C:\\dp11\\test.data";
                using (var file = File.OpenWrite(filename))
                {
                    var writer = new BinaryFormatter();
                    writer.Serialize(file, data); // Writes the entire list.
                }
    
                //long EndingTime = Stopwatch.GetTimestamp();
                //long ElapsedTime = EndingTime - StartingTime;
    
                //double ElapsedSeconds = ElapsedTime * (1.0 / Stopwatch.Frequency);
                stopwatch.Stop();
                double elapsed_time = stopwatch.ElapsedMilliseconds;
    
                Console.WriteLine("Operation completed in: " + elapsed_time + " (ms)");
                Console.ReadLine();
            }
        }
    
    }
    


    Many Thanks & Best Regards, Hua Min

    Monday, March 25, 2019 8:56 AM
  • If you are serializing a list of structures, then you need to deserialize that file into a list of the same structures.  You can't hope to pull it an item at a time; the serialization process writes type information into the file as well as the data.

    What are you really trying to do here?


    Tim Roberts | Driver MVP Emeritus | Providenza &amp; Boekelheide, Inc.

    Tuesday, March 26, 2019 6:17 AM
  • Hi,
    Why is there "out of memory" exception? I only want to read the binary file using exactly same struct.

    Many Thanks & Best Regards, Hua Min

    Thursday, March 28, 2019 1:43 AM

  • Why is there "out of memory" exception? I only want to read the binary file using exactly same struct.


    Did you not understand what Tim said about deserializing the file?

    A file created via Serialize must be retrieved via Deserialize.

    The file contains data related to the serialized data at the beginning.
    It is not a simple binary file containing individual structures.
    It contains all of the data and info needed to recreate the List of structures.

    Examine the file you created using a binary or hex file viewer and you should
    see a lot of control data before the actual List<Rec1> data begins.

    Build this program. It creates the serialized file as per your posted code.

    It then deserializes that file back into another List: "datain"

    Follow the prompts. At the end after you ensure you have a breakpoint set at
    the last line of Main(), continue to that breakpoint. This will allow you to
    use the debugger to examine the contents of datain and compare it to the
    original List "data".

    using System;
    using System.Collections.Generic;
    using System.Linq;
    using System.Text;
    using System.IO;
    using System.Runtime.Serialization.Formatters.Binary;
    using System.Runtime.Serialization;
    using System.Text.RegularExpressions;
    using System.Globalization;
    using System.Threading.Tasks;
    using System.Diagnostics; // for using Stopwatch
    //using System.Windows.Forms;
    using System.Runtime.InteropServices;
    
    namespace ConsoleApplication2
    {
        [StructLayout(LayoutKind.Sequential, CharSet = CharSet.Unicode, Pack = 1)]
        [Serializable]
        public struct Rec1
        {
            public int Id;
            [MarshalAs(UnmanagedType.ByValTStr, SizeConst = 20)]
            public string Name;
            public char[] c0;
            public char[] c1;
            public char[] c2;
            public char[] c3;
            public char[] c4;
            public char[] c5;
            public char[] c6;
            public char[] c7;
            public char[] c8;
            public string s0;
            public string s1;
            public string s2;
            public string s3;
            public string s4;
            public byte[] b0;
            public byte[] b1;
            public byte[] b2;
            public byte[] b3;
            public byte[] b4;
            public byte[] b5;
        }
        class Program
        {
            static void Main(string[] args)
            {
                var stopwatch = new Stopwatch();
                stopwatch.Start();
    
                List<Rec1> data = new List<Rec1>();
    
                for (int i = 0; i < 1000000; ++i)
                    data.Add(new Rec1 { Id = i, Name = i.ToString().Trim(), s1 = "??????aaaaaaaaaaaaaaaaabbbbbbbbbbbccccccccccccccccdddddddddddddddddddddddeeeeeeeeeeeeeeeeeeeeeeeeeeeefffffffffffffffffffffffffggggggggggggggggghhhhhhhhhhhhhhh" + i.ToString().Trim(), s2 = "abbbbbbbbbbbcccccccccccccccc" });
    
                //string filename = "C:\\dp11\\test.data";
                string filename = "test.data";
                using (var file = File.OpenWrite(filename))
                {
                    var writer = new BinaryFormatter();
                    writer.Serialize(file, data); // Writes the entire list.
                }
    
                stopwatch.Stop();
                double elapsed_time = stopwatch.ElapsedMilliseconds;
    
                Console.WriteLine("Operation completed in: " + elapsed_time + " (ms)");
                Console.WriteLine("Press Enter to begin deserializing file ...");
                Console.ReadLine();
    
    
                List<Rec1> datain = new List<Rec1>();
    
                if (File.Exists(filename))
                {
                    Console.WriteLine("Reading saved file");
                    Stream openFileStream = File.OpenRead(filename);
                    BinaryFormatter deserializer = new BinaryFormatter();
                    datain = (List<Rec1>)deserializer.Deserialize(openFileStream);
                    openFileStream.Close();
                }
                Console.WriteLine("deserializing file complete!");
                Console.WriteLine("set a breakpoint at the end of Main() ...");
                Console.WriteLine("Press Enter to continue to the breakpoint ...");
                Console.WriteLine("Examine contents of List \"datain\" in debugger ...");
                Console.ReadLine();
            }
        }
    }
    
    

    E&OE

    - Wayne

    Thursday, March 28, 2019 3:44 AM
  • I did deserialize the same file using

                Hashtable addresses = null;
    
                // Open the file containing the data that you want to deserialize.
                FileStream fs = new FileStream(@"C:\\dp11\\test.data", FileMode.Open);
                try
                {
                    BinaryFormatter formatter = new BinaryFormatter();
    
                    // Deserialize the hashtable from the file and 
                    // assign the reference to the local variable.
                    addresses = (Hashtable)formatter.Deserialize(fs);
                }
                catch (SerializationException e)
                {
                    Console.WriteLine("Failed to deserialize. Reason: " + e.Message);
                    throw;
                }
                finally
                {
                    fs.Close();
                }
    

                
    and have got

    System.InvalidCastException was unhandled
      HResult=-2147467262
      Message=Unable to cast object of type 'System.Collections.Generic.List`1[ConsoleApplication2.Rec1]' to type 'System.Collections.Hashtable'.
      Source=ConsoleApplication2
      StackTrace:
           at ConsoleApplication2.Program.Main(String[] args) in C:\ConsoleApplication2\Program.cs:line 64
           at System.AppDomain._nExecuteAssembly(RuntimeAssembly assembly, String[] args)
           at System.AppDomain.ExecuteAssembly(String assemblyFile, Evidence assemblySecurity, String[] args)
           at Microsoft.VisualStudio.HostingProcess.HostProc.RunUsersAssembly()
           at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
           at System.Threading.ExecutionContext.RunInternal(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
           at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state, Boolean preserveSyncCtx)
           at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
           at System.Threading.ThreadHelper.ThreadStart()
      InnerException: 



    due to Deserialize line above. Why?

    Many Thanks & Best Regards, Hua Min

    Thursday, March 28, 2019 7:41 AM
  • I did deserialize the same file using

                Hashtable addresses = null;
    
                // Open the file containing the data that you want to deserialize.
                FileStream fs = new FileStream(@"C:\\dp11\\test.data", FileMode.Open);
                try
                {
                    BinaryFormatter formatter = new BinaryFormatter();
    
                    // Deserialize the hashtable from the file and 
                    // assign the reference to the local variable.
                    addresses = (Hashtable)formatter.Deserialize(fs);
                }
    

                
    and have got

    System.InvalidCastException was unhandled
      HResult=-2147467262
      Message=Unable to cast object of type 'System.Collections.Generic.List`1[ConsoleApplication2.Rec1]' to type 'System.Collections.Hashtable'.
    

    due to Deserialize line above. Why?


    The message tells you why. You serialized a List, not a Hashtable.

    - Wayne

    Thursday, March 28, 2019 9:00 AM