none
Problem with extracting an image object through async socket

    Question

  • Hello everyone,

    I am building a SL4 MVVM OOB client-server application. I am trying to extract an image object (in jpg, png and bmp formats) from server (writen in python) through socket and bind it to an Image control in the View. I am using converter below. The application throws an error saying that Invalid lenght for a Base 64-char array. From other hand everything works fine in python client application. SL doesn't support Ascii so I think the problem is in these

    byte[] imageBytes = Convert.FromBase64String(e.Data);

    //this.CurrentPatientImage.Photo = Encoding.UTF8.GetBytes(e.Data);

    part of the program. Please advise.

    public class ImageConverter : IValueConverter
        {
            public object Convert(object value, Type targetType, object parameter, CultureInfo culture)
            {
                BitmapImage bi = new BitmapImage();
                
                if (value != null)
                {
                    if (value != null)
                    {
                        MemoryStream stream = new MemoryStream((Byte[])value);
                        bi.SetSource(stream);
                        return bi;
                    }
                }
                return bi;
            }


            public object ConvertBack(object value, Type targetType, object parameter, CultureInfo culture)
            {
                throw new NotImplementedException("PropertyViewer does not use ConvertBack.");
            }
        }

    ViewModel class:

     public PatientImage CurrentPatientImage
            {
                get
                {
                    return _currentPatientImage;
                }
                set
                {
                    if (_currentPatientImage != value)
                    {
                        _currentPatientImage = value;
                        RaisePropertyChanged("CurrentPatientImage");
                    }
                }
            }

    void _client_MessageReceived(object sender, SocketMessageEventArgs e)
            {
                System.Windows.Deployment.Current.Dispatcher.BeginInvoke(() =>
                {              

                    if (e.Error == null)
                    {
                        if (((string)e.Data).StartsWith("1"))
                        {
                            try
                            {
                                string jsonText = (e.Data.Trim()).Remove(0, 1);

                                this.CurrentPatientImage = new PatientRepository().GetPatientPicture(jsonText);

                                switch (this.CurrentPatientImage.FileName.GetFileExtension())
                                {
                                    case (".jpg"):
                                    case (".jpeg"):
                                    case (".png"):
                                        {
                                            byte[] imageBytes = Convert.FromBase64String(e.Data);

                                            using (MemoryStream ms = new MemoryStream(imageBytes, 0, imageBytes.Length))
                                            {
                                                this.CurrentPatientImage.Photo = ms.ToArray();
                                            }
                                            //this.CurrentPatientImage.Photo = Encoding.UTF8.GetBytes(e.Data);
                                        }
                                        break;
                                    default:
                                        break;
                                }
                                }
                            }
                            catch (Exception ex)
                            {
                                ShowErrorWindow(ex.Message);
                                //throw ex;
                            }
                        }
            }

    View.xaml

    <Image Source="{Binding Photo, Converter={StaticResource ImageConverter}}" Stretch="Uniform" Height="150" Width="150" />

    Model class:

    public class PatientImage : ModelBase, IPatientImage
        {
            #region Private Members

            private string _fileName;
            private int _length;
            private DateTime _uploadDate;
            private byte[] _photo;

            #endregion

            #region Constructor

            public PatientImage() { }

            internal PatientImage(PatientImage patientImage)
            {
                _fileName = patientImage.FileName;
                _length = patientImage.Length;
                _uploadDate = patientImage.UploadDate;
                _photo = patientImage.Photo;
            }

            #endregion

            #region Public Properties

            [Display(Name = "File Name:", Description = "Patient's Picture")]
            [JsonProperty("filename")]
            public string FileName
            {
                get { return _fileName; }
                set
                {
                    if (_fileName == value) return;
                    _fileName = value;
                    base.FirePropertyChanged("FileName");
                }
            }

            [Display(Name = "File Length:", Description = "Length of the File")]
            [JsonProperty("length")]
            public int Length
            {
                get { return _length; }
                set
                {
                    if (_length == value) return;
                    _length = value;
                    base.FirePropertyChanged("Length");
                }
            }

            [Display(Name = "File Upload Date:", Description = "File Upload Date")]
            [JsonProperty("uploadDate")]
            public DateTime UploadDate
            {
                get { return _uploadDate; }
                set
                {
                    if (_uploadDate == value) return;
                    _uploadDate = value;
                    base.FirePropertyChanged("UploadDate");
                }
            }

            [Display(Name = "Photo:", Description = "Patient photo")]
            //[JsonIgnore]
            public byte[] Photo
            {
                get { return _photo; }
                set
                {
                    if (_photo == value) return;
                    _photo = value;
                    base.FirePropertyChanged("Photo");
                }
            }

            #endregion

        }

    Friday, October 28, 2011 3:27 AM

Answers

  • I have no experience at all with BSON or MongoDB. But I did a quick look at the documentation you've linked to and saw some sample PHP code which gave me the right clue:

    // Find image to stream
    $image = $gridFS->findOne("chunk-screaming.jpg");
    
    // Stream image to browser
    header('Content-type: image/jpeg');
    echo $image->getBytes();

    The "getBytes" method returns the raw file contents, which is a good thing. If you are sending the file from the server to the client in a similar way, then you indeed can directly consume it without any manual processing.

    The mistake you then have in your code is that you are apparently using some additional code or component that converts the received data from the socket to a string (according to your code fragment, "SocketMessageEventArgs.Data" is a string). Instead of doing that, you have to get ahold of the original raw byte data received by the socket. That data already is your image. Simply create a memory stream from that data and use it as source for a bitmap image - if it is a PNG or JPEG - or to convert it using the ImageTools library.

    Friday, October 28, 2011 9:09 AM

All replies

  • Hi. What you're doing in your code is pretty confusing. First you're treating the received data as normal string (checking if it starts with "1"), then you're apparently using it as json, and finally you're trying to treat it as Base64 encoded string to extract an image - it's obvious that something's wrong here.

    Since it's not apparent what is actually returned from the server, nobody will be able to answer your question. You should carefully look at the format that is returned and revise the steps you perform here to correct them. One thing you're right about is that Silverlight does not support ASCII out of the box, so if any parts of the data are encoded that way you'll have to manually convert it or make the server side return UTF8 instead.

    Friday, October 28, 2011 5:14 AM
  • Thanks a lot for prompt reply.

    Please find below further details on the workflow:

    1. My SL4 client-server app sends a message to the Python based server in the following format:

    A) api_message + mac of computer + user name + api command (getImage + pictureid in this particular case)

    Server checks if the data is valid. If it's valid the server returnes the json representation of the picture metadata like below:

        "1{\"method_name\": \"getPatientDocument\", \"module_name\": \"pciPatient\", \"result_set\": {\"chunkSize\": 262144, \"_closed\": true, \"p_id\": \"None\", \"filename\": \"test.bmp\", \"length\": 270054, \"uploadDate\": \"2011-10-27\", \"_id\": \"4ea9acb7c19a841037000000\", \"md5\": \"56f7567c7ca4deb80cfe74778cec0dfb\"}, \"rec_count\": 8}"
     
    If the data is valid server add "1" to the json string. (Please node that we are using MongoDB to store images)

    So if the metadata of the image file is valid I need to send one more message to the server in the following format:

    B) api_message + mac of computer + user name + "1"

    After receiving this message the server returns an image object.

    void _client_MessageReceived(object sender, SocketMessageEventArgs e)
            {
                System.Windows.Deployment.Current.Dispatcher.BeginInvoke(() =>
                {              

                    if (e.Error == null)
                    {
                       // I am checking if the image metadata is valid
                        if (((string)e.Data).StartsWith("1"))
                        {
                            try
                            {
                              // if metadata is valid
                              // a) I extract "1" from the json string                         
                               string jsonText = (e.Data.Trim()).Remove(0, 1);
                              // send one more message to the server (message B)
                                 getPatientDocumentByFileIdAndUsername();
                             // call method to deserialize data in the json format into instance of PageImage class (it works fine)
                                 this.CurrentPatientImage = new PatientRepository().GetPatientPicture(jsonText);
                                }
                            }
                            catch (Exception ex)
                            {
                                ShowErrorWindow(ex.Message);
                                //throw ex;
                            }
                        }
    The I am waiting the second response from the server. This time it returns an an image object.
    Python app just cast this object to image like so: (image)obj and bind it to control.

    I handle the response like below:

    try
                            {
                                switch (this.CurrentPatientImage.FileName.GetFileExtension())
                                {
                                    case (".bmp"):
                                    case (".jpg"):
                                    case (".jpeg"):
                                    case (".png"):
                                        {
                                           byte[] imageBytes = Convert.FromBase64String(e.Data);

                                            using (MemoryStream ms = new MemoryStream(imageBytes, 0, imageBytes.Length))
                                            {
                                                this.CurrentPatientImage.Photo = ms.ToArray();
                                            }                                     

                                        }
                                        break;


    No success so far.

    I've also tried to convert the data into ASCII using an extension method below. No luck.

    public static byte[] StringToAscii(this string s)
            {
                byte[] retval = new byte[s.Length];
                for (int ix = 0; ix < s.Length; ++ix)
                {
                    char ch = s[ix];
                    if (ch <= 0x7f) retval[ix] = (byte)ch;
                    else retval[ix] = (byte)'?';
                }
                return retval;
            }


    byte[] b = e.Data.ToString().StringToAscii();
                                            
    this.CurrentPatientImage.Photo = b;

    The Python based server is using MongoDB and Mongo GridFS API to store and handle images and documents.

    Friday, October 28, 2011 6:00 AM
  • Ok. But you're still not saying what the server is returning as image data

    byte[] imageBytes = Convert.FromBase64String(e.Data);

    This of course only works if the image was Base64 encoded on the server side.

    Also please note that Silverlight has no built-in support for BMP (only PNG and JPEG), so testing/using that logic with BMPs doesn't work anyway. For BMPs you would have to use a third party library like ImageTools, for example.

    Friday, October 28, 2011 7:43 AM
  • Thanks a lot for prompt reply!

    > Ok. But you're still not saying what the server is returning as image data

    Frankly speaking I am still also trying to figure out the current type of the returning data. Most probably BSON object.
    I've already read GriFS tech docs http://www.mongodb.org/display/DOCS/GridFS+Specification from A to Z  and looks like server returns the image as binary object
    http://learnmongo.com/posts/getting-started-with-mongodb-gridfs/ So when I send the second message to the server, it streams the file back to the app through socket using MongoDB Shell.
    So I don't need to connect to MongoDB and search for it, server does it for me.

        Connect to MongoDB
        Do a findOne() on the file
        Load it into memory using getBytes()
        Set the proper headers
        Stream the file back to the my app through socket

    So all I have to do is to convert the image data stream (probably  BSON object) into byte stream and send it to converter. And this is the problem since I still can't find a right way to do it.
    I tried to deserialize the object with new BSON features of JSON.NET library, but still no luck. Looks like I missed something while learning MongoDB documentation.
       
    > Also please note that Silverlight has no built-in support for BMP (only PNG and JPEG), so testing/using that logic with BMPs doesn't work anyway. For BMPs you would have to use a third party library like ImageTools, for example.
    Thanks a lot for this reference, great tool.

    Friday, October 28, 2011 8:30 AM
  • I have no experience at all with BSON or MongoDB. But I did a quick look at the documentation you've linked to and saw some sample PHP code which gave me the right clue:

    // Find image to stream
    $image = $gridFS->findOne("chunk-screaming.jpg");
    
    // Stream image to browser
    header('Content-type: image/jpeg');
    echo $image->getBytes();

    The "getBytes" method returns the raw file contents, which is a good thing. If you are sending the file from the server to the client in a similar way, then you indeed can directly consume it without any manual processing.

    The mistake you then have in your code is that you are apparently using some additional code or component that converts the received data from the socket to a string (according to your code fragment, "SocketMessageEventArgs.Data" is a string). Instead of doing that, you have to get ahold of the original raw byte data received by the socket. That data already is your image. Simply create a memory stream from that data and use it as source for a bitmap image - if it is a PNG or JPEG - or to convert it using the ImageTools library.

    Friday, October 28, 2011 9:09 AM
  • You are absolutely right. The MongoDB database returns files. It was hard to believe it when you get used to Sql Server database, but it's true, it returns file)). I've already done a simple wcf service and successfuly bound test images (jpeg and png) to the xaml form. 
    The problem in this case is wrong response from the server. I was just looking for a mistake in wrong place.

    So I've created a simple extension method below for converting file stream into byte array:

        public static byte[] ConvertFileToByteArray(this string fileName)
            {
                byte[] returnValue = null;

                using (FileStream fr = new FileStream(fileName, FileMode.Open))
                {
                    using (BinaryReader br = new BinaryReader(fr))
                    {
                        returnValue = br.ReadBytes((int)fr.Length);
                    }
                }

                return returnValue;

            }

    And I call it like so in the ViewModel class of the app like so: this.CurrentPatientImage.Photo = strPath.ConvertFileToByteArray();

    It works fine with both jpeg and png formats now (I am using converter shown in my previous thread) but I am going to take care of all existing formats. ImageTools is da bomb but I've got some restrictions on using third party libraries in my part of the app. Only .net framework.

    Time and again appreciate for taking care of my request and for prompt reply.

    Saturday, October 29, 2011 4:22 AM

  • Yes, Peter, the cause of the problem was python server's response. Since Silverlight doesn't have an Encoding.ASCII class, I implemented several my own ASCII bytes to UTF-8 convertors by all them didn't address the problem with python server's output. I am using NoSQL database (Azure) in my new Silverlight 4 project and I don't have problems with processing images this time. I'm using FJCode library to convert service response.
    Sunday, December 01, 2013 6:42 PM