none
Dynamic Collections

    Question

  • I am currently modifying the JIT (just-in-time) project code samples provided and using this is a primer to build my own Silverlight application that dynamically builds collections from a SQL backend.

    I was able to get the sample project to work against my own database and it loaded images just fine. But I have some questions and issues that I need help with:

    1. What is the limit (if any) of collection items when building dynamically from a backend DB source? When I try to a few couple thousand records, it takes forever (more than 5 minutes) to load. It's not super fast doing just 100 records but at least it's tolerable. This is not a hardware issue. The backend query generates the data in 2 seconds. This seems to be an issue with dynamically building the CXML file. Can this be confirmed by anyone? Any fixes or suggestions?

    2. How do I add multiple items to a facet? For example, if my subject is web pages, and one of my facet categories is "tags", I want to show each tag separately under this category that apply. In DB terms, this is just a simple one-to-many relationship. I just don't know how to code for this in the application. Again, I am trying to do this dynamically...not using the Excel tool.

    3.  When I tried to specify the "description" field when adding a collection item, it doesn't show up. Again, this is using the sample code provided in the JIT project sample. There is a argument to specify the "description", which I pass from my DB recordset. It does not show up in the viewer or the generated CXML file. Any ideas?

    Thanks in advance for the help!

    Saturday, July 10, 2010 10:15 PM

Answers

  • The DZI pyramids are being generated on the fly. At least if you have your HTTPHandler setup correctly.  When a request is made to "*/dzi/*_files/*/*_*.jpg" the response is handled by the DeepZoomImageHandler which calls PivotHttpHandlers.ServeDeepZoomImage which creates an implementation of itself and creates a DZI tile based on the level and x_y requested. 

    The DZC image tiles are created similarly, except they use the ImageTileHandler (and work differently). 

    From what I have found so far, when these images are created, they are not cached on the server side....however they can get cached by the client. I myself am working on caching these images on server side as well to lower the processor load caused by generating these image tiles when many users are using the collections.

    Thursday, July 15, 2010 3:59 PM
  • What's the best or recommended way to save the CXML content to a physical file?
     

    here's a pile of links:

    The command line tool from the pivot team allows easy conversion of excel/csv/cxml data into a deepzoom ready collection
    http://getpivot.com/developer-info/cmd-line-tools.aspx

    The backing library for the collection tool is also available, if you want to plug into the collection object model in memory, and then write out to a file:
    http://pauthor.codeplex.com/

    Community built tools are also available:
    http://www.tweetpivot.com/PivotAPI
    http://pivotcollectiontools.codeplex.com/

     

     

     

     

     

    Thursday, July 15, 2010 4:42 PM

All replies

  • 1. I think you need to break up the collection, my guess is after 5 k or 10k rows, it might not be good

    2. I read somewhere there that they can be seperated by '|'(pipe)

    Sunday, July 11, 2010 5:18 AM
  • 2.   Adding multiple items to a facet is straight forward: just pass the Facet constructor an array.  I usually keep my data in List<T> objects, so I have to do something like the following:

     

    collection.AddItem(name, null, null, image,

       new Facet("Foo", listOfInterestingStuff.ToArray()));

    Monday, July 12, 2010 12:17 PM
  • Some datapoints on #1:

    - I build a 14K collection every evening... using the XML seralizer class it takes very little time to write out the xml. I build the object graph in memory and then write it out to disk

    - When I load up that giant collection performance is not that good (the Pivot folks say that 3000 is about the limit when it comes to performance for the SL control). But loading all 14K images and displaying them takes less than 30 seconds. So the 5 minutes seems to be very long, especially if all the important stuff is generated in 2 seconds.

      Good luck!

    Monday, July 12, 2010 12:59 PM
  • Issue #1 is because the Deep Zoom images are all being generated on the fly. If you'll implement some caching, the load time will be dramatically reduced. That JIT project doesn't work well with large result sets for that reason. The CXML is generated very quickly. It's the Deep Zoom images that take up the majority of the processing time.

    Monday, July 12, 2010 3:06 PM
  • Deep Zoom images are not generated on the fly.  This is a misconception with the sample code that I think they need to make a bit clearer.  Take a look at the CollectionItem.SetImage() method.  Unless you specify a path or URI with a .DZI extension, it uses a simple FileImage or WebImage provider.  (I, too, thought I was generating DZIs on the fly using the sample code -- alas, it's too good to be true.)

    I suspect this has something to do with the performance issues.  I haven't tried switching to actual DZIs yet, but perhaps that could improve performance?

    Monday, July 12, 2010 8:11 PM
  • Maybe it's a semantic difference, but whatever images are being used are being generated on the fly. They are not being cached. They are generated every time they are requested. Is that not the case?

    Tuesday, July 13, 2010 8:25 AM
  • It depends on your definition of "generated": the files do have to be read off the disk and loaded into memory, however, the sample code does not generate deep zoom image pyramids on the fly.  Caching the images on the server would most definitely speed things up.

    Tuesday, July 13, 2010 9:18 AM
  • Thanks for all the feedback! While I admit I am a C# and Silverlight novice, the feedback was very helpful. I have decided to take a different approach. Instead of trying to generate large datasets at run-time, I am now going to use the deep-zoom command line tools to convert and create the collection image files. I then generate the CXML file and simply change the ImgBase to point to my specific collection.xml file.

    I confirmed that my performance issue was because I was specifying URL paths so there was a cost to pulling down those images. Once you get over a 1,000 records, the load time just becomes unacceptable. There is simply no way to do that efficiently without caching or having the images rendered locally.

    I want to try to save the XML to a file more automatically. I was thinking of modifying the sample code to create a service that I can schedule along with the creation of the deep-zoom image files. I don't really need a real-time representation of data but my data will change daily so I want to capture those changes after business hours.

    What's the best or recommended way to save the CXML content to a physical file? It seems like a simple enough task but I just don't know enough of C# or .net to do it. I typically live on the database side of development so this is all kind of new to me.  

    Once I get that wrapped up, I should have a fairly good sample project to share with everyone. What I have so far looks great! I am eager to test the limits of how many records the PivotViewer can handle assuming everything is pre-rendered (CXML + images).

    Again, thanks for the help!

    Wednesday, July 14, 2010 10:08 PM
  • The DZI pyramids are being generated on the fly. At least if you have your HTTPHandler setup correctly.  When a request is made to "*/dzi/*_files/*/*_*.jpg" the response is handled by the DeepZoomImageHandler which calls PivotHttpHandlers.ServeDeepZoomImage which creates an implementation of itself and creates a DZI tile based on the level and x_y requested. 

    The DZC image tiles are created similarly, except they use the ImageTileHandler (and work differently). 

    From what I have found so far, when these images are created, they are not cached on the server side....however they can get cached by the client. I myself am working on caching these images on server side as well to lower the processor load caused by generating these image tiles when many users are using the collections.

    Thursday, July 15, 2010 3:59 PM
  • What's the best or recommended way to save the CXML content to a physical file?
     

    here's a pile of links:

    The command line tool from the pivot team allows easy conversion of excel/csv/cxml data into a deepzoom ready collection
    http://getpivot.com/developer-info/cmd-line-tools.aspx

    The backing library for the collection tool is also available, if you want to plug into the collection object model in memory, and then write out to a file:
    http://pauthor.codeplex.com/

    Community built tools are also available:
    http://www.tweetpivot.com/PivotAPI
    http://pivotcollectiontools.codeplex.com/

     

     

     

     

     

    Thursday, July 15, 2010 4:42 PM
  • I see that now and I see how that code path is working.  Thanks very much for clarifying -- I had assumed since there was no reference to DeepZoomTools.dll that the code wasn't capable of generating the tiles.  I'm still at a loss as how to get the sample code to work with an existing DZC though.

    Thursday, July 15, 2010 4:57 PM
  • I am working on the same issue with no luck. Have you make any progress with caching..?


    The DZI pyramids are being generated on the fly. At least if you have your HTTPHandler setup correctly.  When a request is made to "*/dzi/*_files/*/*_*.jpg" the response is handled by the DeepZoomImageHandler which calls PivotHttpHandlers.ServeDeepZoomImage which creates an implementation of itself and creates a DZI tile based on the level and x_y requested. 

    The DZC image tiles are created similarly, except they use the ImageTileHandler (and work differently). 

    From what I have found so far, when these images are created, they are not cached on the server side....however they can get cached by the client. I myself am working on caching these images on server side as well to lower the processor load caused by generating these image tiles when many users are using the collections.


    Tuesday, August 03, 2010 12:00 PM
  • What I went with makes the collection more of a "Build and store on demand" collection than a JIT one. Using the example code with some minor modifications (the biggest changes being adding an overload to the PivotHttpHandlers.ServeImageTile sub to allow passing in a custom output stream and modifying the collection and related constructors to work with serialization) to allow my HTTPhandlers to use MSSQL to store the generated Collection object serialized as a string in a table I setup. The handler checks the DB for the latest version of the collection, if it is out of date then it builds it as normal except that it then serializes and stores the collection in the table for future use. 

    The CXML itself is also stored in the DB table after it is generated (I pass a context object with a custom response output stream that I send to both the DB and the real response). On subsequent requests I just return the value in the DB.

    The DZC is almost identical to the CXML.

    Images (both tile and DZI jpg images) are checked to see if they exist in a directory I created for the purpose of storing these images (naming convention identifies the collection type [I serve several different types], collection key, item index (if needed), level, X, and Y). If the file exists, then I send that through the response (thus saving all the processing time), if not then I go through the image generation process (only I use my new ServeImageTile that takes in a custom stream) and save the stream as an image on the disk and return that as the response as well.

    This means that although the first generation of a collection (these collections largely update daily although the smaller ones may happen more often) can take a while for large collections, the follow up requests are quite quick. In our case we will probably automate the initial building so a customer doesn't have that experience. I am also working on possibly putting the objects in the server cache (overwriting the current server cache solution) and checking it before hitting the DB. Just depends on the overall size of the objects.


    Wednesday, August 04, 2010 1:58 PM
  • What's the best or recommended way to save the CXML content to a physical file? 

    Of course, an automated method could use the following, where you pass a url to a method and then write the output to the file...

    public static string GetPage(string url) { WebResponse response = null; Stream stream = null; StreamReader reader = null; try { HttpWebRequest request = (HttpWebRequest) WebRequest.Create(url); response = request.GetResponse(); stream = response.GetResponseStream(); if (!response.ContentType.ToLower().StartsWith("text/")) return null; string buffer = "", line; reader = new StreamReader(stream); while ((line = reader.ReadLine()) != null) { buffer += line + "\r\n"; } return buffer; } catch (WebException e) { System.Console.WriteLine("Can't download:" + e); return null; } catch (IOException e) { System.Console.WriteLine("Can't download:" + e); return null; } finally { if (reader != null) reader.Close(); if (stream != null) stream.Close(); if (response != null) response.Close(); } } public static void WriteToLog(string logFileName, string data) { using (FileStream fileStream = new FileStream(logFileName, FileMode.Append, FileAccess.Write, FileShare.None)) { using (StreamWriter streamWriter = new StreamWriter(fileStream)) { streamWriter.WriteLine(data); } } } WriteToLog("c:\something.cxml",GetPage("http://someURL"));



    Wednesday, August 04, 2010 7:13 PM
  • Thank you for your reply,

    I am also interested to make a "Build and store on demand" collection rather than a JIT one, because for the JIT collocation on my high performance server somebody have to wait about 7-8 minutes to get the images (complex GDI+ and a lot of data).

    If you can share with us (or send me a PM) your ServeImageTile implementation it will be very helpful and I will appreciate it.

    Thank you in advance and once again your previous post was very helpful and I am trying to implement this.

    What I went with makes the collection more of a "Build and store on demand" collection than a JIT one. Using the example code with some minor modifications (the biggest changes being adding an overload to the PivotHttpHandlers.ServeImageTile sub to allow passing in a custom output stream and modifying the collection and related constructors to work with serialization) to allow my HTTPhandlers to use MSSQL to store the generated Collection object serialized as a string in a table I setup. The handler checks the DB for the latest version of the collection, if it is out of date then it builds it as normal except that it then serializes and stores the collection in the table for future use.

    Tuesday, August 24, 2010 5:44 PM
  • This is the order after somebody load the pivot:

    web.config --> <add path="*_files/*/*_*.jpg" verb="GET" type="PivotServer.ImageTileHandler"/>

    ImageTileHandler --> ProcessRequest(HttpContext context)

    PivotHttpHandlers --> ServeImageTile(HttpContext context)

    CollectionImageTileBuilder --> Write(HttpResponse response)

    CollectionImageTileBuilder --> Write(Stream outStream)

    SystemWindowsMediaTileImage --> WriteTo(Stream stream)


    I have changed the method WriteTo in the SystemWindowsMediaTileImage class in order to save the image in a folder and in the next request I plan to check if the image already exists. This is my WriteTo implementation:


    public void WriteTo(Stream stream, bool isCachingOutput, string cachingOutputPath, string cachingOutputImageName)
    {
    //EndDrawSubImages
    CloseContext();

    const double dotsPerInch = 96.0;

    RenderTargetBitmap tileBitmap = new RenderTargetBitmap(m_tileDimension, m_tileDimension,
    dotsPerInch, dotsPerInch, PixelFormats.Default);
    tileBitmap.Render(m_visual);
    CloseDisposables();

    //Write
    using (MemoryStream memStream = new MemoryStream())
    {
    Debug.WriteLine(cachingOutputImageName + " -- " + cachingOutputPath);
    WriteJpgToStream(memStream, tileBitmap);
    memStream.Position = 0; //Rewind the stream
    CopyStream(memStream, stream);

    //added lines
    memStream.Position = 0; //Rewind the stream

    Image image = Image.FromStream(memStream);
    if (!Directory.Exists(cachingOutputPath))
    Directory.CreateDirectory(cachingOutputPath);

    //image.Save(cachingOutputPath + cachingOutputImageName);
    image.Dispose();

    }
    }


    The output of this line  "Debug.WriteLine(cachingOutputImageName + "    --    " + cachingOutputPath);" for my small collection (10 items) which i am using for debugging reasons is:

    0_0.jpg    --    D:\.....\bin\Cache\pivot-b4a22bf348464aada900407b9015375c\0\
    0_0.jpg    --    D:\.....\bin\Cache\pivot-b4a22bf348464aada900407b9015375c\2\
    0_0.jpg    --    D:\.....\bin\Cache\pivot-b4a22bf348464aada900407b9015375c\3\
    0_0.jpg    --    D:\.....\bin\Cache\pivot-b4a22bf348464aada900407b9015375c\1\
    0_0.jpg    --    D:\.....\bin\Cache\pivot-b4a22bf348464aada900407b9015375c\5\
    0_0.jpg    --    D:\.....\bin\Cache\pivot-b4a22bf348464aada900407b9015375c\4\
    0_1.jpg    --    D:\.....\bin\Cache\pivot-b4a22bf348464aada900407b9015375c\7\
    0_0.jpg    --    D:\.....\bin\Cache\pivot-b4a22bf348464aada900407b9015375c\6\
    0_0.jpg    --    D:\.....\bin\Cache\pivot-b4a22bf348464aada900407b9015375c\7\
    2_1.jpg    --    D:\.....\bin\Cache\pivot-b4a22bf348464aada900407b9015375c\8\
    1_0.jpg    --    D:\.....\bin\Cache\pivot-b4a22bf348464aada900407b9015375c\7\
    3_1.jpg    --    D:\.....\bin\Cache\pivot-b4a22bf348464aada900407b9015375c\8\
    0_2.jpg    --    D:\.....\bin\Cache\pivot-b4a22bf348464aada900407b9015375c\8\
    1_2.jpg    --    D:\.....\bin\Cache\pivot-b4a22bf348464aada900407b9015375c\8\
    3_0.jpg    --    D:\.....\bin\Cache\pivot-b4a22bf348464aada900407b9015375c\8\
    0_1.jpg    --    D:\.....\bin\Cache\pivot-b4a22bf348464aada900407b9015375c\8\
    1_1.jpg    --    D:\.....\bin\Cache\pivot-b4a22bf348464aada900407b9015375c\8\
    1_0.jpg    --    D:\.....\bin\Cache\pivot-b4a22bf348464aada900407b9015375c\8\
    0_0.jpg    --    D:\.....\bin\Cache\pivot-b4a22bf348464aada900407b9015375c\8\
    2_0.jpg    --    D:\.....\bin\Cache\pivot-b4a22bf348464aada900407b9015375c\8\


    When I unncomment the image.Save(cachingOutputPath + cachingOutputImageName); in order to actually save the image I am getting only:

    0_0.jpg    --    D:\......\bin\Cache\pivot-811cb623aa084f7287926364b4b4557a\0\
    0_0.jpg    --    D:\.......\bin\Cache\pivot-811cb623aa084f7287926364b4b4557a\5\

    and I can see only these 2 images in the pivot

    Anyone who can help me.. :)


    Update: something with s_lock = new ReaderWriterLockSlim(); in PivotHttpHandlers class and threats

    Update2: I find the way and fix it. The performance is much better..

    I make the changes to: ServeImageTile, ServeDzc etc in the PivotHttpHandlers and extend the model of CollectionCache.

    Wednesday, August 25, 2010 8:29 AM
  • i'm currently working on JIT(juste-in-time)projet ,and i don't know how use images stored in a sqlserver database

    as bytes format.

    because pivotviewer ImageItem have two costructor : ImageItem(String) and ImageItem(Uri)

    Thanks in advance for the help!

    Friday, August 27, 2010 10:21 AM
  • Viennas,

    Sounds like my situation as well, we are building several collections of 8 to 10K items and the build time is too long for JIT to be very useful. I have no problem with providing some of my imageTile Solution, but because it's for a rather custom implementation I think it would be best if I explain and provide snippets of code.

    To set some of the groundwork, the collection business object (CBO...pivotservertools.collection), the CXML, and the DZC are all being stored in our Database when the collection is being built (in some cases this is a process that is scheduled in others it is caused by a change in the collection data..depends on the size and frequency of updates to the collection). The CBO is stored by serializing, compressing, and encrypting the object as a string. The other two are just compressed and encrypted.

    When a tile request comes in, the first thing I do is ensure that the collection being requested still resides in the server cache... if not I load it back in from the database. Then I build the filePath that would point to the cached imageTile (I use the format "/imgCache/{collectionType}_files/{collectionKey}_{level}_{x}_{y}.jpg"). If the file exists, I serve it and move on. If not I do the following:

                           If Not Directory.Exists(context.Server.MapPath("/visualsearch/imgCache/" & collectionType & "_files/")) Then
                                'create the directory
                                Directory.CreateDirectory(context.Server.MapPath("/visualsearch/imgCache/" & collectionType & "_files/"))
                            End If
    
                            Dim fs As New FileStream(context.Server.MapPath(cachedFileName), FileMode.OpenOrCreate, FileAccess.ReadWrite)
    
                            PivotServerTools.PivotHttpHandlers.ServeImageTile(context, fs)
                            fs.Close()
    
                            'serve the new file
                            context.Response.ContentType = "image/jpeg"
                            context.Response.WriteFile(context.Server.MapPath(cachedFileName))

    Now obviously the standard ServeImageTile in Pivot Server Tools doesn't have an option for a custom stream, so I added an overload to it. Here is what I added in PivotHttpHandlers.cs

            public void ServeImageTile(HttpContext context, Stream customStream)
            {
                ImageRequest request = new ImageRequest(context.Request.Url);
    
                Collection collection = m_collectionCache.Get(request.DzcName);
                if (null == collection)
                {
                    //TODO: Draw this message onto an image tile so it can be seen in Pivot.
    
                    context.Response.StatusCode = (int)HttpStatusCode.NotFound;
                    context.Response.StatusDescription = "Pivot image not found. Cache may have expired.";
                    return;
                }
    
                CollectionImageTileBuilder builder = new CollectionImageTileBuilder(collection, request,
                    DzcSerializer.DefaultMaxLevel, DzcSerializer.DefaultTileDimension);
                builder.Write(customStream);
            }

    Fairly straightforward stuff.... in fact I used similar techniques to have the CXML and DZC handlers write to a custom stream (so I could output to both the request and our SQL server). I hope this helps.


    Maarek







    Thursday, September 16, 2010 11:35 AM
  • Hi gwatts,

     

    Are you generating 14K records dynamically (means using JIT collection)?

    My case I've 16K records, but its loading very slow.

    - KiriKamal

    Wednesday, November 24, 2010 4:27 AM
  • I'm currently working on Pivot Viewer.I'm desperately looking for some help on how to do JIT dynamic collection as I'm totally new to pivot and .net aswell.

    Do I need to create 3 tables for the collections or wht r the tables required and where shd i change according to my need.I've my images in my server folder,how do I retrive it.I also want to add more collections .Please help me step by step how to do my collections otherwise some sample code would be much appreciated.

    Your Help is much appreciated.

    Tuesday, January 10, 2012 1:57 PM