Some JSON Help needed

Question

I am having some success parsing JSON from "http://api.tvmaze.com/schedule?country=US" but there are parts I just cannot get to. The code below should explain one example. Ideas welcome...

Imports System.Net
Imports System.IO
Imports Newtonsoft.Json
Imports Newtonsoft.Json.Linq
Public Class Form1
    Private Sub Button2_Click(sender As System.Object, e As System.EventArgs) Handles BtnGo.Click
        Dim request As HttpWebRequest
        Dim response As HttpWebResponse = Nothing
        Dim reader As StreamReader

        Try
            request = DirectCast(WebRequest.Create("http://api.tvmaze.com/schedule?country=US"), HttpWebRequest)
            '
            response = DirectCast(request.GetResponse(), HttpWebResponse)
            reader = New StreamReader(response.GetResponseStream())

            Dim rawresp As String
            rawresp = reader.ReadToEnd()

            Dim results As JArray = JsonConvert.DeserializeObject(rawresp)
            Dim Count As Integer = 0 ' Just show a few while testing
            For Each item As JObject In results
                '"schedule":{"time":"11:00","days":["Monday","Tuesday","Wednesday","Thursday","Friday"]}
                'this is the part I'm not getting, it's like an item with sub-items

                'TXTResults.AppendText(item.ToString & vbNewLine) ' display all of each object
                Try
                    TXTResults.AppendText("ID = " & item("id").ToString & vbNewLine)
                    TXTResults.AppendText("Name = " & item("name").ToString & vbNewLine)
                    TXTResults.AppendText("Season = " & item("season").ToString & vbNewLine)
                    TXTResults.AppendText("Number = " & item("number").ToString & vbNewLine)
                    'TXTResults.AppendText("Schedule = " & item("schedule").ToString & vbNewLine)
                    'TXTResults.AppendText("Schedule = " & item("schedule").Item("time").ToString & vbNewLine)
                    'TXTResults.AppendText("Schedule = " & item("schedule.time").ToString & vbNewLine)
                    'All these Failed

                    TXTResults.AppendText(vbNewLine)
                Catch ex As Exception
                    TXTResults.AppendText("Error reading Item Detail" & vbNewLine)
                End Try
                Count += 1
                If Count > 10 Then
                    TXTResults.AppendText(vbNewLine & "Limit reached")
                    Exit Sub 'enough already
                End If
            Next
        Catch ex As Exception
            MsgBox("Error reading Results:" & vbNewLine & ex.Message)
        Finally
            If Not response Is Nothing Then response.Close()
        End Try
    End Sub
End Class

Answer 1

Hi

I have been playing around with using JSON, and have had some small success - still a lot of restructuring needed. You can see my lack of understanding from the

On Error GoTo errorhandler

I have actually stopped this project because it seems to have a large time overhead. However, here is one Sub in my project that *might* shed some light (maybe not) - in any case here it is.

BTW: I have some inclination to restart work on this, so if you get some more knowledgable responses here, then I will be very interested too.

    Public Function FetchChannelEPG(chan As String) As List(Of EPGitem)
        ' provides EPG data List for a single channel
        Dim epg As New List(Of EPGitem)
        Dim startdate As String = Now.ToString("yyyy-MM-ddTHH:mm:ssZ")
        Dim enddate As String = Now.AddDays(My.Settings.EPGdays).ToString("yyyy-MM-ddT00:00:00Z")
        Dim src As String = "schedule.json?channel_id=" & chan & "&publisher=pressassociation.com&annotations=description,broadcasts,brand_summary,series_summary,people,extended_description&from=" & startdate & "&to=" & enddate & "&apiKey=" & My.Settings.ATLASkey

        Dim response As String = ReturnTextFromURL(src)
        If response = Nothing Then
            MessageBox.Show("ERROR, couln't retrieve EPG data. Check request string:" & vbCrLf & src & vbCrLf & vbCrLf & "(Particularly, check within maximum of 14 days.)")
            Return Nothing
        End If
        Dim alldata As JObject = JObject.Parse(response)
        For Each result As KeyValuePair(Of String, JToken) In alldata
            For Each schedule As JToken In result.Value
                For Each channel As JToken In schedule.Item("items")
                    Dim ch As New EPGitem
                    On Error GoTo errorhandler
                    ch.episode_number = channel.Item("episode_number").ToString
                    ch.starttime = CDate(channel.Item("broadcasts").Item(0).Item("transmission_time"))
                    ch.duration = CInt(channel.Item("broadcasts").Item(0).Item("broadcast_duration"))
                    ch.title = channel.Item("container").Item("title").ToString
                    ch.subtitle = channel.Item("title").ToString
                    ch.uri = channel.Item("uri").ToString
                    ch.id = channel.Item("id").ToString
                    ch.curie = channel.Item("curie").ToString
                    ch.type = channel.Item("type").ToString
                    ch.description = channel.Item("description").ToString
                    ch.short_description = channel.Item("short_description").ToString
                    ch.medium_description = channel.Item("medium_description").ToString
                    ch.container = GetCont(channel.Item("container"))
                    ch.people = GetPeople(channel.Item("people"))
                    ch.series = GetSeries(channel.Item("series_summary"))
                    epg.Add(ch)
                Next
            Next
        Next
        Return epg
errorhandler:
        Select Case Err.Description
            Case "Object reference not set to an instance of an object."
                Resume Next
        End Select
    End Function

---

Regards Les, Livingston, Scotland

Answer 2

The data you get back is an Array of Objects.  Some of the property values of those Objects are also Arrays or other Objects.

Here's a glimpse of the data in a tree format:

You can see that the root of the data is an Array and then there are a whole bunch of Objects each with the properties shown in the image.  Most of these properties have string, integer, or date values but some, like "network" contain another object with its own properties.

So you are exactly right; you have items with subitems that you'll need to enumerate.

---

Reed Kimble - "When you do things right, people won't be sure you've done anything at all"

Answer 3

Perhaps the following link may help:

http://www.newtonsoft.com/json

---

Lloyd Sheen

Answer 4

Devon,

In my opinion - and it's only just that - just like with the tough one that Les was working on, the best way would be to fully understand the structure of the data then create a class (possibly with nested classes) and then Newton can deserialize it and create instances of the class directly - or at least fairly directly.

Looking at what Reed posted, it'll be a tough row to hoe, but in time it could be done.

---

Knowledge rests not upon truth alone, but upon error also. Carl Jung

Answer 5

At least my guess was right, implementing it so far is going nowhere

Item.Descendants does not help, it only shows the root item all over again.

Item.HasValues repeats the full range over and over

I'm still looking thru it

The javascript source uses things like "Item.show.name" to get the actual name of the show. Of course I can't use that or find anything resembling it.

Answer 6

Devon,

In my opinion - and it's only just that - just like with the tough one that Les was working on, the best way would be to fully understand the structure of the data then create a class (possibly with nested classes) and then Newton can deserialize it and create instances of the class directly - or at least fairly directly.

Looking at what Reed posted, it'll be a tough row to hoe, but in time it could be done.

---

Knowledge rests not upon truth alone, but upon error also. Carl Jung

This is correct.  The Newtonsoft library works best when you have POCO (plain old class objects) that mirror the structure of the JSON data.  Then you can just serialize/deserialize between data and type instance.

The screen shot above is from some code I was working on which is meant to provide a T4 text template you can use in a project to generate POCO class from JSON data.  I parse the JSON into a much simpler object model to generate the POCO and technically you could continue to just use that parsing along with the generated objects (or in lieu of them).  But you could also take the resulting POCO and use it with Newtonsoft JSON if you wanted to take advantage of that object model.

The editor I'm working on isn't finished yet; I still need to deal with Arrays of values and would like to provide an easier way to find all of the objects which require naming (and/or add default names to everything).  The parser works pretty well though and the resulting object model is easy to use.  I may be making some changes to it to better support the POCO generation but I can share the parser and object model if you'd like to take a look or use it as a basis for your own parser (instead of using Newtonsoft).

There's also some JSON support in VS2015 that Karen pointed out to me in this thread:

https://social.msdn.microsoft.com/Forums/vstudio/en-US/2d0e9bc8-96d6-4ffd-8326-924aac8d1e8f/an-easier-way-to-consume-json?forum=vbgeneral

And actually, I have an early version of the parser and object model in that thread so you check it out there if you want.

---

Reed Kimble - "When you do things right, people won't be sure you've done anything at all"

Answer 7

I removed the folder again, restarted the computer, made a new project and etc and all is working. Perhaps I can get the structure now and make sense of it

Answer 8

OK, I think I've about got this POCO generator working.  Here's what I was able to generate for the URL in this thread:

Partial Public Class TvShowWinnersCollection
    Inherits ObjectModel.Collection(Of TvShowWinners)
End Class

Partial Public Class TvShowWinners
    Public Overridable Property id As Integer
    Public Overridable Property url As String
    Public Overridable Property name As String
    Public Overridable Property season As Integer
    Public Overridable Property number As Integer
    Public Overridable Property airdate As Date
    Public Overridable Property airtime As Date
    Public Overridable Property airstamp As Date
    Public Overridable Property runtime As Integer
    Public Overridable Property image As TvShowImage
    Public Overridable Property summary As String
    Public Overridable Property show As TvShow
    Public Overridable Property _links As TvShowLinks
End Class

Partial Public Class TvShow
    Public Overridable Property id As Integer
    Public Overridable Property url As String
    Public Overridable Property name As String
    Public Overridable Property type As String
    Public Overridable Property language As String
    Public Overridable Property genres As JsonStringCollection
    Public Overridable Property status As String
    Public Overridable Property runtime As Integer
    Public Overridable Property premiered As Date
    Public Overridable Property schedule As TvShowSchedule
    Public Overridable Property rating As TvShowRating
    Public Overridable Property weight As Integer
    Public Overridable Property network As TvNetwork
    Public Overridable Property webChannel As String
    Public Overridable Property externals As TvShowExternals
    Public Overridable Property image As TvShowImage
    Public Overridable Property summary As String
    Public Overridable Property updated As Integer
    Public Overridable Property _links As TvShowLinks
End Class

Partial Public Class JsonStringCollection
    Inherits ObjectModel.Collection(Of String)
End Class

Partial Public Class TvShowSchedule
    Public Overridable Property time As Date
    Public Overridable Property days As JsonStringCollection
End Class

Partial Public Class TvShowRating
    Public Overridable Property average As Integer
End Class

Partial Public Class TvNetwork
    Public Overridable Property id As Integer
    Public Overridable Property name As String
    Public Overridable Property country As TvNetworkCountry
End Class

Partial Public Class TvNetworkCountry
    Public Overridable Property name As String
    Public Overridable Property code As String
    Public Overridable Property timezone As String
End Class

Partial Public Class TvShowExternals
    Public Overridable Property tvrage As Integer
    Public Overridable Property thetvdb As Integer
    Public Overridable Property imdb As String
End Class

Partial Public Class TvShowImage
    Public Overridable Property medium As String
    Public Overridable Property original As String
End Class

Partial Public Class TvShowLinks
    Public Overridable Property self As LinkUrl
    Public Overridable Property previousepisode As LinkUrl
    Public Overridable Property nextepisode As LinkUrl
End Class

Partial Public Class LinkUrl
    Public Overridable Property href As String
End Class

---

Reed Kimble - "When you do things right, people won't be sure you've done anything at all"

Answer 9

Here's a link to the generator.

Add a reference to SimpleJson.dll and then add a copy of MakePoco.tt to the project.  From the solution explorer, right click the copy of MakePoco.tt in the project and select Run Custom Tool.  You'll have to agree to a security warning.

You'll then get the generator dialog:

Load the JSON from a file, web resource, or directly entered text to populate the tree view.  Then use the toolbar above the treeview to discover the data objects which require type names.  Start with the Object type selected in the drop down and click the first button on the toolbar.  This will find the first Object requiring a type name.  In the property grid, give the type a name:

Now name the new object.  For the first object you usually don't have much to go on and will have to kind of know something about the data you are working with.  In this case (and in the example POCO previously posted) I named the root object "TvShowWinner".

After you name the first object, click the Find Next button again.  You can also press CTRL+N to find the next Object to name without leaving the keyboard.  Continue this process until you get the message that all objects have been named.

Then change the dropdown from Object to Array.  Click the Find Next button.  You can then click the second button "Attempt to resolve types for empty arrays".  If any instance of the object in the data has values in this array, that data type will be used for this empty array.  Continue to use Find Next/Resolve Array until you get the message that all objects have been assigned a type (should just take one time through).

Finally, click the third button to "Fix properties with null values".  This one attempt to set null property value instances to the correct type based on a sibling entry if one is found.  If none of the JSON objects has a value for the property you'll be given a warning and taken to the first instance that requires a manual type name.  Provide a type for the empty array such as String or Integer (Object should work too, though I haven't tried it).  As with the first two, continue to click the third button until you get the message that everything is good.

Now you can click OK on the main dialog window and the POCO will be generated for the JSON model.  This will be in a hidden MakePoco.vb file under MakePoco.tt.  Just click the Show All Files button in the solution explorer to browse the resulting code.

That's the quickest way to use the generator, but you can also browse through the tree and manually update generated type names.  In most cases the program will attempt to find all other instances of the same element and update them as well.

This code has been through a couple of iterations and was a "figure-it-out-as-you-go" kind of development so I know there are some things which can be optimized yet and there may be a bug or two hiding in there.  It would be great if any of you who are interested in this could try it out with a few various sources and try using the resulting POCO with Newtonsoft or any other JSON tools, and see how it works and if you run into any problems.

Its also possible to just use the SimpleJson parser and object model for your own purposes.  I still need to comment it, but it's fairly straight forward.

---

Reed Kimble - "When you do things right, people won't be sure you've done anything at all"

Answer 10

I've found a couple bugs already myself when working with Newtonsoft.  One thing I've already fixed is using Nullable(Of T) for value types so that Null values can be set during deserialization.

The other thing I've found is that some numeric property values may not have a decimal point in every instance so its possible to misinterpret a number as an integer when other instance have a float.  I'll have to add another check to the generator to look for this kind of situation.

---

Reed Kimble - "When you do things right, people won't be sure you've done anything at all"

Answer 11

OK the known bugs are fixed and the code now generates an object model that works with Newtonsoft.

I've updated the package on OneDrive and have an updated link incase the old one is invalid.

There's a fourth button in the interface now which consolidates property types and makes sure they are consistent.  The third button also defaults null property values to Object if none of the item instances have a value, so you only have to click that button once now.

Here's what the generated code looks like with the updates:

Partial Public Class TvShowWinnerCollection
    Inherits ObjectModel.Collection(Of TvShowWinner)
End Class

Partial Public Class TvShowWinner
    Public Overridable Property id As Nullable(Of Integer)
    Public Overridable Property url As String
    Public Overridable Property name As String
    Public Overridable Property season As Nullable(Of Integer)
    Public Overridable Property number As Nullable(Of Integer)
    Public Overridable Property airdate As Nullable(Of Date)
    Public Overridable Property airtime As String
    Public Overridable Property airstamp As Nullable(Of Date)
    Public Overridable Property runtime As Nullable(Of Integer)
    Public Overridable Property image As TvShowImage
    Public Overridable Property summary As String
    Public Overridable Property show As TvShow
    Public Overridable Property _links As TvShowLinks
End Class

Partial Public Class TvShow
    Public Overridable Property id As Nullable(Of Integer)
    Public Overridable Property url As String
    Public Overridable Property name As String
    Public Overridable Property type As String
    Public Overridable Property language As String
    Public Overridable Property genres As JsonStringCollection
    Public Overridable Property status As String
    Public Overridable Property runtime As Nullable(Of Integer)
    Public Overridable Property premiered As Nullable(Of Date)
    Public Overridable Property schedule As TvShowSchedule
    Public Overridable Property rating As TvShowRating
    Public Overridable Property weight As Nullable(Of Integer)
    Public Overridable Property network As TvNetwork
    Public Overridable Property webChannel As Object
    Public Overridable Property externals As TvShowExternals
    Public Overridable Property image As TvShowImage
    Public Overridable Property summary As String
    Public Overridable Property updated As Nullable(Of Integer)
    Public Overridable Property _links As TvShowLinks
End Class

Partial Public Class JsonStringCollection
    Inherits ObjectModel.Collection(Of String)
End Class

Partial Public Class TvShowSchedule
    Public Overridable Property time As String
    Public Overridable Property days As JsonStringCollection
End Class

Partial Public Class TvShowRating
    Public Overridable Property average As Nullable(Of Decimal)
End Class

Partial Public Class TvNetwork
    Public Overridable Property id As Nullable(Of Integer)
    Public Overridable Property name As String
    Public Overridable Property country As TvNetworkCountry
End Class

Partial Public Class TvNetworkCountry
    Public Overridable Property name As String
    Public Overridable Property code As String
    Public Overridable Property timezone As String
End Class

Partial Public Class TvShowExternals
    Public Overridable Property tvrage As Nullable(Of Integer)
    Public Overridable Property thetvdb As Nullable(Of Integer)
    Public Overridable Property imdb As String
End Class

Partial Public Class TvShowImage
    Public Overridable Property medium As String
    Public Overridable Property original As String
End Class

Partial Public Class TvShowLinks
    Public Overridable Property self As LinkUrl
    Public Overridable Property previousepisode As LinkUrl
    Public Overridable Property nextepisode As LinkUrl
End Class

Partial Public Class LinkUrl
    Public Overridable Property href As String
End Class

And here is an example of how you can parse the JSON into the above object model using Newtonsoft:

Private Async Sub Button1_Click(sender As Object, e As EventArgs) Handles Button1.Click
    Dim jsonString As String
    Dim winnerList As New TvShowWinnerCollection
    Using client As New Net.Http.HttpClient
        jsonString = Await client.GetStringAsync("http://api.tvmaze.com/schedule?country=US")
    End Using
    winnerList = JsonConvert.DeserializeAnonymousType(jsonString, winnerList)
    For Each winner In winnerList
        ListBox1.Items.Add(winner.name)
    Next
End Sub

---

Reed Kimble - "When you do things right, people won't be sure you've done anything at all"

Answer 12

Just added one more bug fix for an issue that was introduced in the last update that had to do with null entries in a top-level array.  So far I've run this against three different sources of JSON data and its worked well with all of them.

---

Reed Kimble - "When you do things right, people won't be sure you've done anything at all"

Answer 13

Absolutely fantastic, transformed my task from a few hundred shameful looking lines of code into a simple, logical, easy to read and understand parsing.

Devon

Answer 14

Absolutely fantastic, transformed my task from a few hundred shameful looking lines of code into a simple, logical, easy to read and understand parsing.

Devon

Devon, glad it helped.  Out of curiosity, did you run the POCO generation yourself or just go with what I posted?  If you ran the generator I'm interested in your experience.

---

Reed Kimble - "When you do things right, people won't be sure you've done anything at all"

Answer 15

Absolutely fantastic, transformed my task from a few hundred shameful looking lines of code into a simple, logical, easy to read and understand parsing.

Devon

Devon, glad it helped.  Out of curiosity, did you run the POCO generation yourself or just go with what I posted?  If you ran the generator I'm interested in your experience.

---

Reed Kimble - "When you do things right, people won't be sure you've done anything at all"

I ran the generator several times with your post right there so I could use the same names, here are the results of the run (as a list of differences). I am running it on VS2015 Community.

'Yours - lines 43,44,45
Partial Public Class JsonStringCollection
    Inherits ObjectModel.Collection(Of String)
End Class

'Mine - lines 43,44,45
Partial Public Class JsonStringCollection
    Inherits ObjectModel.Collection(Of )
End Class

'Yours - lines 85,86,87 - Mine is same - included to show the end
Partial Public Class LinkUrl
    Public Overridable Property href As String
End Class

'Mine - lines 85,86,87
Partial Public Class LinkUrl
    Public Overridable Property href As String
End Class

'Yours - Finished

'Mine - Lots More
Partial Public Class GenresData2
    Inherits ObjectModel.Collection(Of )
End Class

Partial Public Class GenresData3
    Inherits ObjectModel.Collection(Of )
End Class

Partial Public Class GenresData4
    Inherits ObjectModel.Collection(Of )
End Class

Partial Public Class GenresData5
    Inherits ObjectModel.Collection(Of )
End Class

Partial Public Class GenresData6
    Inherits ObjectModel.Collection(Of )
End Class

Partial Public Class GenresData7
    Inherits ObjectModel.Collection(Of )
End Class

Partial Public Class GenresData8
    Inherits ObjectModel.Collection(Of )
End Class

Partial Public Class GenresData9
    Inherits ObjectModel.Collection(Of )
End Class

Partial Public Class JsonStringCollection0
    Inherits ObjectModel.Collection(Of )
End Class

Partial Public Class JsonStringCollection1
    Inherits ObjectModel.Collection(Of )
End Class

Partial Public Class JsonStringCollection3
    Inherits ObjectModel.Collection(Of )
End Class

Partial Public Class JsonStringCollection3
    Inherits ObjectModel.Collection(Of )
End Class

Partial Public Class JsonStringCollection4
    Inherits ObjectModel.Collection(Of )
End Class

Partial Public Class JsonStringCollection5
    Inherits ObjectModel.Collection(Of )
End Class

Partial Public Class JsonStringCollection6
    Inherits ObjectModel.Collection(Of )
End Class

Partial Public Class JsonStringCollection7
    Inherits ObjectModel.Collection(Of )
End Class

Partial Public Class JsonStringCollection8
    Inherits ObjectModel.Collection(Of )
End Class

Answer 16

Well now that's interesting...  I'm not sure how you wound up with all of those empty collection objects.  I've haven't seen that behavior before.

---

Reed Kimble - "When you do things right, people won't be sure you've done anything at all"

Answer 17

Well now that's interesting...  I'm not sure how you wound up with all of those empty collection objects.  I've haven't seen that behavior before.

---

Reed Kimble - "When you do things right, people won't be sure you've done anything at all"

If there is anything I can provide, I'd be glad to do so - run again, etc.

Edit - I tried the first two examples from json.org and rec'd this

Unexpected Lexer Error

Edit2 - I just tried http://api.tvmaze.com/shows and got an exception : "At least one object must implement IComparable"

Answer 18

...

If there is anything I can provide, I'd be glad to do so - run again, etc.

Edit - I tried the first two examples from json.org and rec'd this

Unexpected Lexer Error

Edit2 - I just tried http://api.tvmaze.com/shows and got an exception : "At least one object must implement IComparable"

Thanks so much.

Something sounds strange... I just ran the examples from json.org and that URL and everything parsed perfectly.  I wonder if you have the most recent publish?

-edit-

When pasting multiline JSON, you have to use the [..] button and enter it through the sub-dialog; you can't paste multiline text directly into the textbox on the main dialog form.

---

Reed Kimble - "When you do things right, people won't be sure you've done anything at all"

Answer 19

...

If there is anything I can provide, I'd be glad to do so - run again, etc.

Edit - I tried the first two examples from json.org and rec'd this

Unexpected Lexer Error

Edit2 - I just tried http://api.tvmaze.com/shows and got an exception : "At least one object must implement IComparable"

Thanks so much.

Something sounds strange... I just ran the examples from json.org and that URL and everything parsed perfectly.  I wonder if you have the most recent publish?

-edit-

When pasting multiline JSON, you have to use the [..] button and enter it through the sub-dialog; you can't paste multiline text directly into the textbox on the main dialog form.

---

Reed Kimble - "When you do things right, people won't be sure you've done anything at all"

The Link I used to download the Generator was in this post

Thursday, April 07, 2016 9:44 PM

OK the known bugs are fixed and the code now generates an object model that works with Newtonsoft.

I've updated the package on OneDrive and have an updated link incase the old one is invalid.

https://onedrive.live.com/redir?resid=8357F4A651E4838B!1522&authkey=!AJaSyhFVIIkYS3s&ithint=file%2czip