Anyway to filter this?

Question

Given the code below, is there anyway I can parse content in the response stream using where clause in "query"?  In other words the responsestream will have a bunch of string data, I would like the query to parse it out. 

        public void ObservableHTTPGetRequest()
        {
            var request = (HttpWebRequest)WebRequest.Create(http://Someurl);
            request.Method = "GET";
            var query =
                from response in request.GetResponseObserverable()                 
                select response.GetResponseStream();
            query.Subscribe(
                s => XTB.Dispatcher.Invoke(new Action<Stream>(TheDelegateToCall), s)
                );
        }

---

JP Cowboy Coders Unite!

Answer 1

Hi there,

If I understand your question correctly, you need to do something like this;

from response in request.GetResponseObservable()
let stream = response.GetResponseStream();
let result = stream. /* translate stream */
where /* filter */
select result

Hope that makes sense.

---

James Miles http://enumeratethis.com

Answer 2

Ok so here's the current code for the Observer's filtering after the stream arrives.

  HtmlAgilityPack.HtmlDocument doc = new HtmlDocument();
            doc.LoadHtml(data);
            var a = doc.DocumentNode.DescendantNodesAndSelf().Where(p => p.Name == "a").Select<HtmlNode, KeyValuePair<string, string>>(q => new KeyValuePair<string, string>(q.InnerText, q.OuterHtml)
            );

How would I point to this code in

let result = stream. /* translate stream */

Is the right side able to take func delegate, anonymous delgate?  like this:

let result = stream => ...bunch of code....

---

JP Cowboy Coders Unite!

Answer 3

>Is the right side able to take func delegate, anonymous delgate?  like this:

Yes but you would actually call you func delegate like this;

let result = func(stream)

---

James Miles http://enumeratethis.com

Answer 4

Thanks James I'll give that a try...  Still pretty new at RX and still a bit confused...

But how does this work under the covers, where is the parsing done?  In a separate thread on client?  As each parse occurs one row of data is pushed via the Observable right?   Can I change the "query" to .Observe on XTB.Dispatcher instead of putting it here:

query.Subscribe(s => XTB.Dispatcher.Invoke(new Action<Stream>(TheDelegateToCall), s));

---

JP Cowboy Coders Unite!

Answer 5

let creates a "range varible" in your query. It is like the LINQ/functional equivilent of writing;

var result = /* some expression */;

working example;

from x in Enumerable.Range(1,10)
let y = x * 3
where y == 21
select new{x,y}

In this example, with out let, we would have to compute y more than once.

Hope that makes sense.

---

James Miles http://enumeratethis.com

Answer 6

Ok so now I understand the LINQ operator LET a bit better, thanks!

As far as Observable, the query portion is done in another thread on client? Each result then of the select is pushed to the subscriber, right? 

The reason this is confusing to me is that the server is ALL PULL based (in this case HTTPWEBREQUEST).  The server of the request started by an HTTPGet which is PULL based.  The PUSHING in this case is all client initiated in another thread.  It also seems to PUSH just one parse at a time instead of entire collection (which I don't mind as user sees things sooner)... 

---

JP Cowboy Coders Unite!

Answer 7

It is hard to say, seeing as I don't have all of your requirements, but I'd say you should probably end up with something like this;

        public void ObservableHTTPGetRequest()
        {
            var request = (HttpWebRequest)WebRequest.Create(http://Someurl);
            request.Method = "GET";
            var query =
                from response in request.GetResponseObserverable()
		let stream = response.GetResponseStream()
		let result = Translate(stream)
		where result == /*filter*/
                select result;

            query.ObserveOnDispatcher(XTB.Dispatcher).Subscribe(result => /* */);
        }

	public TResult Translate(Stream stream)
	{
		// read the stream and convert to TResult
	}

You should be able to fill in the blanks. Depending on your requirements you will need to add error handlers, and potentially consider what should happen to the disposable resource returned by calling subscribe.

---

James Miles http://enumeratethis.com

Answer 8

OK. RE: your questions about threads.

The let/where/select statements will run on the thread that calls back with the HTTP response. This will most likely be an thread pool (probably IO) thread, but ultimately that will depend on the implementation of the GetResponseObservable method and the underlying infrastucture it uses. The final result will then be marshalled onto the dispatcher thread, making it safe for you to display the results in the UI.

Unless I'm mistaken you probably want something like this;

[Dispatcher] -> makes request (subscribes) ....... [ThreadPool] -> response ->let -> where -> select -> [Dispatcher] -> update UI

---

James Miles http://enumeratethis.com

Answer 9

Thanks a lot James I will work this up sometime today and post results back here.  This code will allow anyone to use HTTPWEBREQUEST, and with the filter, monitor anything on a web page...  I'll eventually look into using timers but for now, all I'm trying to do is get a list of links that are pre-filtered by end user in the application before the PUSH of data.  If web page has a bunch of links all containing same string this will work perfect for that!

---

JP Cowboy Coders Unite!

Answer 10

Final Solution...  C# / WPF      Thanks to James Miles and Dave Sexton..

How to use RX and HTTPWEBRequest to get all links for any site. 

 On the left is the navigator for all of the links found just a minute ago on Yahoo.com main page.  When clicking on Autos in the navigator the page displays.

USAGE

 Imagine you have a internet site, like we do, that has tons of HTM content.  Each of the files are handled by ASPX which shows them all as a set of links.  But you don't want server side function to sort or exclude the links, you want client side function to parse all the links using RX in the returned OBSERVABLE Collection and not the entire STREAM for HTTPWEBREQUEST (GET).  You also don't want to use the WEBBROWSER To reveal the location where these links were found. 

 

public void ObservableHTTPGetRequest()
{
    //url is hardcoded for test only
    var request = (HttpWebRequest)WebRequest.Create("http://www.yahoo.com");
    request.Method = "GET";
    //set the func to be filtered in query
    Func<Stream, ObservableCollection<KeyValuePair<string, string>>> FilterTheStream = _FilterTheStream;

    //query turns the response into observable
    //ThisStream will be HTTPResponseStream
    //The Results are filtered for specific node type in response stream
    //selection converts to a KeyValuePair of link's InnerText and HREF
    var query =
        from response in request.GetResponseObserverable()
        let ThisStream = response.GetResponseStream()
        let TheResults = FilterTheStream(ThisStream)
        select (TheResults);
    //XWB is a System.Windows.Control.Webbrowser at GUI layer
    //The content is a property of ObservableCollection<KeyValuePair<string,string>> which is bound to GUI control of Items (ofButtons)
    query.ObserveOn(XWB.Dispatcher).Subscribe(s => TheContent = s);
}

private ObservableCollection<KeyValuePair<string, string>> _FilterTheStream(Stream stream)
{
    StreamReader sr = new StreamReader(stream);
    string data = sr.ReadToEnd();
    //Use HTMLAgilityPack to parse the returned stream, create HTMLDOCUMENT
    HtmlAgilityPack.HtmlDocument doc = new HtmlDocument();
    //Load the document from the stream
    doc.LoadHtml(data);
    //grab all nodes of type a == link types pass in HTMLNODE and return a KeyValuePair<string,string>
    var a = doc.DocumentNode.DescendantNodesAndSelf()
        .Where(p => p.Name == "a")
        .Select<HtmlNode, KeyValuePair<string, string>>(
            q => new KeyValuePair<string, string>(q.InnerText, q.OuterHtml)
        );
    var oc = new ObservableCollection<KeyValuePair<string, string>>();
    foreach (var thing in a)
    {
        oc.Add(thing);
    }
    //send back the Observable collection
    return oc;
}

public static readonly DependencyProperty TheContentProperty =
    DependencyProperty.Register("TheContent", typeof(ObservableCollection<KeyValuePair<string, string>>), typeof(RxHttpWebRequest));

public ObservableCollection<KeyValuePair<string, string>> TheContent
{
    get { return (ObservableCollection<KeyValuePair<string, string>>)GetValue(TheContentProperty); }
    set { SetValue(TheContentProperty, value); }
}

private void UserControl_Loaded(object sender, RoutedEventArgs e)
{
    ObservableHTTPGetRequest();
}

private void XBNav_Click(object sender, RoutedEventArgs e)
{
    var btn = sender as Button;
    HtmlAgilityPack.HtmlDocument doc = new HtmlDocument();
    doc.LoadHtml(btn.Tag.ToString());
    foreach (HtmlNode link in doc.DocumentNode.SelectNodes("a"))
    {
        string href = link.Attributes["href"].Value;
        href = Ext.ChangeSlashDirection(href);
        XWB.Source = new Uri(href);
    }
}

The XBNAV_Click event will parse the content of the button's tag and use it to navigate the embedded web browser. The ChangeSlashDirection just get's rid of slashes that are in wrong direction.

Thanks RX forum for the help!

---

JP Cowboy Coders Unite!

Answer 11

The GUI is pleasantly surprized by the Update to the navigator on the left as there is no perceptable delay.  Click on any of the buttons on left immediatly fill the browser with appropriate content.  Note that all the filtering is done within the query and not after the content is pushed.

---

JP Cowboy Coders Unite!