none
Can the F# HtmlProvider access password protected pages? RRS feed

  • Question

  • I know how to grab data from a web page in F#...

    let url = @"https://en.wikipedia.org/wiki/Doctor_Who"
    type DoctorWhoData = HtmlProvider<url>
    let data = DoctorWhoData.GetSample()

    ...which is really neat, but I want to be able to do this for a password protected page. I can't seem to find any documentation on how to do this.

    Anyone any suggestions? Thanks

    Monday, April 9, 2018 4:43 PM

All replies

  • No-one able to answer this? Surely I'm not the first person to want to do this!
    Sunday, April 15, 2018 3:05 PM
  • hello,

    I'm an F# beginner, and I tried to see if there is a possibility to inherit HtmlProvider : 

    // I tried something like this : 
    // [<TypePorvider>]
    // type public MyHtmlProvider(url: string) as this = 
    //    inherit HtmlProvider<url>(cfg:TypeProviderConfig)    
    // not working I don't know how to deal with this situation 
    // where the type still requiring the url literal
    // 
    
    
    // but came to conclusion that for such a use case 
    // the best solution is to use the Http FSharp.Data extensions, 
    // using for example Http.Request method gives you the ability to specify headers and other options 
    
        let response = Http.Request("https://api.github.com/users/microsoft",
            headers=[  
                ("Authorization", "Basic");
                ("User-Agent", "ie");
                ("Username", "mail@server.com"); 
                ("Password", "P@$$w0rd") ])
    
    

    here is the source code for Http extensions , and source code for HtmlProvider, I think HtmlProvider is to act on html that is already loaded.

    Hope it help;

    Tuesday, September 4, 2018 3:43 PM
  • Wow, that's great! Haven't tried it yet, but the FSharp.Data package is well supported, so I'm sure this will work fine.

    Thanks

    Wednesday, September 5, 2018 3:07 PM
  • Welcome, I'm very happy that it help, for me anything about F# is amazing , I just try it, maybe using it on custom TypeProvider will be as nice as using Authentication with html provider, but I need to learn how to pass string literal the same way as generic parameters, which was what blocked me when i tried to inherit HtmlProvider;

    for now I found an article about custom TypeProviders maybe I'll find the solution there :

    writing an fsharp type provider

    Happy to help;

    Wednesday, September 5, 2018 4:06 PM