locked
how to start using azure for search, aggregation and visual prototype RRS feed

  • Question

  • Hi. Is there a free environment where I can prove azure is an alternative for me? I'd like to prototype an app that can search thru millions of records or xml like packets, aggregate one or two measures in that data grouped on chosen filters, and visualize the results in a pivot like display/page where the next set of filters and measures can be chosen. I'd like to avoid sql (or any db) for now.  I feel like I can take this and run if someone can get me started.  I'm hearing about products like elastic search, solr, kibana, google's search engine.  I don't know if these products require Linux where maybe aws is a better choice.  I have a ton of background in db products including cubes.  But am trying to think outside the box a little as those products don't always measure up.  I'm looking for world class performance.
    • Edited by db042190 Monday, June 18, 2018 5:10 PM mentioning performance
    Monday, June 18, 2018 5:09 PM

All replies

  • You can create 1 free service per subscription. You can also use our sample real estate data to play.

    You can also use AzSearch.js to generate UI (that can be later changed/extended): https://github.com/jj09/AzSearch.js

    This video about how to get started might be useful as well: https://channel9.msdn.com/Shows/Data-Exposed/Building-Search-Apps-with-Azure-Search-and-AzSearchjs

    Monday, June 18, 2018 5:28 PM
  • thx Jacob.   For someone new, is there a layman's way of explaining what a service is and what a subscription is? I work for a company with 10s of thousands of employees.  What if we already have a subscription in the company?  Can one employee get his/her own subscription?

    ok, I think I see what a service is.  I went to the portal (https://portal.azure.com/) and see stuff listed on the left side like web app, sql server etc that must be services.  So depending on what a subscription is, I think u r telling me that I can pick one of those for free?  If I pick 2, how will they bill me?  If I avoid db's, can I achieve my prototype with just one service?

    • Edited by db042190 Thursday, June 21, 2018 3:10 PM maybe i see what a service is
    Thursday, June 21, 2018 2:17 PM
  • thx, I looked at the video.  Would have to look at it again to digest.  But how do I get started if I want elastic search, kibana  , solr?  I'm guessing they are more grassroots than azsearch which possibly means I would have more control.
    Thursday, June 21, 2018 3:28 PM
  • thx Jacob.   For someone new, is there a layman's way of explaining what a service is and what a subscription is? I work for a company with 10s of thousands of employees.  What if we already have a subscription in the company?  Can one employee get his/her own subscription?

    ok, I think I see what a service is.  I went to the portal and see stuff listed on the left side like web app, sql server etc that must be services.  So depending on what a subscription is, I think u r telling me that I can pick one of those for free?  If I pick 2, how will they bill me?  If I avoid db's, can I achieve my prototype with just one service?

    If you create free service you will not be charged (you choose pricing tier during service creation), you won't be able to create second free service so no worries about being charged. Yes - you can build prototype with 1 service. Like in the linked video.
    Thursday, June 21, 2018 10:18 PM
  • thx, I looked at the video.  Would have to look at it again to digest.  But how do I get started if I want elastic search, kibana  , solr?  I'm guessing they are more grassroots than azsearch which possibly means I would have more control.
    ElasticSearch, Kibana and Solar are different services. Azure Search is built on top of ElasticSearch.
    Thursday, June 21, 2018 10:19 PM
  • appreciated, but how do I get started with these more foundational services?  I didn't see those items in the list of services.
    • Edited by db042190 Tuesday, July 3, 2018 11:12 AM clarity. focus.
    Tuesday, July 3, 2018 11:05 AM
  • appreciated, but how do I get started with these more foundational services?  I didn't see those items in the list of services.

    You do not have to learn about ElasticSearch in order to use Azure Search. It is "implementation detail". Think about it like Operating System in AppService. It is not your problem. The service solves it for you.

    If you are interested in how Azure Search works, this doc might be helpful: https://docs.microsoft.com/en-us/azure/search/search-lucene-query-architecture

    Can you elaborate more on your data. You mentioned that you have "xml like packets". Are these xml files?

    If so you can put them on Azure Storage and Azure Search can import them to search index: https://docs.microsoft.com/en-us/azure/search/search-howto-indexing-azure-blob-storage

    Then it is up to you how you will query and present the data using either our .NET SDK or REST API directly.

    Tuesday, July 3, 2018 6:19 PM
  • thx Jacob.  I understand its not my problem but if its possible, I want to learn a little about it, solr, kibana etc first.  that way I'm not committed to azure.  If that doesn't make sense, sure I'll start with azure search first but azure search sounds experimental at this time and probably not reusable in other cloud environs.

    in my app, I can create one file with a million xml "packets" , one for each acct.  inside each packet are about 5 numerics I want to aggregate on 1 or all of 5 dimensions also included in each packet. 

    My wish is to develop a poor man's visual pivot tool to accomplish this using a powerful search engine (solr?, elastic search?) and viz tool like kibana.  And then get a sense of the performance so I can decide how it compares to tabular cubes, pdw, available pivoting viz tools etc.

    So lets say the 5 numeric tags are sales$, cost$, relationship tenure in months, #employees, #months in business.  And the 5 dimensions are Standard Industrial Code, State, Country, Public vs Private,  Political Affiliation.

    On both the vertical and horiz axis I'd like to be able to drag dimensions, spot where dropped determines hierarchy, and similarly drag numeric and see aggregates at cross section of dimensions.  I'd even be happy starting with one dimension and one numeric (fact).          

       
    Thursday, July 5, 2018 12:41 PM
  • thx Jacob.  I understand its not my problem but if its possible, I want to learn a little about it, solr, kibana etc first.  that way I'm not committed to azure.  If that doesn't make sense, sure I'll start with azure search first but azure search sounds experimental at this time and probably not reusable in other cloud environs.

    in my app, I can create one file with a million xml "packets" , one for each acct.  inside each packet are about 5 numerics I want to aggregate on 1 or all of 5 dimensions also included in each packet. 

    My wish is to develop a poor man's visual pivot tool to accomplish this using a powerful search engine (solr?, elastic search?) and viz tool like kibana.  And then get a sense of the performance so I can decide how it compares to tabular cubes, pdw, available pivoting viz tools etc.

    So lets say the 5 numeric tags are sales$, cost$, relationship tenure in months, #employees, #months in business.  And the 5 dimensions are Standard Industrial Code, State, Country, Public vs Private,  Political Affiliation.

    On both the vertical and horiz axis I'd like to be able to drag dimensions, spot where dropped determines hierarchy, and similarly drag numeric and see aggregates at cross section of dimensions.  I'd even be happy starting with one dimension and one numeric (fact).          

       
    Solr and kibana are not being used in Azure Search.

    Solr is different search engine than ElasticSearch, also built on Apache Lucene. You can look at their comparison here: http://solr-vs-elasticsearch.com/

    Kibana is visualization plugin for ElasticSearch. You would use it if you have your own ElasticSearch instance (not managed by Azure Search). You can learn more here: https://www.elastic.co/products/kibana

    Azure Search is not experimental. It's Generally Available service in multiple Azure Regions that provides SLA. Many organizations are using it in production. What's more, Azure Search takes care of configuring, setting up and scaling your search service. You also get 99.9% SLA.

    From your description it seems like the critical issue in your problem is visualization, not searching. Azure Search will not help with visualization unfortunately. You would have to build it on top of data that you can query using our .NET SDK or REST APIs.
    Thursday, July 5, 2018 10:39 PM