none
HDInsight Emulator version 1.0.0.0 is released RRS feed

  • General discussion

  • Hi all, 

    In conjunction with the announcement of the general availability of Windows Azure HDInsight, I am pleased to announce we have also released version 1.0.0.0 of the Microsoft HDInsight Emulator for Windows Azure, formerly known as Microsoft HDInsight Developer Preview. It is installable via the Web Platform Installer. This product continues to target developer scenarios and as such only supports single-node deployments.

    How do I get this?

    Simply by installing HDInsight out of WebPI onto a new machine after uninstalling (in place upgrade is not currently supported).  To completely uninstall previous versions, first uninstall “Microsoft HDInsight Developer Preview” and then the “Hortonworks Data Platform”.  We will not preserve data in HDFS using uninstall/reinstall, so archive your data prior to uninstalling the older version(s).

    What’s new?

    • Scripts have been added to simplify setting all Apache Hadoop services to automatic or manual start. The default will still be automatic as before, but all services can now be changed using the set-onebox-manualstart.cmd or set-onebox-autostart.cmd scripts that are installed in C:\Hadoop.
    • The number of required installation dependencies has been reduced significantly, allowing for faster installations.
    • Bug fix in the command used to run Pig samples in the RunSamples.ps1 script installed in the GettingStarted folder.
    • This version contains an update to the Hortonworks Data Platform version 1.1 that matches the Hortonworks Data Platform services available with Windows Azure HDInsight cluster version 1.6.

    What if something doesn’t work?

    Please report issues here on the forum. We’re also tracking known issues via the release notes.

    Thanks,

    Sarah Parra, Microsoft HDInsight Team


    This posting is provided "AS IS" with no warranties, and confers no rights.


    Tuesday, October 29, 2013 4:19 PM

All replies

  • I installed the emulator, created a Hive table, and used SQOOP to import data from SQL Azure. That all worked great and required very little "fiddling with" to complete. But now I can not find the data that was loaded. On Azure, the data is stored in blobs that are easy to find and view.

    So here are my questions...

    Does the HDInsight emulator use blob storage?

    Does it use blob storage on the Azure emulator?

    Can I connect to it from the Visual Studio Server Explorer?


    Bill McCann CTO AlignCare Services, LLC

    Tuesday, November 5, 2013 5:00 PM
  • By default the emulator uses HDFS and not blob storage. You can view the data in HDFS using the "hadoop fs" commands from the Hadoop Command Line that is installed with the emulator. More details on the available options can be found here. I think the VS explorer just supports blob storage, so it wouldn't help with this.

    Having said that, you can also configure your emulator to use blob storage, so it will be closer to the experience you get with Azure, and then you'll be able to access the data just like you can with an Azure cluster. Take a look at the release notes here. Look for the section titled "Configuring ASV on the local cluster". Add the 3 sections listed there to your core-site.xml file, restart the services as described, then the default will use blob storage in Azure. In the places where you need to provide an account name, provide the full name, e.g. mystorage.blob.core.windows.net.

    Regarding the Azure emulator, I think it should be possible to use the instructions above to configure the HDInsight emulator to use the Azure emulator instead of a regular blob storage account, but we don't yet support that, so I don't have exact instructions for you on the settings required to make it work. The instructions referenced in the link above should help you get it working with blob storage though.

    Thanks,

    Sarah


    This posting is provided "AS IS" with no warranties, and confers no rights.

    Tuesday, November 5, 2013 11:36 PM
  • I could not find HDInsight in WebPI 4.6.  I tried searching HDInsight in the upper right hand corner search box, still none shows up.  what am I doing wrong?

    Gaja Kannan

    Wednesday, November 6, 2013 5:35 PM
  • Do I need to add any custom feeds to webpi?  my webpi shows "default" as a feed and I am using win 7 machine

    Gaja Kannan

    Wednesday, November 6, 2013 5:43 PM
  • Gaja,

    You will find HDInsight in Web PI 4.6. Click on the "Products" tab, then choose "Server" from the menu in the left panel. You are looking for "Microsoft HDInsight Emulator for Windows Azure". The listed applications changed from time to time, but as of this morning it is the eighth item listed. You can also type HDInsight in the search box (this is what I did).

    Note: this is the development platform, not the full-blown Hortonworks Data Platform.

    Bill


    Bill McCann CTO AlignCare Services, LLC

    Thursday, November 7, 2013 2:36 PM
  • Hey Sarah,

    Thanks for the quick response and explanation. I'll try your suggestions.

    Bill


    Bill McCann CTO AlignCare Services, LLC

    Thursday, November 7, 2013 2:38 PM
  • We will have a new article published to windowsazure.com in 1-2 weeks.  Use these keywords: hdinsight emulator to find the article.

    The article covers connecting to the Windows Azure Storage emulator, connecting to a Windows Azure Blob storage container, and configuring a Windows Azure Blob Storage container as the default file system for HDInsight emulator.

    To connect to the storage emulator, you don't need to configure the core-site.xml file.  The syntax is WASB://<ContainerName>@storageemulator.  For example: hadoop fs -ls wasb://myContainer@storageemulator. 

    Thursday, November 7, 2013 3:48 PM
  • HDInsight doesn't show up in WebPI if your operating system doesn't meet the minimum requirements. It has to be 64-bit, and one of the following versions or higher:

    • Windows 8
    • Windows Server 2012
    • Windows 2008 R2 SP1
    • Windows 7 SP1

    Since you mentioned Win7, I would guess you either have a 32-bit machine or don't have SP1.

    Thanks,

    Sarah


    This posting is provided "AS IS" with no warranties, and confers no rights.

    Friday, November 8, 2013 6:43 PM
  • That makes sense.  I have Win7 SP1 (32-bit machine).  In the process of building a 64-bit. 

    Thanks,


    Gaja Kannan

    Friday, November 8, 2013 6:49 PM
  • Are there plans to have a version of the emulator that matches the HDP services available with HDInsight 2.1? I want to use the Avro SerDe, but it doesn't exist in HIVE 0.9. 
    Tuesday, November 19, 2013 10:04 PM
  • I expect that will happen at some point, but we don't have a set timeframe for exactly when it will be.

    Thanks,

    Sarah


    This posting is provided "AS IS" with no warranties, and confers no rights.

    Wednesday, November 20, 2013 12:19 AM
  • By default, HDInsight emulator uses HDFS.  However you can configure it to use either Windows Azure storage emulator or Windows Azure Blob storage. It is documented here:

    http://www.windowsazure.com/en-us/manage/services/hdinsight/get-started-with-windows-azure-hdinsight-emulator/#blobstorage

    Thursday, December 5, 2013 5:32 PM
  • Thanks a lot,

    I've followed instructions but after installation (of HDisnight emulator via WebInstaller), I see that only "Apache hadoop oozieservice " and "Apache hadoop templeton" are running, but none of the other services. Please tell me what wrong with my installation (which was done on win 7 professionnal) or if there's some issue with this problem.

    Monday, December 9, 2013 10:26 PM
  • Are you installing on a non-English installation of Windows, by any chance? We only support English at this point, but I have seen similar behavior where only Oozie and Templeton install on a non-English OS, so I am wondering if that might be the case for you.

    Thanks,

    Sarah


    This posting is provided "AS IS" with no warranties, and confers no rights.

    Monday, December 9, 2013 10:36 PM
  • Great thaks for your answer, in fact i'm on a french win7 version.

    Should I install it on a VM with english version of win7?

    Or there's other issue ?

    Tuesday, December 10, 2013 1:32 PM
  • Yes, please try on English, that is the only language that is currently supported.

    Thanks,

    Sarah


    This posting is provided "AS IS" with no warranties, and confers no rights.

    Tuesday, December 10, 2013 6:33 PM
  • Thanks,

    I'll try that and give back any issues.

    Wednesday, December 11, 2013 9:03 AM