locked
How to access IP-restricted resources from Azure Databricks RRS feed

  • Question

  • The question is rather straightforward. How can I define a predictable outbound IP address for virtual machines running in an Azure Databricks cluster? The VMs are inside a VNET dedicated to Databricks. The resources I'm trying to access are non-native so VNET storage endpoints are not an option - the resources themselves also cannot be provisioned inside a VNET.

    In other words, I have a storage resource e.g. a database protected with a simple IP-based ACL. How would I go about enabling access to said resource from Azure Databricks - from Databricks-managed VMs inside a VNET?

    Azure Databricks does allow me to provision cluster VMs within a VNET under my control, in which case I would be able to create UDRs. They call this feature "VNET injection".

    Wednesday, June 26, 2019 6:06 AM

Answers

  • I am unsure from the Databricks perspective, but if it is deployed using a VMSS and a load balancer, and the VMs do not have public IPs, all of the requests will come from the public IP of the load balancer. 

    If Azure Databricks is inside your VNET, you can always use UDR to send outbound internet traffic to an NVA, and then all requests will come from the Public IP of the NVA. 

    Thursday, June 27, 2019 11:35 PM

All replies

  • I am unsure from the Databricks perspective, but if it is deployed using a VMSS and a load balancer, and the VMs do not have public IPs, all of the requests will come from the public IP of the load balancer. 

    If Azure Databricks is inside your VNET, you can always use UDR to send outbound internet traffic to an NVA, and then all requests will come from the Public IP of the NVA. 

    Thursday, June 27, 2019 11:35 PM
  • Thank you for the response. Unfortunately the VMs are not deployed using a VMSS and I doubt I have any control over that since the (Spark) cluster configuration is fully managed through Databricks.

    Regarding the other alternative. If cost wasn't an issue, could I use Azure Firewall instead of a 3rd party NVA?

    Furthermore, are those (Azure LB & routing through an NVA) the only options for getting predictable outbound IPs for VMs running inside a VNET?


    • Edited by vjraitila Friday, June 28, 2019 2:37 PM
    Friday, June 28, 2019 2:35 PM
  • Yes, an Azure Firewall can absolutely be used, and you can find instructions here
    Friday, June 28, 2019 6:59 PM