Azure Databricks RRS feed

  • Question

  • I am invoking Azure Databricks Child Notebook from Parent Notebook with some parameters being passed.

    I am facing limitations in number of character while passing parameters. As part of parameter passing, I am passing Table Schema (Name CHAR(5), Age INT etc.,) to the child notebook. This Table Schema would be referred in child notebook instead of default inferSchema as true.

    rawDF = spark.read.csv(v_rawZoneInputPath,header=True,schema=v_odsDBFSTableSchema)

    The problem is, my table is having around 450+ columns and Databricks, notebook does not support passing parameters with more characters.

    The error:

    com.databricks.WorkflowException: com.databricks.common.client.DatabricksServiceException: INVALID_PARAMETER_VALUE: The provided parameters (in json representation) are too long (10046 bytes); it cannot exceeds 10000 bytes.
    • Edited by BalanMurugan Wednesday, October 7, 2020 12:06 PM
    Wednesday, October 7, 2020 12:05 PM