locked
Kafka changes through Event hubs, using capture into blob storage, some files don't appear RRS feed

  • Question

  • Hi All,

    So I'm streaming events through event hubs as you do. Not many tbh, it starts with a full table sync 8k rows, then monitors changes using SQL Server CDC. It seems to work fine and Event Hubs activity charts show my test changes coming through. However, I have blob storage capture set on the topic, and I get some files and not others. Latest test shows it got 600kb of data. Subsequent squirts of events to events hub and no files :-(

    So I've turned the time and size window right down, turned on the box that says "dump empty files if no activity". Nothing. But event hubs still registers the stream coming in.

    Blob storage is a general purpose StorageV2 data lake

    Can anyone tell me what am I doing wrong please?

    Ta
    Jim

    Friday, January 31, 2020 8:47 PM

Answers

  • Hiya @SumanthMarigowda-MSFT, 

    Thanks for your response. I've been reading more into this. I think this is a permissions problem (d'oh). So there are instructions on the event hubs page how to setup capture into a data lake or blob storage. Data lake drop down list doesn't see my gen2 data lake, which is weird. So I select the lake from "blob storage" ddl. This sets up ok. There is then a section in the web documentation that says about setting permissions in data lake gen1, but not gen2. Looks like it's changed a bit and tbh, I had a fiddle around with it but I couldn't get it to work.

    To summarise. I have a stream of events hitting event hubs. There is a "capture" function in event hubs that will take a time limit or size limit of events, put them in a file and stick them in blob storage. I plan to use ADF to run a pipeline into a structured data store, but I can't get it to product the files consistently at first. And now, not at all.

    Thanks for your time

    Jim

    Thursday, February 6, 2020 2:49 PM

All replies

  • @JimboJimmyJimJims Can you provide more information on your query? Is this happening to specific files?

    May I know what kind of files are not showing up?


    Tuesday, February 4, 2020 12:18 PM
  • @JimboJimmyJimJims Just checking in to see if you have had a chance to see the previous response. Could you share the above required information to understand/investigate this issue further?

    Thursday, February 6, 2020 3:07 AM
  • Hiya @SumanthMarigowda-MSFT, 

    Thanks for your response. I've been reading more into this. I think this is a permissions problem (d'oh). So there are instructions on the event hubs page how to setup capture into a data lake or blob storage. Data lake drop down list doesn't see my gen2 data lake, which is weird. So I select the lake from "blob storage" ddl. This sets up ok. There is then a section in the web documentation that says about setting permissions in data lake gen1, but not gen2. Looks like it's changed a bit and tbh, I had a fiddle around with it but I couldn't get it to work.

    To summarise. I have a stream of events hitting event hubs. There is a "capture" function in event hubs that will take a time limit or size limit of events, put them in a file and stick them in blob storage. I plan to use ADF to run a pipeline into a structured data store, but I can't get it to product the files consistently at first. And now, not at all.

    Thanks for your time

    Jim

    Thursday, February 6, 2020 2:49 PM
  • @JimboJimmyJimJims Thank for the update! If you face any issue related to Azure,please feel free to contact us anytime. 

    Thanks! 

    Thursday, February 6, 2020 2:57 PM