Initializing an input factory?

Question

Hello,

I'm working on an input adapter that I'd like to be independent from the code handling the StreamInsight instances, so it can be re-used as is.

This adapter reads a TCP stream in a separate thread, this stream is then shared amongst all the input adapters instantiated by StreamInsight.

I would like to know if StreamInsight provides a mechanism to initialize the input factory so that it receives the TCP server's address when instantiated. I could send the information via the Create() method but it seems to me this is not the proper way, since the information is only needed once. I think a good way is to write a default constructor in the input factory that will fetch the data, but I do not really know what would be considered a best practice.

Also please note that the code should run on both stand-alone and cloud instances of StreamInsight, which makes me look for a very standard way of doing so.

Answer 1

I'd recommend a slight alternation to this approach.  Rather than trying to pre-initialize the factory object with a single configuration (IP binding config) for all adapter instances, I'd recommend going with:

• The adapter factory holds a dictionary of IP Binding -> Listener.
• On creating an adapter instance, the new adapter instance subscribes to the appropriate listener.

This way, if you only ever pass in the request for a single IP resource, you'll only ever create one shared listener.  On Dispose() of the factory, you can then detatch and dispose all of the listeners.

Couple of things to watch out for:

• Binding a TCP listener in Azure is a little different than a standard listener.  You need to bind to the virtual IP endpoint, not the machine's IP (i.e. you need to look up the service endpoint IP from the RoleEnvironment, rather than binding to local addresses)
• Would recommend an intermediary (such as a queue for each adapter instance) between the TCP listeners and the adapter instances

Answer 2

This is something that you would have to do yourself - StreamInsight won't do it for you.

I would, however, recommend against doing that ... doing so would limit you to 1 port/stream for the input adapter as a whole, making your adapter less flexible. The input adapter factory allows you to handle these situations on a case-by-case basis and can work very well for that.

Now ... some things for you to consider. You can't re-use adapter instances between streams. If you try to do that, only the last stream will get the events. You'll have separate data acquistion from the actual enqueuing if you want to be able to share the input events between different streams. The adapter factory would allow you to do that ... you would have an actual adatper instance that subscribes to a publishing data acquisition source that's listening on the port. You'd actually probably be better off using DQC ... see this article on Mark Simms' blog.

If you are using the adapter model, you should have no problems running either on-prem or in the cloud. Now ... the whole opening up a port requirement may pose a problem in the cloud ... I'm not sure. But that's really the only thing.

Does that answer your question?

---

DevBiker (aka J Sawyer)
Microsoft MVP - Sql Server (StreamInsight)

---

---

If I answered your question, please mark as answer.
If my post was helpful, please mark as helpful.

Answer 3

Hi Mark, I didn't want to obscure too much my post with technical details, but I'm going to clarify a few points:

• I'm not using a TCP listener in the input adapter, the input adapter's socket is the client and connects on a remote machine
• I'm using a BlockingCollection<> object in each adapter instance, which seems to be perfect for my use case

Now if I go back to the problem of giving that remote machine's IP address to the input adapter factory, let's say I opt for writing something in the default constructor. I was wondering if StreamInsight would provide some sort of shared memory mechanism, let's say key/value pairs, so first I create a UUID specific to my input adapter, then when a program has to run a query using my input adapter, this UUID must first be used to initialize the values that will be read by the factory's constructor. But maybe I'm being a bit too focused on StreamInsight and .NET itself provides such a scheme for sharing resources among assemblies instantiated in the same VM.

Answer 4

Hi DevBiker, it seems that I lead both Mark and you the wrong way with my first post, please see my reply to his own, I hope it does clarify what I'm doing.

About opening a port in the cloud, I've been told that it might be a bit complicated, unfortunately the administration is still in the process of purchasing an account so I can't give it a try yet. If it can't be avoided, I will fall back to web services (but I'd rather keep the remote machine's code as simple as possible).

Answer 5

Even when using a TCP client - rather than a listener - separating the adapter enqueuing from the data producer would be your ideal scenario. As Mark said, you would then use the adapter factory to keep track of your producers and then associate them with the concrete adapter instance. This will be more efficient on both sides of the equation.

I don't understand what functionality or optimization you are trying to accomplish through pre-initialization of the adapter before any query is started. Yes, you can do this ... but it would be something that you would need to do in your code - StreamInsight won't do it for you. StreamInsight will call your adapter factory and initialize your adapter only when the query is started ... that is, only when you actually need to do it. From what you are describing about your initialization, it seems that you are also limiting yourself to one client connection ... and, while that may be fine for now, I wouldn't count on it and I wouldn't code it with that kind of limitation. I'm concerned that you'll wind up coding yourself into a corner.

Using a data producer/consumer adapter model will also provide you with a core architecture and structure that will allow you to take advantage of some of the enhanced functionality that's coming in the next version of StreamInsight. Take a look at the StreamInsight blog for a high-level preview.

---

DevBiker (aka J Sawyer)
Microsoft MVP - Sql Server (StreamInsight)

---

---

If I answered your question, please mark as answer.
If my post was helpful, please mark as helpful.

Answer 6

Yes you are right, I'm just trying to develop my adapter incrementally rather than doing everything at once, but ideally the TCP code will be separated from the input adapter's code. I guess I will have a better overview of the whole system after coding the main parts and it's no use torturing myself right now on how to do this or that because I can't express very clearly my needs. Never mind if I end up rewriting some parts, we can't always be right from the start.

Thank you for the answers.