locked
Saving subscribed messages received from MQTT publishers RRS feed

  • Question

  • User1960007757 posted

    I'm creating a system where a number of sensors send messages to my ASP.NET API, through the use of MQTT and Mosquitto. I want to save these messages, but I don't really know a way of saving them into the database through the "_context" reference, as I'm subscribing to the routes/queues/topics at startup (not using services, because I don't really know how to create them. just creating a tasks that subscribe to the topics).

    Can anyone please suggest me something to read where I can learn about it, or tell me about your experience in this kind of problem?

    Thanks in advance.

    Friday, October 26, 2018 11:27 AM

All replies

  • User-271186128 posted

    Hi Miguel Costa,

    I'm creating a system where a number of sensors send messages to my ASP.NET API, through the use of MQTT and Mosquitto. I want to save these messages, but I don't really know a way of saving them into the database through the "_context" reference, as I'm subscribing to the routes/queues/topics at startup (not using services, because I don't really know how to create them. just creating a tasks that subscribe to the topics).

    From your description, it seems that you want to save the message into database using entity framework.

    Here are some related articles about how to use Entity Framework, you could refer to them:

    http://www.entityframeworktutorial.net/code-first/simple-code-first-example.aspx

    https://docs.microsoft.com/en-us/aspnet/web-api/overview/data/using-web-api-with-entity-framework/

    Best regards,
    Dillion

    Monday, October 29, 2018 5:20 AM
  • User1960007757 posted

    Yes, using the entity framework is what I want to do and am doing with some parts of my code.

    My problem is that I want to access the database without using any of the model controllers, as I want to publish and subscribe messages to and from my middleware. I want to save the messages that my API receives.

    Sadly, my database's context is disposed of or simply non existant, and I have no idea how I can resolve this.

    I misunderstood your reply, sorry.

    My problem here is that I have a set of queues/topics that I need my API to subscribe to on startup. The name of these queues/topics are stored in my EntityFramework. How can I access them without using an HTTP Request?

    Monday, October 29, 2018 11:43 AM
  • User-271186128 posted

    Hi Miguel Costa,

    Miguel Costa

    The name of these queues/topics are stored in my EntityFramework. How can I access them without using an HTTP Request?

    From your description, you are using EF (Entity Framework) to store the queues/topic message, so we could save the message using the dbContext (_context), instead of using HTTP Request.

    So, I do not quite understand what do you mean about "access them without using Http Request"? Please explain more details about your problem.

    Best regards,
    Dillion

    Monday, November 5, 2018 9:52 AM
  • User1960007757 posted

    I will try to describe the interactions.

    1. The API is launched.
    2. The API accesses the database context automatically and grabs every entity of the type Topic.
    3. With these items, the API opens numerous threads and each of them will subscribe to one topic.
    4. While subscribed, each topic will receive a message. The API must save this message in the context as a Consumption entity.
    5. After starting the multi-threaded subscription process, the Webhost can start.

    From my understanding, the only way I have access to a DB's context reference is when the API receives an HTTP request, like a GET or a POST, as it is only then that the controller is instantiated and a context reference is passed as the system creates a new task for the HTTP request. When this HTTP request is over, the reference is disposed of, and rendered useless. So, how do I access the database to acquire the information I need? I don't know of any way for me to establish a connection with the database for querying and storing without having to manually make a request.

    Thanks,

    Miguel

    Monday, November 5, 2018 2:44 PM
  • User475983607 posted

    Miguel Costa

    I will try to describe the interactions.

    1. The API is launched.
    2. The API accesses the database context automatically and grabs every entity of the type Topic.
    3. With these items, the API opens numerous threads and each of them will subscribe to one topic.
    4. While subscribed, each topic will receive a message. The API must save this message in the context as a Consumption entity.
    5. After starting the multi-threaded subscription process, the Webhost can start.

    From my understanding, the only way I have access to a DB's context reference is when the API receives an HTTP request, like a GET or a POST, as it is only then that the controller is instantiated and a context reference is passed as the system creates a new task for the HTTP request. When this HTTP request is over, the reference is disposed of, and rendered useless. So, how do I access the database to acquire the information I need? I don't know of any way for me to establish a connection with the database for querying and storing without having to manually make a request.

    Thanks,

    Miguel

    Web API is an HTTP service running on a server.  A client which can be a browser or code,  sends a request to the HTTP service and the HTTP Service sends a response. 

    It sounds like you are trying to build a client application or an MQTT server application.  Basically, you are using the wrong technology, WEB API,  to solve a problem.

    I've built tons of sensor acquisition applications and they are pretty simple.  Send an HTTP request (or TCP or UDP) to an endpoint where the request contains sensor data in some known protocol.  The endpoint reads the request and persists the data.  Generally, there is a UI that displays the sensor data stored in the DB.  

    I think you should have two different systems the MQTT system and a web app to View the data. 

    Monday, November 5, 2018 3:13 PM
  • User1960007757 posted

    Miguel Costa

    I will try to describe the interactions.

    1. The API is launched.
    2. The API accesses the database context automatically and grabs every entity of the type Topic.
    3. With these items, the API opens numerous threads and each of them will subscribe to one topic.
    4. While subscribed, each topic will receive a message. The API must save this message in the context as a Consumption entity.
    5. After starting the multi-threaded subscription process, the Webhost can start.

    From my understanding, the only way I have access to a DB's context reference is when the API receives an HTTP request, like a GET or a POST, as it is only then that the controller is instantiated and a context reference is passed as the system creates a new task for the HTTP request. When this HTTP request is over, the reference is disposed of, and rendered useless. So, how do I access the database to acquire the information I need? I don't know of any way for me to establish a connection with the database for querying and storing without having to manually make a request.

    Thanks,

    Miguel

    Web API is an HTTP service running on a server.  A client which can be a browser or code,  sends a request to the HTTP service and the HTTP Service sends a response. 

    It sounds like you are trying to build a client application or an MQTT server application.  Basically, you are using the wrong technology, WEN API,  to solve a problem.

    I've built tons of sensor acquisition applications and they are pretty simple.  Send an HTTP request (or TCP or UDP) to an endpoint where the request contains sensor data in some known protocol.  The endpoint reads the request and persists the data.  Generally, there is a UI that displays the sensor data stored in the DB.  

    I think you should have two different systems the MQTT system and a web app to View the data. 

    I agree with your approach of having two different systems, but what technology do you usually use for your MQTT consumer? The problem with the context still stands, unless you leave the entire ASP.net approach.

    Wednesday, November 7, 2018 9:25 AM
  • User475983607 posted

    I agree with your approach of having two different systems, but what technology do you usually use for your MQTT consumer? The problem with the context still stands, unless you leave the entire ASP.net approach.

    MQTT is a protocol used to allow devices to communicate over TCP.  It is NOT HTTP.  You need a system that translates MQTT to HTTP and invokes an Web API endpoint or simply writes to a database.  Basically, the database persists the data.  The MQTT writes the records.

    You current design is not using Web API correctly.

    Wednesday, November 7, 2018 12:07 PM
  • User1960007757 posted

    You don't understand, haha.

    I'm using MQTT to collect the data by establishing a communication channel between sensors and middleware. My webserver communicates with my middleware server (Mosquitto) using MQTT. From that communication, it receives data and then stores that data in a database.

    Sensor -> Middleware -> API -> Database

    User - makes a request -> API - queries for stored data -> Database

    The Web API is what allows the user to communicate with the system (obtain graphs, register new sensors etc), through calls.

    From what I've studied from the architecture in ASP.NET, I simply have no way of accessing my database except when I make an HTTP call (meaning that right now, I'm not accessing my database at all for MQTT-related-tasks. I use MongoDB as a temporary workaround).

    Hope this clears out my problem.

    Wednesday, November 7, 2018 12:31 PM
  • User475983607 posted

    Correct, you know best... Good luck.

    Wednesday, November 7, 2018 12:43 PM