locked
Web api big data RRS feed

  • Question

  • User-506497988 posted
    Hi, i have to create web API to send huge number of records from our oracle database view to use it in one entity which responsible to generate reports etc. From differents sites so they need to take our data every period time i need to know which type web service i have to use and i need sample or link to start learn i hope it is clear
    Regards
    Friday, July 17, 2020 11:32 PM

All replies

  • User-474980206 posted

    Webapi is not really designed for large datasets. You will need to stream the output, and update the timeouts. You might want to look at gRPC which is a better approach 

     https://grpc.io/docs/languages/csharp/quickstart/

    note: if you need to support legacy systems it may not be an option.

    Saturday, July 18, 2020 3:11 PM
  • User-2054057000 posted

    Large amount of data from web api ?

    You design your api to send data in page by page format. This means the client will also send the page no and no of records per page when making the request to the API. 

    The API's job will be to do custom paging on it's end and get that data which the client is asking for. This is what APIs of large corporations employ.

    Wednesday, July 22, 2020 5:38 AM
  • User-2121988648 posted

    HI,

    For large amount of data it is suggested to implement batch jobs, using SSIS or some other ETL tool and store all resultant data in one place. Then use that data to generate reports. You can run that batch Job Every 30 min or so that suits your business needs :)

    Tuesday, July 28, 2020 11:53 AM
  • User-506497988 posted
    Thanks Alot for your replay can you tell more about SSIS Or ETL if you have references and also you know this service will be used by other site need our data as centerlize department
    Tuesday, July 28, 2020 12:27 PM
  • User-2121988648 posted

    You can refer to online tutorials here 

    https://docs.microsoft.com/en-us/sql/integration-services/sql-server-integration-services?view=sql-server-ver15

    You can read everything from development to deployment.

    You can watch YouTube tutorials and there are several courses available out there :)

    Tuesday, July 28, 2020 2:33 PM
  • User-2121988648 posted

    You can place your final consolidated data generated from SSIS/ETL as need by others at one central location.  This data can be in the form of CSV available at one endpoint. Does that make things easy for you ?

    Tuesday, July 28, 2020 3:18 PM
  • User-2121988648 posted

    Is there anything else for which i can help you ?

    Wednesday, July 29, 2020 10:43 AM