locked
Help with Db Design - Saving Exchange Data RRS feed

  • Question

  • Hello, i was wondering if anyone can help me out start a project, as im new to sql, i have a solid C background, and good understanding of complexity.

    i wanna build a big database, now i can build it on my own using files,arrays, lists ,hashtables etc preety easily, however if i understand correctly ms sql will do it better than me, complexity wise.

    the database will contain quotes from the Exchange market (NASDAQ for the record), i have access to historical data.

    basically i will want to keep these fields StockName, dateTime, DataType, additional fields.

    stockname is the stock name, datetime is the time of the event, datatype referes to what kind of data this table contains it can 1 min data, 1 sec data, 5 mins etc, diffrent kind of data all together (additional fields are diffrent). dataType is not needed if there is a diffrent table for each datatype.

    i will want to querie diffrent things, for example X days of data for a stock, all stocks for a date etc. then i will analyze the data and proceed.

    my question is basically how do i design this in sql, im not sure if its better to build one giant table for all data, diffrent tables for diffrent datatypes or for diffrent symbols, and also diffrent days. there are going to be a lot of records(thousands) for each day per stock. maby i shouold load to the sql my own c classes (after serialazation) with Datetime only etc. whats best in terms of complexity?


    Thanks for your help and time!



    Tuesday, July 26, 2011 1:08 PM

Answers

  • Databases are thought of as sets of data, which is generally different than the way a coder thinks of programs. Databases are also optimized for data writing and retrieval, in a way that no single programmer can realistically match. Saying "i can do this myself" is the first step toward a poor design.

    Coming up with a design can take a bit or mental processing, and most people here are willing to help when they see the first step or two made by the poster. If you would please post your current model, we will be happy to help.

    Tuesday, July 26, 2011 7:15 PM
    Answerer

All replies

  • If you have a good understanding of complexity, and want a big database, you are what we call a programmer. :) A database should not be complex, unless it has to be.

    To the chagrin of database theory purists, i believe that a TABLE represents an object/entity and a COLUMN represents a property of that object. Each record in the TABLE is an instantiation of that object. While there is no inheritance, Relational Integrity is strictly enforced.

    Thus, a good place to start in design is an OO model for this dataset. Create what you think would be the OO model, or at least the basics, and post it here. We can then help you turn that into a Data Model.

    Tuesday, July 26, 2011 3:11 PM
    Answerer
  • the database is indeed not complex no argue there. i understand the diffrence between a table and a column....

    and i understand how to represent my data in an efficiant way. also i allready have my own database with the oo model.

    the problem is when im dealing with a lot of data, it obviusly get somehow slowed, and speed is important to me, 

    i was wondering if il implement my database in ms sql, il get better results.

    i allready specified which data i need to save -> name, datetime, more data which is irrelvant for this debate. i dont need

    any refrences from one table to another and so forth. i just need to save the raw data in an efficiant way.

    lets say i have a table for each stock with its historic data, how will it be compared to one table?

    it depends on how sql implement things. if for example ms sql uses hashtable to find the table, then my cost is the connection

    to the table, if it was one big table im assuming to fetch a series of records will O(logN), and then which is faster?

    i just need to save high amounts of raw data, which is allready in a table like format, that is why there is no need for any fancy deisgn.

     

    thx again

    Tuesday, July 26, 2011 3:41 PM
  • Databases are thought of as sets of data, which is generally different than the way a coder thinks of programs. Databases are also optimized for data writing and retrieval, in a way that no single programmer can realistically match. Saying "i can do this myself" is the first step toward a poor design.

    Coming up with a design can take a bit or mental processing, and most people here are willing to help when they see the first step or two made by the poster. If you would please post your current model, we will be happy to help.

    Tuesday, July 26, 2011 7:15 PM
    Answerer