none
Initializing a vector<BYTE> from existing BYTE array

    Question

  • Hello,

    I wish to initialize a vector<BYTE> from an existing BYTE array.

    Simple example:


    BYTE buf[] = {0,1,2,3,4,5};
    std::vector<BYTE> vec(buf, buf + sizeof(buf)/sizeof(buf[0]));

    This sample works and does what I asked for. The only difference is that it copies the buffer. I want the vector to point to the original BYTE array and not allocate+copy it.
    The reason behind this is because I have an old API that sends me a BYTE array (that was read from a file) and I want it in a vector. I can't afford duplicating the BYTE array that I receive because it can get to huge sizes.

    Thanks.

    Tuesday, February 05, 2008 5:47 PM

Answers

  • There's no way to have a std::vector use an existing allocation of memory. 

     

    You could write a std::vector-like wrapper around an existing byte array (naturally, operations that would need to resize the array would have to be left out of the implementation).

     

    There's also the class boost::array (http://www.boost.org/doc/html/array.html) that might be what you're looking for.

     

    Tuesday, February 05, 2008 7:07 PM
  • What exactly are you expecting? 

     

    The size of BYTE buf[] = {0,1,2,3,4,5} is also constant - the only way it won't be constant is if you copy the data into dynamically allocated memory, which you've said you cannot do.  If you need the full capability of std::vector, then you must copy the data, unless you can modifiy the code that's supplying the data to write into an existing std::vector (which it sounds like you also can't do).

     

     

    Tuesday, February 05, 2008 7:33 PM

All replies

  • There's no way to have a std::vector use an existing allocation of memory. 

     

    You could write a std::vector-like wrapper around an existing byte array (naturally, operations that would need to resize the array would have to be left out of the implementation).

     

    There's also the class boost::array (http://www.boost.org/doc/html/array.html) that might be what you're looking for.

     

    Tuesday, February 05, 2008 7:07 PM
  • Hi, thanks for the quick response.
    Unfortunately boost::array is not what I'm looking for since it's size is constant.

    I couldn't figure out what you meant by writing a vector-like wrapper class, can you please elaborate?
    Tuesday, February 05, 2008 7:13 PM
  • What exactly are you expecting? 

     

    The size of BYTE buf[] = {0,1,2,3,4,5} is also constant - the only way it won't be constant is if you copy the data into dynamically allocated memory, which you've said you cannot do.  If you need the full capability of std::vector, then you must copy the data, unless you can modifiy the code that's supplying the data to write into an existing std::vector (which it sounds like you also can't do).

     

     

    Tuesday, February 05, 2008 7:33 PM
  •  Idan K wrote:
    Hi, thanks for the quick response.
    I couldn't figure out what you meant by writing a vector-like wrapper class, can you please elaborate?

     

    Something like this:

     

    Code Snippet

    template <typename T> class fixed_vector

    {

    private:

    T* m_begin;

    T* m_end;

    public:

    typedef T elem_type;

    tyepdef T* iterator;

    typedef const T* const_iterator;

    typedef size_t size_type;

    typedef T& reference;

    typedef const T& const_reference;

     

    fixed_vector(T* begin, T* end)

     : m_begin(begin), m_end(end)

    {
    }

     

    fixed_vector(T*begin, size_t size)

      : m_begin(begin), m_end(begin+size)

    {

    }

     

    size_t size() { return m_end - m_begin; }

    iterator begin() { return m_begin; }

    const_iterator begin() const { return m_begin; }

    iterator end() { return m_end; }

    const_iterator end() const { return m_end; }

     

    reference operator[](size_t idx) { return m_begin[idx]; }

    const_reference T& operator[](size_t idx) const { return m_begin[idex]; }

     

    reference at(size_t idx)

    {

    if (idx > size())

    throw std::out_of_range("fixed vector size exceeded");

    return m_begin[idx];

    }

    const_reference at(size_t idx) const

    {

    if (idx > size())

    throw std::out_of_range("fixed vector size exceeded");

    return m_begin[idx];

    }

     

    reference front() { return m_begin[0]; }

    const_reference front() const { return m_begin[0]; }

     

    // ... and so on


    };

     

 

Reproduce as much (or as little) of the std::vector interface as you require.  Since you're overlaying an existing memory allocation, you cannot implement any of the operations that would require the vector to grow or shrink (push_back, resize, reserve, insert, erase, etc).

 

Tuesday, February 05, 2008 8:23 PM
  • You must have misunderstood my example. As my post indicated it's only an example.

    In my real application I don't have a constant array, I have a dynamically allocated array, such as:

    BYTE* buf = new BYTE[50000000];
    ...

    Now I want to use _this_ array, which is dynamically allocated inside a vector. Allocating more space during the copy of initializing the vector is something I can't afford. Hence my question.
    Wednesday, February 06, 2008 5:47 PM
  • Nice! But as I mentioned in my previous post, I might still need to grow/shrink the vector (could be a parameter to the class indicating wether it can gain ownership of the pointer therefore reallocating it).
    Anyhow this still seems abit messy "copying" the vector behaviour (infact I already made a wrapper class to a memory buffer that currently manages it's memory using a byte array, so I started this whole discussion because I wanted to test the possibility of changing the byte array to a vector).

    Today I thought of a different approach: How about I implement my own allocator class and use it with the vector? In this allocator class I could control the memory managment of the normal STL vector (without having to modify vector code) thereby preventing it from allocating the initial memory in vector's constructor.
    What do you think?
    Wednesday, February 06, 2008 5:55 PM
  • OK, so you want a vector that can grow/shrink, but that takes over ownership of an initial allocation of memory.  Hmmm...

     

    You might be able to do that using a custom allocator - but remember that the vector will use your allocator for all allocations, so your allocator would have to be stateful so that it can return the existing allocation of memory for the initial allocation but then allocate new memory if the vector needs to grow.  Stateful allocators can run afoul of some optimizations that the standard allows (it allows all instances of a particular allocator type to be considered equivalent).

     

    Another option would be to start with an implementation of std::vector (e.g. from StlPort) and modify it to have the ability to pass in an initial allocation of memory (and possibly a way to deallocate that memory - there's no guarantee that std::allocator will use new[] and delete[]).  If you can tollerate relaxed guarantees about exception safety, it's not too hard to simply write your own implementation of std::vector from scratch.  Getting an exception safe version is another matter.

     

    Personally, I'd probably just copy the memory and be done with it.  Unless the initial allocation is 100s of MB in size and happens many times during the lifetime of the program, it's probably not worth the trouble of avoiding the copy.

    Wednesday, February 06, 2008 6:34 PM
  • I thought about copying the memory and using vector as it is but at the moment the overhead of having 2 allocated buffers is something I can't afford (practically I have several threads running, each allocating an array for reading a file, which could get to quite big sizes).

    If I do decide to go with implementing my own allocator, I could theoretically copy the behaviour of the original vector allocator (thus letting the vector do all it's other allocations, if there are any, without getting in the way) and simply add special treatment in the new[] method and somehow signal it when it shouldn't allocate. I've yet to come up with a way to signal it but it doesn't sound unsolveable.
    Thursday, February 07, 2008 5:51 AM