locked
Unable to push a git repository

    Question

  • Hi,

    I've created a new git-based project in visualstudio.com and I'm trying to push a local repository to it without success.

    Here is the output from command prompt:

    ->
    Counting objects: 61223, done.
    Delta compression using up to 8 threads.
    Compressing objects: 100% (11127/11127), done.
    POST git-receive-pack (chunked)23)
    Unable to rewind rpc post data - try increasing http.postBuffer
    error: RPC failed; result=56, HTTP code = 0
    fWatal: The remote end hung up unexpectedly
    Writing objects: 100% (61223/61223), 476.73 MiB | 45.93 MiB/s, done.
    Total 61223 (delta 51230), reused 58968 (delta 49424)
    fatal: The remote end hung up unexpectedly
    Everything up-to-date
    <-

    It is a relative large repository (about 470 MB).

    Any Ideas?

    Friday, March 28, 2014 1:13 PM

Answers

  • Following comes from the TFS version control PG (Kartik Cating-Subramanian) and tends to resolve most of these issues with a large Git push to VSO.

    --Trevor

    VSO has a 1 hour timeout for all HTTP calls.  This means that if you try to push a large repo (>300MB or so) to VSO, depending on available bandwidth, you may encounter a variety of errors.  Most of these involve the remote end hanging up after 1 hour with nothing having made it to the server.  The mitigation for this scenario, along with any other scenario that needs to push/fetch/clone/pull that might take > 1 hour is to push chunks of history at a time.

    First, get a list of all your commits.

    git rev-list –all –count

    That should give you a rough idea of how many commits you have.  If you don’t have certain “jumbo” commits where you’re checking in large files, then you can assume that you can try to roughly chunk your repo by fractions of these commits.

    Next, get the depth of your master branch (or the primary branch if it’s other than master).

    git rev-list master –first-parent –count

    This should give you the depth of the “first-parent” lineage of master.  Assuming a fairly “regular” workflow where things branch off of master and merge back into it, you can chunk up master into sections of commits, hoping to evenly distribute any merges and the history of other branches within these chunks.  If there are 100k commits, I’d break it up into 5-10 sections.  Remember that more recent history in any repo is usually larger and more complicated, so chunk accordingly.  Let’s say you decide to break it up into 5 chunks of 20k commits each.  After setting up your remotes as usual:

    git push origin master~80000:refs/heads/tmpmaster

    This will push the history sufficient to reach (<tt>master-8000</tt>)’th commit to a remote branch called <tt>tmpmaster</tt>.  The syntax is <tt>local_ref:remote_ref</tt>.  I use <tt>tmpmaster</tt> just in case there is a real master on the server.  If it’s a pristine repo or you don’t care and have force push permissions, you can simply use master on the remote end as well.  Use <tt>+master~8000:refs/heads/master</tt> if you wish to force push (the + stands for “just follow my orders darn it”).

    git push origin master~6000:refs/heads/tmpmaster
    git push origin master~4000:refs/heads/tmpmaster
    git push origin master~2000:refs/heads/tmpmaster
    git push origin master:refs/heads/tmpmaster

    You can now verify from the online interface that most of your repo has made it in.  Note that we have a fairly expensive server side job that runs after every push that helps calculate history and diffs and all that jazz for the online interface – so you may not see the results immediately.  If most of your repo made it online, you can then do the steps below to just pull the remaining branches and tags all in one go.  If you believe that your branches are fairly deep and have significant history of their own that’s somehow isolated from master (remember, anything reachable from the tip of master has already made it up), then you might need to improvise the above steps for the heavy branches.

    git push origin refs/heads/*
    git push origin refs/tags/*
    git push origin :refs/heads/tmpmaster

    These commands push the remaining branches, then all tags (you can do them all in one go if you really want to), and then finally deletes the <tt>tmpmaster</tt> branch.  The <tt>:foo</tt> format with nothing to the left of : implies that you push a “nothing” to the named remote ref, which is the same as saying “please delete the remote ref”.

    That should bring everything you need up to VSO.  You might have to perform a similar dance with clone if you are having trouble running into the timeout but this happens less often as most people have more download bandwidth than upload.


    Trevor Hancock (Microsoft)
    Please remember to "Mark As Answer" the replies that help.


    Friday, April 04, 2014 8:41 PM
    Owner

All replies

  • Hi _Amro,

    I'd like to know whether you still have the issue. How did you publish the local git repository? When use git command, please make sure the syntax is correct. You can also use Team Explorer to do the operations. See this blog and article for more information. If this issue still exist, please elaborate more details about your scenario. Thanks for your understanding.

    Best regards,


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Monday, March 31, 2014 7:33 AM
    Moderator
  • Hi Charles,

    yes, I'm still having trouble pushing the local repository.
    It is a repository for a long existing project which I'm trying to move over to visualstudio.com

    The commands I used were:
    1) git remote add vs_online https://xxx.visualstudio.com/DefaultCollection/_git/xxx
    2) git push -u vs_online --all

    I didn't try it from Visual Studio and I don't think that I can because the command "Publish to <...>" mentioned in your linked articles dosn't show up at all.

    I'm properly connected to visualstudio.com and I have another, smaller, project which is working without any issues.

    Thank you.

    Monday, March 31, 2014 8:47 AM
  • Hi Amro,

    How about you just push one file to that git team project in VSO?

    Please try the following command:

    git remote add origin https://xxx.visualstudio.com/DefaultCollection/_git/xxx
    git push -u origin --all

    And before you can use the command line, you'll need to enable basic authentication for your account. See: http://msdn.microsoft.com/en-us/library/dd286572(v=vs.120).aspx#setup_basic_auth

    If you can't find the "Publish to <...>" option in your Team Explorer, it seems that you already publish it to your git project. How about you open  https://xxx.visualstudio.com/DefaultCollection/_git/xxxin IE? Do you get something? And please also show me the screenshot of the Unsynced Commites page in your Team Explorer.

    Thanks.


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Thursday, April 03, 2014 3:20 AM
    Moderator
  • Hello Vicky,

    I'm trying to do an initial push from a local repository holding a 4 years old project with lot of branches and myriads of commits!
    I can't push a single file unless the initial push was successfull which is not the case. Or can I?

    Using "origin" or anyother upstream name does'nt really matter. The initial push of this project (and only this project) always fails.

    As I mentioned in my last pos, I was successful in pushing another smaller git project to VSO in exact the same way:
    1) Create a new git Project "MyProject" in VSO
    2) git clone <remote URL> my_project
    3) cd my_project
    4) git remote add vso https://whatever.visualstudio.com/DefaultCollection/_git/MyProject
    5) git push -u vso --all

    Please note that allready in step 2 git automatically sets "origin" to <remote URL>.
    When origin is set to anything (the actual value doesn't matter at all) the "Publish to ..." command in VS becomes unavailable. That is why the command doesn't show up!
    (This might be something worth mentioning in your articles, blogs and/or documentations)

    When I open the problematic VSO project in Visual Studio (either from the web or directly from VS) I'm in an entirely empty solution with a plain empty "Unsynced Commits".

    This makes sense of course because the project is still empty.

    It must be something with the size of the repo (notice the message "Unable to rewind rpc post data - try increasing http.postBuffer")!

    Thank you for your time and assistance.


    Thursday, April 03, 2014 1:56 PM
  • Hi Amro,

    Please run the git config http.postBuffer 524288000 command to extend to 500MB.

    See: https://confluence.atlassian.com/display/STASHKB/Git+Push+Fails+-+fatal%3A+The+remote+end+hung+up+unexpectedly and http://stackoverflow.com/questions/2702731/git-fails-when-pushing-commit-to-github

    Thanks.


    We are trying to better understand customer views on social support experience, so your participation in this interview project would be greatly appreciated if you have time. Thanks for helping make community forums a great place.
    Click HERE to participate the survey.

    Friday, April 04, 2014 2:36 AM
    Moderator
  • Hi Vicky,

    already tried that and it still fails but this time with http code 404:

    Friday, April 04, 2014 6:22 AM
  • Following comes from the TFS version control PG (Kartik Cating-Subramanian) and tends to resolve most of these issues with a large Git push to VSO.

    --Trevor

    VSO has a 1 hour timeout for all HTTP calls.  This means that if you try to push a large repo (>300MB or so) to VSO, depending on available bandwidth, you may encounter a variety of errors.  Most of these involve the remote end hanging up after 1 hour with nothing having made it to the server.  The mitigation for this scenario, along with any other scenario that needs to push/fetch/clone/pull that might take > 1 hour is to push chunks of history at a time.

    First, get a list of all your commits.

    git rev-list –all –count

    That should give you a rough idea of how many commits you have.  If you don’t have certain “jumbo” commits where you’re checking in large files, then you can assume that you can try to roughly chunk your repo by fractions of these commits.

    Next, get the depth of your master branch (or the primary branch if it’s other than master).

    git rev-list master –first-parent –count

    This should give you the depth of the “first-parent” lineage of master.  Assuming a fairly “regular” workflow where things branch off of master and merge back into it, you can chunk up master into sections of commits, hoping to evenly distribute any merges and the history of other branches within these chunks.  If there are 100k commits, I’d break it up into 5-10 sections.  Remember that more recent history in any repo is usually larger and more complicated, so chunk accordingly.  Let’s say you decide to break it up into 5 chunks of 20k commits each.  After setting up your remotes as usual:

    git push origin master~80000:refs/heads/tmpmaster

    This will push the history sufficient to reach (<tt>master-8000</tt>)’th commit to a remote branch called <tt>tmpmaster</tt>.  The syntax is <tt>local_ref:remote_ref</tt>.  I use <tt>tmpmaster</tt> just in case there is a real master on the server.  If it’s a pristine repo or you don’t care and have force push permissions, you can simply use master on the remote end as well.  Use <tt>+master~8000:refs/heads/master</tt> if you wish to force push (the + stands for “just follow my orders darn it”).

    git push origin master~6000:refs/heads/tmpmaster
    git push origin master~4000:refs/heads/tmpmaster
    git push origin master~2000:refs/heads/tmpmaster
    git push origin master:refs/heads/tmpmaster

    You can now verify from the online interface that most of your repo has made it in.  Note that we have a fairly expensive server side job that runs after every push that helps calculate history and diffs and all that jazz for the online interface – so you may not see the results immediately.  If most of your repo made it online, you can then do the steps below to just pull the remaining branches and tags all in one go.  If you believe that your branches are fairly deep and have significant history of their own that’s somehow isolated from master (remember, anything reachable from the tip of master has already made it up), then you might need to improvise the above steps for the heavy branches.

    git push origin refs/heads/*
    git push origin refs/tags/*
    git push origin :refs/heads/tmpmaster

    These commands push the remaining branches, then all tags (you can do them all in one go if you really want to), and then finally deletes the <tt>tmpmaster</tt> branch.  The <tt>:foo</tt> format with nothing to the left of : implies that you push a “nothing” to the named remote ref, which is the same as saying “please delete the remote ref”.

    That should bring everything you need up to VSO.  You might have to perform a similar dance with clone if you are having trouble running into the timeout but this happens less often as most people have more download bandwidth than upload.


    Trevor Hancock (Microsoft)
    Please remember to "Mark As Answer" the replies that help.


    Friday, April 04, 2014 8:41 PM
    Owner
  • Please dont give lenghty explanations but fix the bug!

    Sunday, April 13, 2014 4:29 PM