locked
Problems pushing large changesets with GIT to Windows Azure RRS feed

  • General discussion

  • Dear forum,


    Recently I made a big change to my open-source project "AzureBlob" and tried to push these changes to my Windows Azure Web Sites (WAWS). But unfortunately I came across a nasty error.

    Let me state what I tried to accomplish.
    mvdam@phpbook: $ git push origin master (so it's updating my forked repository https://github.com/dragonBe/azureblob)
    mvdam@phpbook: $ git push azure master (so it's updating the WAWS on Windows Azure)

    Password for 'https://mvdam@azureblob.scm.azurewebsites.net':
    Unable to rewind rpc post data - try increasing http.postBuffer
    error: RPC failed; result=65, HTTP code = 0
    fatal: The remote end hung up unexpectedly
    fatal: The remote end hung up unexpectedly Everything up-to-date
    At first I thought it had to do something with my account, so I turned to FTP to see if I was able to manually transfer my code onto my WAWS account. And using the same credentials as for my git connection I successfully was able to upload my contents onto the WAWS instance.

    Strange!

    Maybe something was wrong with my repo on this WAWS account, so I created on anther Windows Azure account a brand new website instance, activated the GIT push option and tried to push a new Zend Framework skeleton project on it using GIT. But still got the same error.
    mvdam@phpbook: $ zf create project zftest
    mvdam@phpbook: $ cd zftest
    mvdam@phpbook: $ git init
    mvdam@phpbook: $ git add .
    mvdam@phpbook: $ git commit -m 'Initial commit of empty Zend Framework project'
    mvdam@phpbook: $ git remote add azure https://mvdam@zftest.scm.azurewebsites.net
    mvdam@phpbook: $ git push azure master

    Password for 'https://mvdam@zftest.scm.azurewebsites.net':
    Unable to rewind rpc post data - try increasing http.postBuffer
    error: RPC failed; result=65, HTTP code = 0
    fatal: The remote end hung up unexpectedly
    fatal: The remote end hung up unexpectedly
    Everything up-to-date
    Hmm, maybe something wrong with my account? So again I tried to upload the code with FTP and that worked. Something else was going on.

    I started looking at those error messages. Especially the line stating "Unable to rewind rpc post data - try increasing http.postBuffer" caught my eye. A little search on the internet quickly pointed me to articles describing I should increase the buffer size of my git. I followed the steps described in this article: http://blog.lukebennett.com/2011/07/25/git-broken-pipe-error-when-pushing-to-a-repository/ and increased my http.postBuffer setting:
    mvdam@phpbook: $ git config http.postBuffer 209715200
    Still on this Zend Framework skeleton project, I tried again to push my code onto WASW using GIT.
    mvdam@phpbook: $ git push azure master

    remote: New deployment received.
    remote: Updating branch 'master'.
    remote: .....................
    remote: Preparing deployment for commit id 'f2b5a8c882'.
    remote: Preparing files for deployment.
    remote: .....................................................................................................
    remote: Deployment successful.
    To https://mvdam@zftest.scm.azurewebsites.net/zftest.git
     * [new branch]      master -> master
    Yes, it works!!!

    Ok, let's try again with my AzureBlob codebase.
    mvdam@phpbook: $ git push azure master

    Password for 'https://mvdam@azureblob.scm.azurewebsites.net':
    error: RPC failed; result=56, HTTP code = 0
    fatal: The remote end hung up unexpectedly
    fatal: The remote end hung up unexpectedly
    Everything up-to-date
    Still the same error, but no mentioning of http.postBuffer though.

    Maybe my GIT data was too big, so I start to compress and garbage collect the GIT metadata
    mvdam@phpbook: $ git gc --aggressive

    Counting objects: 9473, done.
    Delta compression using up to 4 threads.
    Compressing objects: 100% (9313/9313), done.
    Writing objects: 100% (9473/9473), done.
    Total 9473 (delta 5457), reused 4016 (delta 0)
    Let's try it again, maybe luck is now on our side.
    mvdam@phpbook: $ git push azure master
    Password for 'https://mvdam@azureblob.scm.azurewebsites.net':
    error: RPC failed; result=56, HTTP code = 0
    fatal: The remote end hung up unexpectedly
    fatal: The remote end hung up unexpectedly
    Everything up-to-date
    Now I'm lost. Is my change too big to push to WAWS or is it something else?
    My .git folder is not that big in my opinion
    mvdam@phpbook: $ du -h .git

      0B    .git/branches
     44K    .git/hooks
    8.0K    .git/info
    8.0K    .git/logs/refs/heads
    4.0K    .git/logs/refs/remotes/azure
     12K    .git/logs/refs/remotes/origin
     12K    .git/logs/refs/remotes/upstream
     28K    .git/logs/refs/remotes
     36K    .git/logs/refs
     44K    .git/logs
    4.0K    .git/objects/info
     12M    .git/objects/pack
     12M    .git/objects
      0B    .git/refs/heads
      0B    .git/refs/remotes/azure
      0B    .git/refs/remotes/origin
      0B    .git/refs/remotes/upstream
      0B    .git/refs/remotes
      0B    .git/refs/tags
      0B    .git/refs
     13M    .git
    So, has anyone faced this issue and found a solution for it? If yes, please reply here in this forum.

    Best regards,

    Michelangelo
    Tuesday, September 4, 2012 7:34 AM

All replies

  • I just tried to repro this with your exact repo, and was not able to. So the exact steps I took were simply:

    • Clone your repo from github
    • Push it to a new Azure Web Site

    It took a bit over 5 minutes, but the push succeeded:

    Counting objects: 9473, done.
    Delta compression using up to 8 threads.
    Compressing objects: 100% (4976/4976), done.
    Writing objects: 100% (9473/9473), 12.14 MiB | 302 KiB/s, done.
    Total 9473 (delta 4337), reused 9473 (delta 4337)
    remote: New deployment received.
    remote: Updating branch 'master'.
    remote: .......................................................................................
    remote: Preparing deployment for commit id '6919ffc3d6'.
    remote: Preparing files for deployment.
    remote: ........................................................................................
    ............
    remote: Deployment successful.
    To https://davidebbo@azureblob3.scm.azurewebsites.net/AzureBlob3.git
     * [new branch]      master -> master

    To isolate the problem, can you try with these exact steps? Maybe there is something different about the copy of the repo currently on your machine.

    Tuesday, September 4, 2012 7:57 PM
  • I'm having the EXACT same problem. I tried with a smaller repo first and it worked. Every push since I merged a bunch stuff has failed with the same error "Unable to rewind rpc post data - try increasing http.postBuffer." It hangs around the 5.56mb mark. Below is exactly what I get in the console:

    ➜  www-alpha git:(master) git push azure master
    Password for '[insert_account_info]': 
    Counting objects: 3829, done.
    Delta compression using up to 8 threads.
    Compressing objects: 100% (2072/2072), done.
    Unable to rewind rpc post data - try increasing http.postBuffer
    error: RPC failed; result=65, HTTP code = 0
    fatal: The remote end hung up unexpectedlyB | 3 KiB/s   
    Writing objects: 100% (3829/3829), 7.03 MiB | 2 KiB/s, done.
    Total 3829 (delta 2811), reused 2447 (delta 1737)
    fatal: The remote end hung up unexpectedly
    Everything up-to-date

    Wednesday, September 5, 2012 10:14 PM
  • Changing the the http post buffer size fixed it for me.

    git config http.postBuffer 524288000

    Wednesday, September 5, 2012 10:20 PM
  • @Brandon: can you try pushing from a clean clone of your repo to a new Azure Web Site as a test?

    Unfortunately, I am yet to be able to repro this issue on my end.

    Wednesday, September 5, 2012 10:21 PM
  • I tried that too amongst other things, but to no avail. Changing the http post buffer was the only thing that prevailed in my case.
    Wednesday, September 5, 2012 10:24 PM
  • So I'm trying to understand whether this is more of a client side or server side limit that your repo is hitting. If you revert the increased  postBuffer size, are you able to push to other git servers like github? Make sure you try using https and not ssh, to keep is as similar as possible.

    If that fails as well, then it may simply be a git limitation with certain repos.

    Wednesday, September 5, 2012 10:37 PM
  • Hi,


    I gave my push of "AzureBlob" another go, this time using another computer. I was able to succeed in pushing new version to WAWS, so let me go over the steps in order to be successful.

    1. Getting a clean checkout of my code base from github
    git clone git://github.com/DragonBe/azureblob.git
    cd azureblob/
    2. Adding the windows azure scm as remote git repository
    git remote add https://dragonbe@azureblob.scm.azurewebsites.net/azureblob.git
    3. Pushing to WAWS, with failure
    git push azure master

    Password:
    Counting objects: 5, done.
    Compressing objects: 100% (3/3), done.
    Writing objects: 100% (3/3), 529 bytes, done.
    Total 3 (delta 2), reused 0 (delta 0)
    remote: New deployment received.
    remote: Updating branch 'master'.
    remote: ..........................................................................................................
    remote: Preparing deployment for commit id 'ee83ecdc24'.
    remote: Preparing files for deployment.
    remote: Access to the path 'C:\DWASFiles\Sites\azureblob\VirtualDirectory0\site\wwwroot\application\configs\' is denied.
    remote: Error - Changes committed to remote repository but your website not updated.
    To https://mvdam@azureblob.scm.azurewebsites.net/azureblob.git
       6919ffc..ee83ecd  master -> master
    4. Increasing http.postBuffer size
    git config http.postBuffer 419430400
    6. Pushing with force to WAWS, successfully
    git push --force azure master

    Password for 'https://dragonbe@azureblob.scm.azurewebsites.net':
    remote: New deployment received.
    remote: Updating branch 'master'.
    remote: .....................................................................................................
    remote: Preparing deployment for commit id '34ebb81c57'.
    remote: Preparing files for deployment.
    remote: ..................................................................................................................................................................
    remote: Deployment successful.
    To https://dragonbe@azureblob.scm.azurewebsites.net/azureblob.git
       ee83ecd..34ebb81  master -> master
    I did some further investigation and came up with 2 possible causes of failure:
      • the git client and server need to accept larger code changes (hence the http.postBuffer increase)
      • when behind a firewall or proxy, the push upstream might get corrupted caused by the additional layers

    I need to investigate it more properly using a tool to analyze communications between my client and the server, and follow the suggested path by checking the HTTPS protocol in stead of the SSH/GIT protocol, but I will look at it this weekend.

    Best regards,

    Michelangelo



    Michelangelo van Dam
    Zend Certified Engineer

    Thursday, September 6, 2012 8:17 AM
  • Maybe something I need to specify here.

    When using Mac OS X and GIT versions above 1.7.9 committing to windows azure fails.

    Using a Windows client or Linux client pushing code to Windows Azure has no issues. It's a known issue reported several times with github usages.

    I'm not sure if it helps, but it's at least something to keep in mind.

    Cheers,

    Michelangelo


    Michelangelo van Dam
    Zend Certified Engineer

    Wednesday, October 31, 2012 2:13 PM
  • So you mean that when pushing similar things to github from a Mac, there is a known issue that sometimes causes it to fail as well? Could you point me to more info about this?
    Wednesday, October 31, 2012 2:47 PM
  • Well, I figured it out myself doing a clean checkout on both Mac OS X and Linux using the same git client version 1.8.0

    Where Mac seems to hang and throwing a timeout after 5 minutes, Linux had the stuff deployed to Azure.

    I need further testing myself on other Mac devices and on Windows to see it's really the case.

    My use case for this project is a clean Zend Framework 1 project with Zend added to the library directory. Using zend tool it's easy

    zf create project gitpushtest
    cd gitpushtest/library
    svn export http://framework.zend.com/svn/framework/standard/tags/release-1.12.0/library/Zend Zend
    
    cd ..
    git commit -m 'Creating a vanilla ZF1 project for testing git push to windows azure'
    
    git remote add azure https://johndoe@gitpushtest.scm.azurewebsites.net/gitpushtest.git
    git push azure master

    On my current MacBook Pro I cannot push this big of a change to Windows Azure. Existing projects with limited changes give no problems.

    Since I read about other people having these same issues, I decided to test it on Linux where it actually got through.

    These are the links I'm referring to:

    • https://github.com/projectkudu/kudu/issues/171 (you are assigned there)
    • https://bitbucket.org/site/master/issue/3578/cannot-push-fatal-the-remote-end-hung-up

    I read others but cannot find the links of those articles.

    I'm currently thinking about two things:

    1. Mac OS X has a limitation of data it can send through over https (since ssh is non-existing on Windows) and could be SSL related
    2. There's a bug in the GIT installation for Mac OS X

    Will need to do some more tests to ensure these 2 are the only logical explanations. At this point I'm not really sure it's a Windows Azure thing.

    Best regards,

    Michelangelo


    Michelangelo van Dam
    Zend Certified Engineer

    Sunday, November 4, 2012 12:46 AM
  • Thanks Michelangelo, that's interesting data. So far, our investigation has not led to something conclusive. I wonder what could be different about Mac git compared to Linux & Windows. It could be something in the networking stack rather than git itself. However, I did not see that issue when pushing the same repo to github over https from my Mac, so it's not a general Mac/https issue.
    Monday, November 5, 2012 5:28 AM