[SOLVED in v2.47.0(2)] v2.47.0 Git hangs on fetch

[orgads]

UPDATE: THIS ISSUE WAS FIXED

please upgrade to v2.47.0(2) or later, it fixes this issue.

• I was not able to find an open or closed issue matching what I'm seeing

Setup

• Which version of Git for Windows are you using? Is it 32-bit or 64-bit?

$ git --version --build-options

git version 2.47.0.windows.1
cpu: x86_64
built from commit: 8e380191abeda7a995e87abf4acc9e1702fdc698
sizeof-long: 4
sizeof-size_t: 8
shell-path: F:/git-sdk-64/usr/bin/sh
feature: fsmonitor--daemon
libcurl: 8.10.1
OpenSSL: OpenSSL 3.2.3 3 Sep 2024
zlib: 1.3.1

• Which version of Windows are you running? Vista, 7, 8, 10? Is it 32-bit or 64-bit?

$ cmd.exe /c ver

Microsoft Windows [Version 10.0.19045.4355]

• What options did you set as part of the installation? Or did you choose the
  defaults?

# One of the following:
> type "C:\Program Files\Git\etc\install-options.txt"
> type "C:\Program Files (x86)\Git\etc\install-options.txt"
> type "%USERPROFILE%\AppData\Local\Programs\Git\etc\install-options.txt"
> type "$env:USERPROFILE\AppData\Local\Programs\Git\etc\install-options.txt"
$ cat /etc/install-options.txt

Editor Option: VIM
Custom Editor Path:
Default Branch Option:
Path Option: Cmd
SSH Option: OpenSSH
Tortoise Option: false
CURL Option: WinSSL
CRLF Option: CRLFAlways
Bash Terminal Option: MinTTY
Git Pull Behavior Option: Rebase
Use Credential Manager: Enabled
Performance Tweaks FSCache: Enabled
Enable Symlinks: Enabled
Enable FSMonitor: Disabled

Details

• Which terminal/shell are you running Git from? e.g Bash/CMD/PowerShell/other

Git Bash

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

git fetch

• What did you expect to occur after running these commands?

It should be done successfully after a while.

• What actually happened instead?

It sometimes hangs. I see the following spawned process, which doesn't use any CPU, and doesn't terminate:

git index-pack --stdin -v --fix-thin "--keep=fetch-pack 4272 on my-pc" --pack_header=2,291

The output is:

remote: Counting objects: 195, done
remote: Finding sources: 100% (292/292)
<nothing happens>
<when I press Ctrl-C>
fetch-pack: unexpected disconnect while reading sideband packet

[dscho]

• What commands did you run to trigger this issue? If you can provide a
  Minimal, Complete, and Verifiable example
  this will help us understand the issue.

git fetch

To make this example a little more complete, could you describe from where you are fetching? Is it github.com, a self-hosted git daemon, from a network share? Is it a big fetch, i.e. have already dozens of megabytes been transferred, or is it a small fetch? Did you try to disable sideband?

[orgads]

It's a hosted gerrit 3.8 server. The fetch is small, took a few seconds after I downgraded to 2.46.

I can try to disable sideband tomorrow.

[jfcherng]

Latest update: Now it clones okay... without any local env modification... Okay again it just happened.

---

I tried to clone my https://github.com/jfcherng/copilot-node-server .

It seems to alway hang if I clone [email protected]:jfcherng/copilot-node-server.git. But it's okay if I clone https://github.com/jfcherng/copilot-node-server.git.

---

Update: It still hangs with GIT_CURL_VERBOSE=1 GIT_TRACE=1 git -c sendpack.sideband=false clone [email protected]:jfcherng/copilot-node-server.git

[jfcherng@HOME Desktop]$ GIT_CURL_VERBOSE=1 GIT_TRACE=1 git -c sendpack.sideband=false clone [email protected]:jfcherng/copilot-node-server.git
10:14:30.318887 exec-cmd.c:266          trace: resolved executable dir: C:/Program Files/Git/mingw64/bin
10:14:30.329889 git.c:479               trace: built-in: git clone [email protected]:jfcherng/copilot-node-server.git
Cloning into 'copilot-node-server'...
10:14:30.350892 run-command.c:667       trace: run_command: unset GIT_CONFIG_PARAMETERS GIT_DIR; GIT_PROTOCOL=version=2 ssh -o SendEnv=GIT_PROTOCOL [email protected] 'git-upload-pack '\''jfcherng/copilot-node-server.git'\'''
10:14:30.350892 run-command.c:928       trace: start_command: ssh -o SendEnv=GIT_PROTOCOL [email protected] 'git-upload-pack '\''jfcherng/copilot-node-server.git'\'''
remote: Enumerating objects: 542, done.
remote: Counting objects: 100% (100/100), done.
10:14:33.230184 run-command.c:667       trace: run_command: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 32844 on 3bc-f19' --check-self-contained-and-connected
10:14:33.230184 run-command.c:928       trace: start_command: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 32844 on 3bc-f19' --check-self-contained-and-connected
10:14:33.298183 exec-cmd.c:266          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
10:14:33.311182 git.c:479               trace: built-in: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 32844 on 3bc-f19' --check-self-contained-and-connected
remote: Compressing objects: 100% (65/65), done.
Receiving objects:   1% (6/542)

It just hangs on Receiving objects: 1% (6/542). It is always 1% (6/542) everytime I retried. Eventually it failed as the following.

[jfcherng@HOME Desktop]$ GIT_CURL_VERBOSE=1 GIT_TRACE=1 git -c sendpack.sideband=false clone [email protected]:jfcherng/copilot-node-server.git
10:16:32.600368 exec-cmd.c:266          trace: resolved executable dir: C:/Program Files/Git/mingw64/bin
10:16:32.614373 git.c:479               trace: built-in: git clone [email protected]:jfcherng/copilot-node-server.git
Cloning into 'copilot-node-server'...
10:16:32.653368 run-command.c:667       trace: run_command: unset GIT_CONFIG_PARAMETERS GIT_DIR; GIT_PROTOCOL=version=2 ssh -o SendEnv=GIT_PROTOCOL [email protected] 'git-upload-pack '\''jfcherng/copilot-node-server.git'\'''
10:16:32.653368 run-command.c:928       trace: start_command: ssh -o SendEnv=GIT_PROTOCOL [email protected] 'git-upload-pack '\''jfcherng/copilot-node-server.git'\'''
remote: Enumerating objects: 542, done.
remote: Counting objects: 100% (100/100), done.
10:16:35.393876 run-command.c:667       trace: run_command: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 20220 on 3bc-f19' --check-self-contained-and-connected
10:16:35.393876 run-command.c:928       trace: start_command: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 20220 on 3bc-f19' --check-self-contained-and-connected
10:16:35.455879 exec-cmd.c:266          trace: resolved executable dir: C:/Program Files/Git/mingw64/libexec/git-core
10:16:35.469968 git.c:479               trace: built-in: git index-pack --stdin -v --fix-thin '--keep=fetch-pack 20220 on 3bc-f19' --check-self-contained-and-connected
remote: Compressing objects: 100% (65/65), done.
fetch-pack: unexpected disconnect while reading sideband packet
fatal: fetch-pack: invalid index-pack output

[orgads]

Ah right, my case was ssh too.

[EemilAhonen]

We are having this same problem with SSH on multiple machines that upgraded to the latest 2.47.0. All clones over 1mb from Gitlab are failing, the Git window just freezes at random points.
Uninstalling 2.47.0 and reinstalling 2.44.0 solved the issue.

[dscho]

Does it work if you copy the usr\bin\ssh.exe over from v2.46.2?

[jfcherng]

Does it work if you copy the usr\bin\ssh.exe over from v2.46.2?

In my case, no.

[dscho]

Does it work if you copy the usr\bin\ssh.exe over from v2.46.2?

In my case, no.

How about copying over usr\bin\msys-2.0.dll?

[jfcherng]

How about copying over usr\bin\msys-2.0.dll?

Yes, that's the culprit. I tried in a fresh portable build. No need to replace ssh.exe.

[dscho]

I feared as much. There is a rather big jump in the MSYS2 runtime upgrade between Git for Windows v2.46.2 and v2.47.0. And MSYS2 runtime issues are notoriously hard to fix.

You wouldn't happen to have a good reproducer that other people might be able to use, would you? I am afraid that git fetch as an MCVE does not pass muster.

[elindoorn]

A git pull from a azure repo consistently hung for me today until i reverted to 2.46.2

git pull
remote: Azure Repos
remote: Found 308 objects to send. (757 ms)

the trace suggests it can collect branches/tags and such before it hangs

`> git pull
13:28:53.108594 exec-cmd.c:266 trace: resolved executable dir: C:/s/git/mingw64/bin
13:28:53.118594 git.c:479 trace: built-in: git pull
13:28:53.157446 run-command.c:667 trace: run_command: git merge-base --fork-point refs/remotes/origin/redacted redacted
13:28:53.157446 run-command.c:928 trace: start_command: git merge-base --fork-point refs/remotes/origin/redacted redacted
13:28:53.197744 run-command.c:667 trace: run_command: git fetch --update-head-ok
13:28:53.197744 run-command.c:928 trace: start_command: git fetch --update-head-ok
13:28:53.209496 exec-cmd.c:266 trace: resolved executable dir: C:/s/git/mingw64/libexec/git-core
13:28:53.219721 git.c:479 trace: built-in: git fetch --update-head-ok
13:28:53.223755 run-command.c:667 trace: run_command: unset GIT_PREFIX; GIT_PROTOCOL=version=2 ssh -o SendEnv=GIT_PROTOCOL [email protected] 'git-upload-pack '''v3/redacted''''
13:28:53.224873 run-command.c:928 trace: start_command: ssh -o SendEnv=GIT_PROTOCOL [email protected] 'git-upload-pack '''v3/redacted''''
13:28:53.987805 pkt-line.c:86 packet: fetch< 191d8833b443043e99cd2d3184f4b66c154d3847 HEAD\0 multi_ack thin-pack side-band side-band-64k no-progress multi_ack_detailed no-done shallow allow-tip-sha1-in-want filter symref=HEAD:refs/heads/develop
...
13:28:54.115251 pkt-line.c:86 packet: fetch< NAK
13:28:54.136540 pkt-line.c:86 packet: fetch< ACK eb10cf30356d9de23600c64368d94d664db97888
13:28:54.138227 pkt-line.c:86 packet: sideband< \2Azure Repos
remote: Azure Repos
13:28:54.139224 pkt-line.c:86 packet: sideband< \2\15Found 303 objects to send. (12 ms)
remote: Found 303 objects to send. (12 ms)

nothing after this`

[ryanmkurtz]

I'm having the same issue too with the move from 2.46.0 to 2.47.0. Both clone and fetch now hang:

$ git fetch
remote: Enumerating objects: 8510, done.
remote: Counting objects: 100% (5831/5831), done.
remote: Compressing objects: 100% (782/782), done.
<hang>

I am working with my organization's GitLab repo over ssh.

[derrickstolee]

I was able to reproduce the failing clones and have a workaround:

git -c core.sshCommand="'C:\Windows\System32\OpenSSH\ssh.exe'" clone ...

or

git config --global core.sshCommand "'C:\Windows\System32\OpenSSH\ssh.exe'"
git clone ...

This configures Git to use the SSH agent built into Windows instead of the OpenSSL-based one provided by MSYS2.

[b12k]

Same here.
Hangs on clone with 2.47.0

@derrickstolee Solution helped 👆

[Nbc66]

Can Confirm git clone & git push hang when trying to call em this only happens on version 2.47.0 the other version like 2.46.2 work just fine

this might be related to the MSYS2 upgrade on the latest version

[goostleek]

I can confirm, I thought it was specific to my repo only, but using an example provided at https://github-debug.com/

GIT_TRACE=1 GIT_TRANSFER_TRACE=1 GIT_CURL_VERBOSE=1 git clone [email protected]:github/debug-repo /tmp/debug-repo-ssh

causes the SSH connection to freeze

Tip

The possible workaround is to temporarily switch from ssh:// to https:// when feasible

git remote set-url origin 'https://<repo-url>'

[mm65de]

After installing Git-2.47.0-64-bit.exe the git commands clone and fetch also hang on my machine.
My previous installation without that problem was based on this setup file: Git-2.46.2-64-bit.exe

Here is an example of this problem:

2024-10-10 11:24:35  c:\Tmp\git_test_folder> git clone [email protected]:git-for-windows/git.git
Cloning into 'git'...
remote: Enumerating objects: 622752, done.
remote: Counting objects: 100% (7374/7374), done.
remote: Compressing objects: 100% (2904/2904), done.

Now nothing happens anymore.
If I press Ctrl-C then I get this:

fetch-pack: unexpected disconnect while reading sideband packet
fatal: fetch-pack: invalid index-pack output

2024-10-10 11:26:13  c:\Tmp\git_test_folder>

Here comes the System Info section of the text file generated by the command git bugreport:

[System Info]
git version:
git version 2.47.0.windows.1
cpu: x86_64
built from commit: d53e4648cb65eb75dd8d8a093d17400a18a9a15d
sizeof-long: 4
sizeof-size_t: 8
shell-path: D:/git-sdk-64-build-installers/usr/bin/sh
feature: fsmonitor--daemon
libcurl: 8.10.1
OpenSSL: OpenSSL 3.2.3 3 Sep 2024
zlib: 1.3.1
uname: Windows 10.0 19045 
compiler info: gnuc: 14.2
libc info: no libc information available
$SHELL (typically, interactive shell): <unset>

After executing the setup file Git-2.46.2-64-bit.exe and accepting the downgrade warning the command git clone worked well again:

2024-10-10 11:51:53  c:\Tmp\git_test_folder> git clone [email protected]:git-for-windows/git.git
Cloning into 'git'...
remote: Enumerating objects: 622752, done.
remote: Counting objects: 100% (7379/7379), done.
remote: Compressing objects: 100% (2905/2905), done.
remote: Total 622752 (delta 6166), reused 4919 (delta 4472), pack-reused 615373 (from 1)
Receiving objects: 100% (622752/622752), 310.19 MiB | 5.87 MiB/s, done.
Resolving deltas: 100% (450376/450376), done.
Updating files: 100% (4589/4589), done.

2024-10-10 11:53:16  c:\Tmp\git_test_folder>

[Nbc66]

After installing Git-2.47.0-64-bit.exe the git commands clone and fetch also hang on my machine. My previous installation without that problem was based on this setup file: Git-2.46.2-64-bit.exe

Here is an example of this problem:

2024-10-10 11:24:35  c:\Tmp\git_test_folder> git clone [email protected]:git-for-windows/git.git
Cloning into 'git'...
remote: Enumerating objects: 622752, done.
remote: Counting objects: 100% (7374/7374), done.
remote: Compressing objects: 100% (2904/2904), done.

Now nothing happens anymore. If I press Ctrl-C then I get this:

fetch-pack: unexpected disconnect while reading sideband packet
fatal: fetch-pack: invalid index-pack output

2024-10-10 11:26:13  c:\Tmp\git_test_folder>

Here comes the System Info section of the text file generated by the command git bugreport:

[System Info]
git version:
git version 2.47.0.windows.1
cpu: x86_64
built from commit: d53e4648cb65eb75dd8d8a093d17400a18a9a15d
sizeof-long: 4
sizeof-size_t: 8
shell-path: D:/git-sdk-64-build-installers/usr/bin/sh
feature: fsmonitor--daemon
libcurl: 8.10.1
OpenSSL: OpenSSL 3.2.3 3 Sep 2024
zlib: 1.3.1
uname: Windows 10.0 19045 
compiler info: gnuc: 14.2
libc info: no libc information available
$SHELL (typically, interactive shell): <unset>

After executing the setup file Git-2.46.2-64-bit.exe and accepting the downgrade warning the command git clone worked well again:

2024-10-10 11:51:53  c:\Tmp\git_test_folder> git clone [email protected]:git-for-windows/git.git
Cloning into 'git'...
remote: Enumerating objects: 622752, done.
remote: Counting objects: 100% (7379/7379), done.
remote: Compressing objects: 100% (2905/2905), done.
remote: Total 622752 (delta 6166), reused 4919 (delta 4472), pack-reused 615373 (from 1)
Receiving objects: 100% (622752/622752), 310.19 MiB | 5.87 MiB/s, done.
Resolving deltas: 100% (450376/450376), done.
Updating files: 100% (4589/4589), done.

2024-10-10 11:53:16  c:\Tmp\git_test_folder>

Its an issue with the latest MYSYS2 update they did

[mm65de]

If it helps, here are more configuration details of my machine:

2024-10-10 12:10:46  c:\Tmp\git_test_folder> git config --list --system
diff.astextplain.textconv=astextplain
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
http.sslbackend=openssl
http.sslcainfo=C:/Program Files/Git/mingw64/etc/ssl/certs/ca-bundle.crt
core.autocrlf=false
core.fscache=true
core.symlinks=false
core.editor="C:\\Program Files\\Notepad++\\notepad++.exe" -multiInst -notabbar -nosession -noPlugin
pull.rebase=true
credential.helper=manager
credential.https://dev.azure.com.usehttppath=true
init.defaultbranch=master

2024-10-10 12:11:18  c:\Tmp\git_test_folder>

If I use the HTTPS URL instead of the SSH URL to clone the repository, then the command git clone even works with the new version 2.47.0.windows.1.

[david-siegert]

After update to 2.47 I get the same issue when trying to use git clone using ssh. It hangs indefinatelly and i have to terminate the command using ctrl+c resulting in "unexpected disconnect while reading sideband packet". After rolling back to 2.46 it works fine.

[jeremyVignelles]

You wouldn't happen to have a good reproducer that other people might be able to use, would you? I am afraid that git fetch as an MCVE does not pass muster.

For reference, I came across this issue from mutagen-io/mutagen#511 :
Mutagen detects the ssh from git-for-windows and tries to use it to connect to the remote server. The update in git-for-windows broke the mutagen detection as well, and we had to revert msys-2.0.dll.

I don't know what kind of reproduction project you need, but our use case is just a virtualbox VM and trying to sync something with mutagen mutagen sync create...

[dscho]

I can reproduce the issue, and am bisecting now. Experience suggests that it might take a couple of days to get to the bottom of this, so please have patience.

[dscho]

Okay, I have a fix in hand: It seems to be a regression in Cygwin v3.5.3, and while I am not completely happy about the extent to which I understand what is going wrong, it seems safe enough to revert to the working behavior (even if it should not completely reflect how Linux would behave in the same situation).

Y'all, could I ask you to download the install.zip artifact produced by the PR build, and extract its usr/bin/msys-2.0.dll over your existing one, then verify that it fixes the issue not only for me, but also for all of you gentle people?

[dscho]

@orgads I hope that the updated title is sufficient.

[rotu]

What's the right way to update this in a github-hosted runner?

The following workflow step fails with "Error: Process completed with exit code 2.". I experience similar problems with shell:pwsh and shell:powershell:

- name: Update git for windows
    if: runner.os == 'Windows'
    run: git update-git-for-windows --yes
    shell: cmd

2024-10-24T17:31:41.3652735Z �[36;1mgit update-git-for-windows --yes�[0m
2024-10-24T17:31:41.3677331Z shell: C:\Windows\system32\cmd.EXE /D /E:ON /V:OFF /S /C "CALL "{0}""
2024-10-24T17:31:41.3678050Z ##[endgroup]
2024-10-24T17:31:43.6864522Z Git for Windows 2.47.0.windows.1 (64-bit)
2024-10-24T17:31:44.0033217Z Update 2.47.0.windows.2 is available
2024-10-24T17:31:44.5000031Z Downloading Git-2.47.0.2-64-bit.exe
...
2024-10-24T17:31:44.9470534Z ######################################################################## 100.0%
2024-10-24T17:31:51.5451678Z ##[error]Process completed with exit code 2.

[rotu]

I had success with this:

 - name: Install git for windows
   if: runner.os == 'Windows'
   shell: pwsh
   run: |
     Invoke-WebRequest -Uri "https://github.com/git-for-windows/git/releases/download/v2.47.0.windows.2/Git-2.47.0.2-64-bit.exe" -OutFile "${{runner.temp}}/git-for-windows.exe"
     Start-Process -Filepath "${{runner.temp}}/git-for-windows.exe" -ArgumentList  @("/VERYSILENT","/NORESTART","/NOCANCEL","/SP-","/CLOSEAPPLICATIONS","/RESTARTAPPLICATIONS","/o:PathOption=CmdTools","/o:BashTerminalOption=ConHost","/o:EnableSymlinks=Enabled","/COMPONENTS=gitlfs") -NoNewWindow -Wait

I still have no idea why git update-git-for-windows failed.

[mikeKuester]

I can confirm, that this fixes also my problem with git push and active Git Lfs: git-lfs/git-lfs#5893

[stefan-muc]

For the record: I plan on releasing Git for Windows v2.47.0(2) tomorrow, with the fix for this issue.

Thanks for releasing this as version! I wouldn't have found this bug report as easily otherwise. My colleague and me stumbled on this bug but first suspected changed firewall policies as root cause. During debugging I upgraded git, and the bug was gone.

Regarding your question why people rather do a downgrade: I use a package manager (chocolatey) for software like git. Doing a downgrade is an one liner. Using a snapshot from somewhere and clicking through the installer is more work.