AWX not syncing Gitlab project: ORIGIN not a git repository

[batreseau]

Please confirm the following

• I agree to follow this project's code of conduct.
• I have checked the current issues for duplicates.
• I understand that AWX is open source software provided for free and that I might not receive a timely response.
• I am NOT reporting a (potential) security vulnerability. (These should be emailed to [email protected] instead.)

Bug Summary

Hello community !

I'm facing an issue configuringAWX to pull from a Gitlab instance.

More context:

• Gitlab is omnibus version installed on e dedicated virtual machine
• Gitlab project is public and i use HTTPS link (not ssh)
• AWX deployed using awx-oprator
• AWX is deployed on RKE2 with hardening/SELinux/Firewall enforced

Thank you for help you can provide :)

AWX version

24.6.1

Select the relevant components

• UI
• UI (tech preview)
• API
• Docs
• Collection
• CLI
• Other

Installation method

kubernetes

Modifications

no

Ansible version

core 2.15.12

Operating system

Oracle Linux 9

Web browser

Chrome, Edge

Steps to reproduce

• Login to AWX instance
• Go to project section and select "Add"
• Fill form using Git as SCM, Gitlab url (https://fqdn/project.git) , and Git credentials ( username + password)
• Launch sync

Expected results

Project is downloaded from gitlab and ready to be used

Actual results

Project sync failed to be performed with following error :

{ "cmd": "/usr/bin/git ls-remote origin -h refs/heads/main", "rc": 128, "stdout": "", "stderr": "fatal: 'origin' does not appear to be a git repository\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n", "msg": "fatal: 'origin' does not appear to be a git repository\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.", "invocation": { "module_args": { "dest": "/var/lib/awx/projects/_10__test", "repo": "https://mygitlab.corp/myrepo.git", "version": "main", "force": true, "track_submodules": false, "remote": "origin", "clone": true, "update": true, "verify_commit": false, "gpg_whitelist": [], "accept_hostkey": false, "accept_newhostkey": false, "bare": false, "recursive": true, "single_branch": false, "refspec": null, "reference": null, "depth": null, "key_file": null, "ssh_opts": null, "executable": null, "umask": null, "archive": null, "archive_prefix": null, "separate_git_dir": null } }, "stdout_lines": [], "stderr_lines": [ "fatal: 'origin' does not appear to be a git repository", "fatal: Could not read from remote repository.", "", "Please make sure you have the correct access rights", "and the repository exists." ], "_ansible_no_log": false, "changed": false }

Additional information

If i'm connecting in awx-tasks container and performing:

/usr/bin/git ls-remote https://mygitlab.corp/myrepo.git -h refs/heads/main

Its working well .

[batreseau]

I tried to change credentials type, even if project is public.

Switched between:

• Github Key -> token
• Gitlab Key -> token
• SCM Key -> no username, token as password

Same error.

[batreseau]

Full job log:

`ansible-playbook [core 2.15.12]
config file = None
configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
ansible python module location = /usr/local/lib/python3.9/site-packages/ansible
ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
executable location = /usr/local/bin/ansible-playbook
python version = 3.9.19 (main, Jun 11 2024, 00:00:00) [GCC 11.4.1 20231218 (Red Hat 11.4.1-3)] (/usr/bin/python3)
jinja version = 3.1.4
libyaml = True
No config file found; using defaults
host_list declined parsing /tmp/awx_49_lv5x6r7l/inventory/hosts as it did not pass its verify_file() method
script declined parsing /tmp/awx_49_lv5x6r7l/inventory/hosts as it did not pass its verify_file() method
auto declined parsing /tmp/awx_49_lv5x6r7l/inventory/hosts as it did not pass its verify_file() method
Parsed /tmp/awx_49_lv5x6r7l/inventory/hosts inventory source with ini plugin
Skipping callback 'awx_display', as we already have a stdout callback.
Skipping callback 'default', as we already have a stdout callback.
Skipping callback 'minimal', as we already have a stdout callback.
Skipping callback 'oneline', as we already have a stdout callback.

PLAYBOOK: project_update.yml ***************************************************
3 plays in project_update.yml

PLAY [Update source tree if necessary] *****************************************

TASK [Delete project directory before update] **********************************
task path: /tmp/awx_49_lv5x6r7l/project/project_update.yml:28
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: root
<127.0.0.1> EXEC /bin/sh -c 'echo ~root && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "echo /root/.ansible/tmp"&& mkdir "echo /root/.ansible/tmp/ansible-tmp-1732922312.7720482-3019-87937980802594" && echo ansible-tmp-1732922312.7720482-3019-87937980802594="echo /root/.ansible/tmp/ansible-tmp-1732922312.7720482-3019-87937980802594" ) && sleep 0'
Using module file /usr/local/lib/python3.9/site-packages/ansible/modules/command.py
<127.0.0.1> PUT /root/.ansible/tmp/ansible-local-3009z9xun9ux/tmppn6jev2h TO /root/.ansible/tmp/ansible-tmp-1732922312.7720482-3019-87937980802594/AnsiballZ_command.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1732922312.7720482-3019-87937980802594/ /root/.ansible/tmp/ansible-tmp-1732922312.7720482-3019-87937980802594/AnsiballZ_command.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python3 /root/.ansible/tmp/ansible-tmp-1732922312.7720482-3019-87937980802594/AnsiballZ_command.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1732922312.7720482-3019-87937980802594/ > /dev/null 2>&1 && sleep 0'
changed: [localhost] => {
"changed": true,
"cmd": "set -o pipefail && find . -delete -print | tail -2",
"delta": "0:00:00.022055",
"end": "2024-11-29 23:18:33.086135",
"invocation": {
"module_args": {
"_raw_params": "set -o pipefail && find . -delete -print | tail -2",
"_uses_shell": true,
"argv": null,
"chdir": "/var/lib/awx/projects/_10__test",
"creates": null,
"executable": null,
"removes": null,
"stdin": null,
"stdin_add_newline": true,
"strip_empty_ends": true
}
},
"msg": "",
"rc": 0,
"start": "2024-11-29 23:18:33.064080",
"stderr": "",
"stderr_lines": [],
"stdout": "./inventory\n.",
"stdout_lines": [
"./inventory",
"."
]
}

TASK [Update project using git] ************************************************
task path: /tmp/awx_49_lv5x6r7l/project/project_update.yml:41
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: root
<127.0.0.1> EXEC /bin/sh -c 'echo ~root && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "echo /root/.ansible/tmp"&& mkdir "echo /root/.ansible/tmp/ansible-tmp-1732922313.1549914-3038-69230927487215" && echo ansible-tmp-1732922313.1549914-3038-69230927487215="echo /root/.ansible/tmp/ansible-tmp-1732922313.1549914-3038-69230927487215" ) && sleep 0'
Using module file /usr/local/lib/python3.9/site-packages/ansible/modules/git.py
<127.0.0.1> PUT /root/.ansible/tmp/ansible-local-3009z9xun9ux/tmp3on3cikn TO /root/.ansible/tmp/ansible-tmp-1732922313.1549914-3038-69230927487215/AnsiballZ_git.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x /root/.ansible/tmp/ansible-tmp-1732922313.1549914-3038-69230927487215/ /root/.ansible/tmp/ansible-tmp-1732922313.1549914-3038-69230927487215/AnsiballZ_git.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c '/usr/bin/python3 /root/.ansible/tmp/ansible-tmp-1732922313.1549914-3038-69230927487215/AnsiballZ_git.py && sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'rm -f -r /root/.ansible/tmp/ansible-tmp-1732922313.1549914-3038-69230927487215/ > /dev/null 2>&1 && sleep 0'
fatal: [localhost]: FAILED! => {
"changed": false,
"cmd": "/usr/bin/git ls-remote origin -h refs/heads/main",
"invocation": {
"module_args": {
"accept_hostkey": false,
"accept_newhostkey": false,
"archive": null,
"archive_prefix": null,
"bare": false,
"clone": true,
"depth": null,
"dest": "/var/lib/awx/projects/_10__test",
"executable": null,
"force": true,
"gpg_whitelist": [],
"key_file": null,
"recursive": true,
"reference": null,
"refspec": null,
"remote": "origin",
"repo": "https://gitlab.corp/myrepo.git",
"separate_git_dir": null,
"single_branch": false,
"ssh_opts": null,
"track_submodules": false,
"umask": null,
"update": true,
"verify_commit": false,
"version": "main"
}
},
"msg": "fatal: 'origin' does not appear to be a git repository\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.",
"rc": 128,
"stderr": "fatal: 'origin' does not appear to be a git repository\nfatal: Could not read from remote repository.\n\nPlease make sure you have the correct access rights\nand the repository exists.\n",
"stderr_lines": [
"fatal: 'origin' does not appear to be a git repository",
"fatal: Could not read from remote repository.",
"",
"Please make sure you have the correct access rights",
"and the repository exists."
],
"stdout": "",
"stdout_lines": []
}

PLAY RECAP *********************************************************************
localhost : ok=1 changed=1 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
`

[thedoubl3j]

@batreseau for extra testing, are you able to point this at a public/private github repo to test? This error seems pretty generic of not being able to find the repo. And to double check, since you redacted the URLs, can you confirm that the command that you ran in the task container is the same that was run by the playbook?

We were unable to reproduce on devel.

[batreseau]

@thedoubl3j thank you for you answer ! Im redeploying with 24.5.0 and will test again .

[batreseau]

So, what an interesting afternoon ... But i finally made it worked :)

To make it quick : root cause was permissions issue on projects folder . Deployed on kubernetes as RWX pvc, it's using NFS protocol to provision volume. On NFS backend, Root Squash Security was enabled with a rule that say: if storage domain =! client domain, rewrite root perms to nobody.

After patching, going back to 24.6.1, permissions were as expected in container and clone worked.

Thank you @thedoubl3j for time spent on my issue.

Best regard

[thedoubl3j]

glad you got things working @batreseau