Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

exp push: unexpected http resp 403 with gitlab CI from ec2 instance #9981

Closed
BradyJ27 opened this issue Sep 27, 2023 · 7 comments
Closed

exp push: unexpected http resp 403 with gitlab CI from ec2 instance #9981

BradyJ27 opened this issue Sep 27, 2023 · 7 comments
Labels
awaiting response we are waiting for your reply, please respond! :) git Related to git and git backends

Comments

@BradyJ27
Copy link
Contributor

Bug Report

Description

I have an EC2 instance which runs a cml runner. The cml runner picks up a gitlab ci job which pulls data, runs experiments, and pushes experiments. The data is being pulled properly and I can run dvc push with no errors. I have verified that aws cli commands work just fine on the EC2 instance, but I am unable to exp push. When I run dvc exp push origin --run-cache -vv I am getting the following output:

$ dvc exp push origin --run-cache -vv
2023-09-27 01:46:09,530 DEBUG: v3.23.0 (pip), CPython 3.8.10 on Linux-4.14.322-244.539.amzn2.x86_64-x86_64-with-glibc2.29
2023-09-27 01:46:09,530 DEBUG: command: /home/runner/HZtmPsdD/0/group/subgroup/test-yolo-dvc/.venv/bin/dvc exp push origin --run-cache -vv
2023-09-27 01:46:09,530 TRACE: Namespace(all_commits=False, cd='.', cmd='push', cprofile=False, cprofile_dump=None, dvc_remote=None, experiment=[], force=False, func=<class 'dvc.commands.experiments.push.CmdExperimentsPush'>, git_remote='origin', instrument=False, instrument_open=False, jobs=None, num=1, parser=DvcParser(prog='dvc', usage=None, description='Data Version Control', formatter_class=<class 'argparse.RawTextHelpFormatter'>, conflict_handler='error', add_help=False), pdb=False, push_cache=True, quiet=0, rev=None, run_cache=True, show_stack=False, verbose=2, viztracer=False, viztracer_async=False, viztracer_depth=None, yappi=False, yappi_separate_threads=False)
2023-09-27 01:46:09,875 DEBUG: git push experiment ['refs/exps/de/3a9c73a2bb1e1496425ba61e5dd9fefe6e6e81/test-m-model-exp:refs/exps/de/3a9c73a2bb1e1496425ba61e5dd9fefe6e6e81/test-m-model-exp'] -> 'origin'
2023-09-27 01:46:09,988 ERROR: unexpected error - unexpected http resp 403 for https://gitlab-ci-token:[MASKED]@gitlab.com/group/subgroup/test-yolo-dvc.git/info/refs?service=git-receive-pack
Traceback (most recent call last):
  File "/home/runner/HZtmPsdD/0/group/subgroup/test-yolo-dvc/.venv/lib/python3.8/site-packages/dvc/cli/__init__.py", line 209, in main
    ret = cmd.do_run()
  File "/home/runner/HZtmPsdD/0/group/subgroup/test-yolo-dvc/.venv/lib/python3.8/site-packages/dvc/cli/command.py", line 26, in do_run
    return self.run()
  File "/home/runner/HZtmPsdD/0/group/subgroup/test-yolo-dvc/.venv/lib/python3.8/site-packages/dvc/commands/experiments/push.py", line 55, in run
    result = self.repo.experiments.push(
  File "/home/runner/HZtmPsdD/0/group/subgroup/test-yolo-dvc/.venv/lib/python3.8/site-packages/dvc/repo/experiments/__init__.py", line 381, in push
    return push(self.repo, *args, **kwargs)
  File "/home/runner/HZtmPsdD/0/group/subgroup/test-yolo-dvc/.venv/lib/python3.8/site-packages/dvc/repo/__init__.py", line 61, in wrapper
    return f(repo, *args, **kwargs)
  File "/home/runner/HZtmPsdD/0/group/subgroup/test-yolo-dvc/.venv/lib/python3.8/site-packages/dvc/repo/scm_context.py", line 151, in run
    return method(repo, *args, **kw)
  File "/home/runner/HZtmPsdD/0/group/subgroup/test-yolo-dvc/.venv/lib/python3.8/site-packages/dvc/repo/experiments/push.py", line 120, in push
    push_result = _push(repo, git_remote, exp_ref_set, force)
  File "/home/runner/HZtmPsdD/0/group/subgroup/test-yolo-dvc/.venv/lib/python3.8/site-packages/dvc/repo/experiments/push.py", line 162, in _push
    results: Mapping[str, SyncStatus] = repo.scm.push_refspecs(
  File "/home/runner/HZtmPsdD/0/group/subgroup/test-yolo-dvc/.venv/lib/python3.8/site-packages/scmrepo/git/__init__.py", line 292, in _backend_func
    result = func(*args, **kwargs)
  File "/home/runner/HZtmPsdD/0/group/subgroup/test-yolo-dvc/.venv/lib/python3.8/site-packages/scmrepo/git/backend/dulwich/__init__.py", line 630, in push_refspecs
    result = client.send_pack(
  File "/home/runner/HZtmPsdD/0/group/subgroup/test-yolo-dvc/.venv/lib/python3.8/site-packages/dulwich/client.py", line 2015, in send_pack
    old_refs, server_capabilities, url = self._discover_references(
  File "/home/runner/HZtmPsdD/0/group/subgroup/test-yolo-dvc/.venv/lib/python3.8/site-packages/dulwich/client.py", line 1940, in _discover_references
    resp, read = self._http_request(url, headers)
  File "/home/runner/HZtmPsdD/0/group/subgroup/test-yolo-dvc/.venv/lib/python3.8/site-packages/scmrepo/git/backend/dulwich/client.py", line 49, in _http_request
    result = super()._http_request(
  File "/home/runner/HZtmPsdD/0/group/subgroup/test-yolo-dvc/.venv/lib/python3.8/site-packages/dulwich/client.py", line 2222, in _http_request
    raise GitProtocolError(
dulwich.errors.GitProtocolError: unexpected http resp 403 for https://gitlab-ci-token:[MASKED]@gitlab.com/group/subgroup/test-yolo-dvc.git/info/refs?service=git-receive-pack
2023-09-27 01:46:10,029 DEBUG: link type reflink is not available ([Errno 95] no more link types left to try out)
2023-09-27 01:46:10,029 DEBUG: Removing '/home/runner/HZtmPsdD/0/group/subgroup/.5wW5NPytAzoaq7B7g6GNNy.tmp'
2023-09-27 01:46:10,030 DEBUG: Removing '/home/runner/HZtmPsdD/0/group/subgroup/.5wW5NPytAzoaq7B7g6GNNy.tmp'
2023-09-27 01:46:10,030 DEBUG: Removing '/home/runner/HZtmPsdD/0/group/subgroup/.5wW5NPytAzoaq7B7g6GNNy.tmp'
2023-09-27 01:46:10,030 DEBUG: Removing '/home/runner/HZtmPsdD/0/group/subgroup/test-yolo-dvc/.dvc/cache/files/md5/.ViXmrNyXbf3quG46EcanGS.tmp'
2023-09-27 01:46:10,038 DEBUG: Version info for developers:
DVC version: 3.23.0 (pip)
-------------------------
Platform: Python 3.8.10 on Linux-4.14.322-244.539.amzn2.x86_64-x86_64-with-glibc2.29
Subprojects:
	dvc_data = 2.16.4
	dvc_objects = 1.0.1
	dvc_render = 0.6.0
	dvc_task = 0.3.0
	scmrepo = 1.3.1
Supports:
	http (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
	https (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
	s3 (s3fs = 2023.6.0, boto3 = 1.28.17)
Config:
	Global: /root/.config/dvc
	System: /etc/xdg/dvc
Cache types: hardlink, symlink
Cache directory: overlay on overlay
Caches: local
Remotes: s3
Workspace directory: overlay on overlay
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/47e73b34787c67e05beba0a9fb162219
Having any troubles? Hit us up at https://dvc.org/support, we are always happy to help!
2023-09-27 01:46:10,040 DEBUG: Analytics is enabled.
2023-09-27 01:46:10,083 DEBUG: Trying to spawn '['daemon', '-q', 'analytics', '/tmp/tmp7gawsdhf']'
2023-09-27 01:46:10,085 DEBUG: Spawned '['daemon', '-q', 'analytics', '/tmp/tmp7gawsdhf']'

Reproduce

Expected

Environment information

Output of dvc doctor:

$ dvc doctor
DVC version: 3.23.0 (pip)
-------------------------
Platform: Python 3.8.10 on Linux-4.14.322-244.539.amzn2.x86_64-x86_64-with-glibc2.29
Subprojects:
	dvc_data = 2.16.4
	dvc_objects = 1.0.1
	dvc_render = 0.6.0
	dvc_task = 0.3.0
	scmrepo = 1.3.1
Supports:
	http (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
	https (aiohttp = 3.8.5, aiohttp-retry = 2.8.3),
	s3 (s3fs = 2023.6.0, boto3 = 1.28.17)
Config:
	Global: /root/.config/dvc
	System: /etc/xdg/dvc
Cache types: hardlink, symlink
Cache directory: overlay on overlay
Caches: local
Remotes: s3
Workspace directory: overlay on overlay
Repo: dvc, git
Repo.site_cache_dir: /var/tmp/dvc/repo/47e73b34787c67e05beba0a9fb162219

Additional Information (if any):

It's possible that this is a version incompatibility issue, but I was able to exp push before I was using dvclive with dvc 3.12.0. I can also run everything locally with python 3.10, but upgrading my cloud python version is kind of a pain, it would be easier to downgrade dvc and dependencies. I tried with dvc 3.22 and 3.20, but there was the same issue.
@pmrowla
Copy link
Contributor

pmrowla commented Sep 27, 2023

Just to clarify, the issue here is pushing the experiment commit to the gitlab git server, not with pushing DVC cache to S3

@pmrowla pmrowla added the git Related to git and git backends label Sep 27, 2023
@pmrowla pmrowla changed the title exp push: unexpected http resp 403 with s3 from ec2 instance exp push: unexpected http resp 403 with gitlab CI from ec2 instance Sep 27, 2023
@dberenbaum
Copy link
Collaborator

Does the token you use have the correct scope to push to the repo? See https://stackoverflow.com/a/73394648/3127500 for example.

@dberenbaum dberenbaum added the awaiting response we are waiting for your reply, please respond! :) label Sep 27, 2023
@BradyJ27
Copy link
Contributor Author

BradyJ27 commented Sep 27, 2023 •
edited
Loading

Does the token you use have the correct scope to push to the repo? See https://stackoverflow.com/a/73394648/3127500 for example.

The token does have write repository permissions. I don't explicitly config the git remote like in the SO answer you've linked. I am able to use cml pr and it pushes and creates PRs in the repo fine, so that is why I've been assuming the token is setup correctly. Should I be explicitly configuring the git remote like in the answer above?

@BradyJ27
Copy link
Contributor Author

Also, I was able to push experiments in the past with the same project access token and an older version of DVC

@dberenbaum
Copy link
Collaborator

For the sake of completeness, do you mind checking if using git push directly in your CI script works?

@BradyJ27
Copy link
Contributor Author

It looks like running cml ci before the dvc actions begin fixes this. Interesting fix but I guess I should've been doing that anyway for completeness.

@dberenbaum
Copy link
Collaborator

That makes sense. See these related issues:

Maybe we can add into the docs (for example, here) some guidance on this. Thoughts @dacbd @pmrowla?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
awaiting response we are waiting for your reply, please respond! :) git Related to git and git backends
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@pmrowla @dberenbaum @BradyJ27