Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make URL path modification more robust #54

Merged
merged 3 commits into from
Mar 7, 2022
Merged

Make URL path modification more robust #54

merged 3 commits into from
Mar 7, 2022

Conversation

IdamkinI
Copy link

Instead of relying on '?' to be at the end of path parse the URL and modify path directly than format it back to URL string.

This should fix #53

Kalanyr and others added 3 commits June 25, 2021 20:29
@Kalanyr
Merge pull request Kalanyr#44 from Kalanyr/master
90598fb 
Update Dev with bugfixes from master
@Kalanyr
Merge pull request Kalanyr#52 from Kalanyr/master
0c9638c 
Resync Dev to Master
@IdamkinI
Fix URL path modification
c28b231 
 * Instead of relying on '?' to be at the end of path parse the URL and
modify path directly than format it back to URL string.
@FeuerTiger86
Copy link

I'm still getting the error for a few games:

12:26:19 | xml parsing error occurred trying to get md5 data for setup_divinity_2_developers_cut_1.4.700.38_lang_update_(german)_(27128)-1.bin
12:26:19 | xml parsing error occurred trying to get md5 data for setup_divinity_2_developers_cut_1.4.700.38_lang_update_(german)_(27128)-2.bin

12:26:19 | xml parsing error occurred trying to get md5 data for setup_divinity_2_developers_cut_1.4.700.38_lang_update_(german)_(27128)-1.bin
12:26:19 | xml parsing error occurred trying to get md5 data for setup_divinity_2_developers_cut_1.4.700.38_lang_update_(german)_(27128)-2.bin

12:30:42 | xml parsing error occurred trying to get md5 data for setup_prison_architect_-_cleared_for_transfer_the_tower_8524_(64bit)_(53609).exe

12:30:56 | xml parsing error occurred trying to get md5 data for setup_robin_hood_-_the_legend_of_sherwood_1.1_hotfix_(24778)-1.bin

12:38:47 | xml parsing error occurred trying to get md5 data for setup_stronghold_hd_1.41_(20341)-1.bin
12:39:04 | xml parsing error occurred trying to get md5 data for setup_stronghold_hd_1.41_(german)_(20341)-1.bin

12:39:12 | xml parsing error occurred trying to get md5 data for setup_syberia_1.0.0_hotfix4_(german)_(53936)-1.bin

12:45:36 | xml parsing error occurred trying to get md5 data for setup_system_shock_enhanced_edition_1.2.16_(64bit)_(44378).exe

12:47:33 | xml parsing error occurred trying to get md5 data for setup_toonstruck_1.0_(german)_(20270).exe
12:47:33 | xml parsing error occurred trying to get md5 data for setup_toonstruck_1.0_(german)_(20270)-1.bin

12:49:22 | xml parsing error occurred trying to get md5 data for setup_tropico_3_gold_1.0_(22808)-1.bin

12:49:50 | xml parsing error occurred trying to get md5 data for setup_ultima_underworld_1.0_cs_(28190).exe

12:52:07 | xml parsing error occurred trying to get md5 data for setup_unreal_2_-_the_awakening_special_edition_v2001_(19575).exe

In some cases, these were accompanied by this error:

12:35:13 | request failed: HTTPSConnectionPool(host='gog-cdn-lumen.secure2.footprint.net', port=443): Read timed out. (3 retries left) -- will retry in 5s...

Also, this (the same thing happened for System Shock 2):

12:39:13 | (36 / 97) fetching game details for syberia_2...
12:39:52 | request failed: HTTPSConnectionPool(host='gog-cdn-lumen.secure2.footprint.net', port=443): Read timed out. (3 retries left) -- will retry in 5s...
12:40:28 | request failed: HTTPSConnectionPool(host='gog-cdn-lumen.secure2.footprint.net', port=443): Read timed out. (2 retries left) -- will retry in 5s...
12:41:03 | request failed: HTTPSConnectionPool(host='gog-cdn-lumen.secure2.footprint.net', port=443): Read timed out. (1 retries left) -- will retry in 5s...
12:41:38 |
Traceback (most recent call last):
  File "D:\Programme\Python39\lib\site-packages\urllib3\response.py", line 438, in _error_catcher
    yield
  File "D:\Programme\Python39\lib\site-packages\urllib3\response.py", line 519, in read
    data = self._fp.read(amt) if not fp_closed else b""
  File "D:\Programme\Python39\lib\http\client.py", line 463, in read
    n = self.readinto(b)
  File "D:\Programme\Python39\lib\http\client.py", line 507, in readinto
    n = self.fp.readinto(b)
  File "D:\Programme\Python39\lib\socket.py", line 704, in readinto
    return self._sock.recv_into(b)
  File "D:\Programme\Python39\lib\ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "D:\Programme\Python39\lib\ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Programme\Python39\lib\site-packages\requests\models.py", line 758, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "D:\Programme\Python39\lib\site-packages\urllib3\response.py", line 576, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "D:\Programme\Python39\lib\site-packages\urllib3\response.py", line 541, in read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
  File "D:\Programme\Python39\lib\contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "D:\Programme\Python39\lib\site-packages\urllib3\response.py", line 443, in _error_catcher
    raise ReadTimeoutError(self._pool, None, "Read timed out.")
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='gog-cdn-lumen.secure2.footprint.net', port=443): Read timed out.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "F:\GOG\gogrepoc.py", line 229, in request
    response = session.get(url, params=args,stream=stream,timeout=HTTP_TIMEOUT)
  File "D:\Programme\Python39\lib\site-packages\requests\sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "D:\Programme\Python39\lib\site-packages\requests\sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "D:\Programme\Python39\lib\site-packages\requests\sessions.py", line 697, in send
    r.content
  File "D:\Programme\Python39\lib\site-packages\requests\models.py", line 836, in content
    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
  File "D:\Programme\Python39\lib\site-packages\requests\models.py", line 765, in generate
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='gog-cdn-lumen.secure2.footprint.net', port=443): Read timed out.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Programme\Python39\lib\site-packages\urllib3\response.py", line 438, in _error_catcher
    yield
  File "D:\Programme\Python39\lib\site-packages\urllib3\response.py", line 519, in read
    data = self._fp.read(amt) if not fp_closed else b""
  File "D:\Programme\Python39\lib\http\client.py", line 463, in read
    n = self.readinto(b)
  File "D:\Programme\Python39\lib\http\client.py", line 507, in readinto
    n = self.fp.readinto(b)
  File "D:\Programme\Python39\lib\socket.py", line 704, in readinto
    return self._sock.recv_into(b)
  File "D:\Programme\Python39\lib\ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "D:\Programme\Python39\lib\ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Programme\Python39\lib\site-packages\requests\models.py", line 758, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "D:\Programme\Python39\lib\site-packages\urllib3\response.py", line 576, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "D:\Programme\Python39\lib\site-packages\urllib3\response.py", line 541, in read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
  File "D:\Programme\Python39\lib\contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "D:\Programme\Python39\lib\site-packages\urllib3\response.py", line 443, in _error_catcher
    raise ReadTimeoutError(self._pool, None, "Read timed out.")
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='gog-cdn-lumen.secure2.footprint.net', port=443): Read timed out.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "F:\GOG\gogrepoc.py", line 229, in request
    response = session.get(url, params=args,stream=stream,timeout=HTTP_TIMEOUT)
  File "D:\Programme\Python39\lib\site-packages\requests\sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "D:\Programme\Python39\lib\site-packages\requests\sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "D:\Programme\Python39\lib\site-packages\requests\sessions.py", line 697, in send
    r.content
  File "D:\Programme\Python39\lib\site-packages\requests\models.py", line 836, in content
    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
  File "D:\Programme\Python39\lib\site-packages\requests\models.py", line 765, in generate
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='gog-cdn-lumen.secure2.footprint.net', port=443): Read timed out.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Programme\Python39\lib\site-packages\urllib3\response.py", line 438, in _error_catcher
    yield
  File "D:\Programme\Python39\lib\site-packages\urllib3\response.py", line 519, in read
    data = self._fp.read(amt) if not fp_closed else b""
  File "D:\Programme\Python39\lib\http\client.py", line 463, in read
    n = self.readinto(b)
  File "D:\Programme\Python39\lib\http\client.py", line 507, in readinto
    n = self.fp.readinto(b)
  File "D:\Programme\Python39\lib\socket.py", line 704, in readinto
    return self._sock.recv_into(b)
  File "D:\Programme\Python39\lib\ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "D:\Programme\Python39\lib\ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Programme\Python39\lib\site-packages\requests\models.py", line 758, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "D:\Programme\Python39\lib\site-packages\urllib3\response.py", line 576, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "D:\Programme\Python39\lib\site-packages\urllib3\response.py", line 541, in read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
  File "D:\Programme\Python39\lib\contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "D:\Programme\Python39\lib\site-packages\urllib3\response.py", line 443, in _error_catcher
    raise ReadTimeoutError(self._pool, None, "Read timed out.")
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='gog-cdn-lumen.secure2.footprint.net', port=443): Read timed out.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "F:\GOG\gogrepoc.py", line 229, in request
    response = session.get(url, params=args,stream=stream,timeout=HTTP_TIMEOUT)
  File "D:\Programme\Python39\lib\site-packages\requests\sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "D:\Programme\Python39\lib\site-packages\requests\sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "D:\Programme\Python39\lib\site-packages\requests\sessions.py", line 697, in send
    r.content
  File "D:\Programme\Python39\lib\site-packages\requests\models.py", line 836, in content
    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
  File "D:\Programme\Python39\lib\site-packages\requests\models.py", line 765, in generate
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='gog-cdn-lumen.secure2.footprint.net', port=443): Read timed out.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Programme\Python39\lib\site-packages\urllib3\response.py", line 438, in _error_catcher
    yield
  File "D:\Programme\Python39\lib\site-packages\urllib3\response.py", line 519, in read
    data = self._fp.read(amt) if not fp_closed else b""
  File "D:\Programme\Python39\lib\http\client.py", line 463, in read
    n = self.readinto(b)
  File "D:\Programme\Python39\lib\http\client.py", line 507, in readinto
    n = self.fp.readinto(b)
  File "D:\Programme\Python39\lib\socket.py", line 704, in readinto
    return self._sock.recv_into(b)
  File "D:\Programme\Python39\lib\ssl.py", line 1241, in recv_into
    return self.read(nbytes, buffer)
  File "D:\Programme\Python39\lib\ssl.py", line 1099, in read
    return self._sslobj.read(len, buffer)
socket.timeout: The read operation timed out

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "D:\Programme\Python39\lib\site-packages\requests\models.py", line 758, in generate
    for chunk in self.raw.stream(chunk_size, decode_content=True):
  File "D:\Programme\Python39\lib\site-packages\urllib3\response.py", line 576, in stream
    data = self.read(amt=amt, decode_content=decode_content)
  File "D:\Programme\Python39\lib\site-packages\urllib3\response.py", line 541, in read
    raise IncompleteRead(self._fp_bytes_read, self.length_remaining)
  File "D:\Programme\Python39\lib\contextlib.py", line 137, in __exit__
    self.gen.throw(typ, value, traceback)
  File "D:\Programme\Python39\lib\site-packages\urllib3\response.py", line 443, in _error_catcher
    raise ReadTimeoutError(self._pool, None, "Read timed out.")
urllib3.exceptions.ReadTimeoutError: HTTPSConnectionPool(host='gog-cdn-lumen.secure2.footprint.net', port=443): Read timed out.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "F:\GOG\gogrepoc.py", line 716, in filter_downloads
    fetch_file_info(d, True,updateSession)
  File "F:\GOG\gogrepoc.py", line 672, in fetch_file_info
    md5_response = request(updateSession,tmp_md5_url)
  File "F:\GOG\gogrepoc.py", line 243, in request
    return request(session=session,url=url, args=args, byte_range=byte_range, retries=retries-1, delay=HTTP_RETRY_DELAY)
  File "F:\GOG\gogrepoc.py", line 243, in request
    return request(session=session,url=url, args=args, byte_range=byte_range, retries=retries-1, delay=HTTP_RETRY_DELAY)
  File "F:\GOG\gogrepoc.py", line 243, in request
    return request(session=session,url=url, args=args, byte_range=byte_range, retries=retries-1, delay=HTTP_RETRY_DELAY)
  File "F:\GOG\gogrepoc.py", line 229, in request
    response = session.get(url, params=args,stream=stream,timeout=HTTP_TIMEOUT)
  File "D:\Programme\Python39\lib\site-packages\requests\sessions.py", line 555, in get
    return self.request('GET', url, **kwargs)
  File "D:\Programme\Python39\lib\site-packages\requests\sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "D:\Programme\Python39\lib\site-packages\requests\sessions.py", line 697, in send
    r.content
  File "D:\Programme\Python39\lib\site-packages\requests\models.py", line 836, in content
    self._content = b''.join(self.iter_content(CONTENT_CHUNK_SIZE)) or b''
  File "D:\Programme\Python39\lib\site-packages\requests\models.py", line 765, in generate
    raise ConnectionError(e)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='gog-cdn-lumen.secure2.footprint.net', port=443): Read timed out.
12:41:38 | failed to fetch https://www.gog.com/downloads/syberia_2/de1installer0 because of non-HTTP Error

@Kalanyr Kalanyr merged commit 1e9fc1e into Kalanyr:master Mar 7, 2022
@Kalanyr
Copy link
Owner

Kalanyr commented Mar 7, 2022

Thank you for this. I've merged this in, but please target the dev branch for merge requests in future.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

xml parsing error
3 participants
@IdamkinI @FeuerTiger86 @Kalanyr