Games that corrupt the manifest DAT file

[bassclarinetl2]

Version String: Python 3.11.0 (main, Oct 24 2022, 18:26:48) [MSC v.1933 64 bit (AMD64)] on win32

Hi Kalanyr,

I thought (in a similar vein to #60 and #61) I'd pass along a number of games that seem to corrupt the manifest when running py -m gogrepoc update (without a gog-manifest.dat) and the py -m download—related explicitly to null characters being present in the data file (which can be manually resolved by looking for the special regex character \0 and removing the entire game entry (e.g. from {'bg_url: to its corresponding right bracket }. The affected games are listed below the ######################### .

I have also noticed that many of the affected sections have excessively long changelog entries if that helps with diagnostics.

An excerpt of the series of outputs from running py -m gogrepoc download can be found here: https://pastebin.com/dwgKU9Jp

NOTE: In the pastebin ">>> preceded by an empty line is my redacted prompt

#########################

Cossacks Anthology: https://www.gog.com/en/game/cossacks_anthology
Cyberpunk 2077: https://www.gog.com/en/game/cyberpunk_2077
Gabriel Knight: Sins of the Fathers: https://www.gog.com/en/game/gabriel_knight_sins_of_the_fathers_20th_anniversary_edition
Geneforge 1: https://www.gog.com/en/game/geneforge_15
Planetscape - Torment: https://www.gog.com/en/game/planescape_torment_enhanced_edition
SpaceChem: https://www.gog.com/en/game/spacechem
Spelunky: https://www.gog.com/en/game/spelunky
Neverwinter Nights Diamond Edition: https://www.gog.com/en/game/neverwinter_nights_enhanced_edition_pack
Risen: https://www.gog.com/en/game/risen
Rise of the Triad: https://www.gog.com/en/game/rise_of_the_triad__dark_war and https://www.gog.com/en/game/rise_of_the_triad
Stellaris: https://www.gog.com/en/game/stellaris
Stronghold Crusader/HD: https://www.gog.com/en/game/stronghold_crusader
The Witcher: https://www.gog.com/en/game/the_witcher
The Witcher 2: Enhanced Edition (there seemed to be two entries for this game although only one was affected): https://www.gog.com/en/game/the_witcher_2
SteamWorld Dig: https://www.gog.com/en/game/steamworld_dig
Thea 2 The Shattering: https://www.gog.com/en/game/thea_2_the_shattering
Yooka Laylee: Impossible Lair: https://www.gog.com/en/game/yookalaylee_and_the_impossible_lair
Alan Wake: https://www.gog.com/en/game/alan_wake
Alan Wake: American Nightmare: https://www.gog.com/en/game/alan_wakes_american_nightmare

[Kalanyr]

Can you provide some more information please ? I have many of these games and don't get a corrupted manifest from them. What language are you fetching the data in ? If the bad character occurs in a place that doesn't leak personal information (or unique URLS) could you please include a copy of that section of the manifest please ? I have encountered situations where long changelogs include invalid characters but I've fixed that.

…

On Sun, Dec 18, 2022 at 1:08 PM Will Heid ***@***.***> wrote: Version String: Python 3.11.0 (main, Oct 24 2022, 18:26:48) [MSC v.1933 64 bit (AMD64)] on win32 Hi Kalanyr, I thought (in a similar vein to #60 <#60> and #61 <#61>) I'd pass along a number of games that seem to corrupt the manifest when running py -m gogrepoc update (without a gog-manifest.dat) and the py -m download—related explicitly to null characters being present in the data file (which can be manually resolved by looking for the special regex character \0 and removing the entire game entry (e.g. from {'bg_url: to its corresponding right bracket }. The affected games are listed below the ######################### . I have also noticed that many of the affected sections have excessively long changelog entries if that helps with diagnostics. An excerpt of the series of outputs from running py -m gogrepoc download can be found here: https://pastebin.com/dwgKU9Jp NOTE: In the pastebin ">>> preceded by an empty line is my redacted prompt ######################### Cossacks Anthology: https://www.gog.com/en/game/cossacks_anthology Cyberpunk 2077: https://www.gog.com/en/game/cyberpunk_2077 Gabriel Knight: Sins of the Fathers: https://www.gog.com/en/game/gabriel_knight_sins_of_the_fathers_20th_anniversary_edition Geneforge 1: https://www.gog.com/en/game/geneforge_15 Planetscape - Torment: https://www.gog.com/en/game/planescape_torment_enhanced_edition SpaceChem: https://www.gog.com/en/game/spacechem Spelunky: https://www.gog.com/en/game/spelunky Neverwinter Nights Diamond Edition: https://www.gog.com/en/game/neverwinter_nights_enhanced_edition_pack Risen: https://www.gog.com/en/game/risen Rise of the Triad: https://www.gog.com/en/game/rise_of_the_triad__dark_war and https://www.gog.com/en/game/rise_of_the_triad Stellaris: https://www.gog.com/en/game/stellaris Stronghold Crusader/HD: https://www.gog.com/en/game/stronghold_crusader The Witcher: https://www.gog.com/en/game/the_witcher The Witcher 2: Enhanced Edition (there seemed to be two entries for this game although only one was affected): https://www.gog.com/en/game/the_witcher_2 SteamWorld Dig: https://www.gog.com/en/game/steamworld_dig Thea 2 The Shattering: https://www.gog.com/en/game/thea_2_the_shattering Yooka Laylee: Impossible Lair: https://www.gog.com/en/game/yookalaylee_and_the_impossible_lair Alan Wake: https://www.gog.com/en/game/alan_wake Alan Wake: American Nightmare: https://www.gog.com/en/game/alan_wakes_american_nightmare — Reply to this email directly, view it on GitHub <#81>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKZ337XMUOQOSAUOACQ4PLWNZ5ZJANCNFSM6AAAAAATCI5HXI> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[Kalanyr]

Hmmm. Not sure that's the same issue. That looks like GOG either sending bad info or GOGrepoc being run multiple times and writing to the manifest simultaneously.

…

On Mon, 19 Dec 2022, 04:38 code8buster, ***@***.***> wrote: I believe I'm getting a similar problem. Here's a demonstrative extract showing things way out of place. I could provide the full file in an email. I also have The Witcher and NVN:Diamond Edition in my list. 'forum_url': 'https://www.gog.com/forum/caves_of_qud', 'galaxyDownloads': [], 'genre': 'Strategy', 'gog_messages': [], 'has_updates': False, 'id': 1625207125, 'image_url': '//images-2.gog-statics.com/423d183ed9056b1640c05a01adffdac2e1b2a4fa874229cb7b2b921e1873e2ae', 'long_title': 'Caves of Qud', 'media_type': '1', 'old_title': None, 'rating': 44, 'release_timestamp': 1531238100, 'serial': '', 'sharedDownloads': [], 'store_url': '/en/game/caves_of_qud', 'title': 'caves_of_qud'}, {'bg_url': '//images-3.gog-statics.com/dc3c4cfbc9ec2fae5c5e1be6956fb926fc0028df2854c8447c73ac0b8807126c', 'changelog': '', 'downloads': [], 'extras': [{'desc': 'manual', 'href': 'https://www.gog.com/downloads/deus_ex/13503', 'lang': '', 'md5': None, 'name': 'deus_ex_manual.zip', 'old_name': None, 'os_type': 'extra', 'prev_verified': False, 'size': 3200151, I'm not sure what to look for in terms of sensitive info for posting the full manifest. — Reply to this email directly, view it on GitHub <#81 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKZ336OPCSUYBX7AXC6L4TWN5K37ANCNFSM6AAAAAATCI5HXI> . You are receiving this because you commented.Message ID: ***@***.***>

[bassclarinetl2]

Sorry for the delay. Just reran it. and it works once (login, update, and download) and then when running update again to make sure it hasn't missed anything, it throws:

14:52:44 | loading local manifest...
14:52:44 | fatal...
Traceback (most recent call last):
  File "H:\gog\gogrepoc.py", line 2931, in <module>
    main(process_argv(sys.argv))
  File "H:\gog\gogrepoc.py", line 2670, in main
    cmd_update(args.os, args.lang, args.skipknown, args.updateonly, not args.full, args.ids, args.skipids,args.skiphidden,args.installers,args.resumemode,args.strictverify)
  File "H:\gog\gogrepoc.py", line 1274, in cmd_update
    gamesdb = load_manifest()
              ^^^^^^^^^^^^^^^
  File "H:\gog\gogrepoc.py", line 369, in load_manifest
    ad = r.read()
         ^^^^^^^^
  File "<frozen codecs>", line 701, in read
  File "<frozen codecs>", line 504, in read
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xfb in position 1004556: invalid start byte```

[bassclarinetl2]

To answer your earlier question regarding the manifest file itself and where I'm seeing the nul characters:

I'm using Sublime Text as the editor.

[Kalanyr]

Actually could you explain what's out of place ? Having looked at it again it looks okay ? Just taken at the intersection. bg_url is the first part of an entry and title is the last (it's sorted alphabetically when output).

…

On Mon, 19 Dec 2022, 07:32 Kalanyr, ***@***.***> wrote: Hmmm. Not sure that's the same issue. That looks like GOG either sending bad info or GOGrepoc being run multiple times and writing to the manifest simultaneously. On Mon, 19 Dec 2022, 04:38 code8buster, ***@***.***> wrote: > I believe I'm getting a similar problem. Here's a demonstrative extract > showing things way out of place. I could provide the full file in an email. > I also have The Witcher and NVN:Diamond Edition in my list. > > 'forum_url': 'https://www.gog.com/forum/caves_of_qud', > 'galaxyDownloads': [], > 'genre': 'Strategy', > 'gog_messages': [], > 'has_updates': False, > 'id': 1625207125, > 'image_url': '//images-2.gog-statics.com/423d183ed9056b1640c05a01adffdac2e1b2a4fa874229cb7b2b921e1873e2ae', > 'long_title': 'Caves of Qud', > 'media_type': '1', > 'old_title': None, > 'rating': 44, > 'release_timestamp': 1531238100, > 'serial': '', > 'sharedDownloads': [], > 'store_url': '/en/game/caves_of_qud', > 'title': 'caves_of_qud'}, > {'bg_url': '//images-3.gog-statics.com/dc3c4cfbc9ec2fae5c5e1be6956fb926fc0028df2854c8447c73ac0b8807126c', > 'changelog': '', > 'downloads': [], > 'extras': [{'desc': 'manual', > 'href': 'https://www.gog.com/downloads/deus_ex/13503', > 'lang': '', > 'md5': None, > 'name': 'deus_ex_manual.zip', > 'old_name': None, > 'os_type': 'extra', > 'prev_verified': False, > 'size': 3200151, > > > I'm not sure what to look for in terms of sensitive info for posting the > full manifest. > > — > Reply to this email directly, view it on GitHub > <#81 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/ABKZ336OPCSUYBX7AXC6L4TWN5K37ANCNFSM6AAAAAATCI5HXI> > . > You are receiving this because you commented.Message ID: > ***@***.***> >

[Kalanyr]

Hmm. Interesting that is definitely extremely corrupted, a large chunk of the entry is missing. Could you backup the current manifest, delete and try a fresh run? Then backup the new manifest as well straight after the update command. Then try the download. My initial suspicion is that an entry was corrupt but processing has propagated that corruption, so starting over with a new manifest should either resolve the issue (if it's a result of external to gogrepoc issues) or let us identify the source more easily.

…

On Mon, 19 Dec 2022, 09:01 Will Heid, ***@***.***> wrote: To answer your earlier question regarding the manifest file itself and where I'm seeing the nul characters: [image: image] <https://user-images.githubusercontent.com/5884987/208323898-147ca368-3b4b-4962-81d1-9c20cec6f151.png> I'm using Sublime Text as the editor. — Reply to this email directly, view it on GitHub <#81 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKZ337FKH7USQBXYXKI2EDWN6JVNANCNFSM6AAAAAATCI5HXI> . You are receiving this because you commented.Message ID: ***@***.***>

[bassclarinetl2]

Sorry for the delayed reply. Deleting the manifest (e.g. rm gog-manifest.dat) followed by an gogrepoc update does appear to fix things. That said, running an update against the existing manifest seems to introduce corruption. At this point its more a nuisance since the workaround seems to fix things.

[Kalanyr]

So as far as you can tell there's no corruption in the existing manifest prior to running the update but there is afterwards ? Could I get you to email me (Kalanyr AT nospam gmail DOT com ) a copy of those two then ? I'd like to examine them. I suspect there's some kind of invisible corruption in the first manifest which the process of reading and updating is spreading. If the update data itself was corrupt the new manifest should be corrupt too.

…

On Thu, 29 Dec 2022, 13:52 Will Heid, ***@***.***> wrote: Sorry for the delayed reply. Deleting the manifest (e.g. rm gog-manifest.dat) followed by an gogrepoc update does appear to fix things. That said, running an update against the existing manifest seems to introduce corruption. At this point its more a nuisance since the workaround seems to fix things. — Reply to this email directly, view it on GitHub <#81 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABKZ3325VMHL3BINJAKPCBLWPUDIRANCNFSM6AAAAAATCI5HXI> . You are receiving this because you commented.Message ID: ***@***.***>