Unsupported Semi-official Alt. Steam Patches [Lin/Mac/Win]

[vv221]

Downloading your patches from filefactory.com just now, which has the same flaws than most free hosts: limited download speed and simultaneous downloads not allowed (unless the downloader is willing to pay)…
Well, I’m definitely going to host them, with no other limitation than my own bandwidth (~5x the speed allowed by filefactory for free downloads)

Plus, thanks to them I will be able to update my scripts to new scripts building the last version of WL2 from any version uploaded by GOG.com.

[vv221]

Done, you can now access the patches from my server, with no limitation other than technical ones:

Windows
2.x -> 3.1: http://www.dotslashplay.it/ressources/w ... -patch3.7z
3.1 -> 4.1: http://www.dotslashplay.it/ressources/w ... -patch4.7z
4.1 -> 5.1: http://www.dotslashplay.it/ressources/w ... -patch5.7z

Mac OS
3.1 -> 4.1: http://www.dotslashplay.it/ressources/w ... -patch4.7z
4.1 -> 5.1: http://www.dotslashplay.it/ressources/w ... -patch5.7z

Linux
2.x -> 3.1: http://www.dotslashplay.it/ressources/w ... -patch3.7z
3.1 -> 4.1: http://www.dotslashplay.it/ressources/w ... -patch4.7z
4.1 -> 5.1: http://www.dotslashplay.it/ressources/w ... -patch5.7z

Feel free to add these links to your opening post

[vv221]

Your patches are not working here, here is what I got:

Code: Select all

dave@HAL9000:~/jeux/wasteland$ rdiffdir patch wasteland-2_4.1/game/WL2_Data linux_patch4-patch5.delta 
Traceback (most recent call last):
  File "/usr/bin/rdiffdir", line 232, in <module>
    main()
  File "/usr/bin/rdiffdir", line 219, in main
    patch(file_args[0], get_fileobj(file_args[1], "rb"))
  File "/usr/bin/rdiffdir", line 191, in patch
    patchdir.Patch(path.Path(dirname), deltafp)
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 46, in Patch
    patch_diff_tarfile( base_path, diff_tarfile )
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 77, in patch_diff_tarfile
    ITR( basis_path.index, basis_path, diff_ropath )
  File "/usr/lib/python2.7/dist-packages/duplicity/lazy.py", line 335, in __call__
    last_branch.fast_process, args)
  File "/usr/lib/python2.7/dist-packages/duplicity/robust.py", line 37, in check_common_error
    return function(*args)
  File "/usr/lib/python2.7/dist-packages/duplicity/patchdir.py", line 299, in fast_process
    assert diff_ropath.difftype == "diff", diff_ropath.difftype
AssertionError: snapshot

Same bug with linux_patch3-patch4.delta & linux_patch4-patch5.delta, with duplicity 0.6.24, 0.6.25 & 0.7.0.
I tried to extract the *.delta with tar, and everything went well. Any idea about what is not going as expected?

A few things might help me find where the fault is:
What is the distribution you use?
Which version of duplicity?
Which version of python?

[tonurics]

If you're patching the GOG.com version: the directory is "/Wasteland 2/game" not "/Wasteland 2/game/WL2_Data" hopefully that solves the problem! If not let me know and I'll help investigate the issue.

For completeness:
Distro: Arch Linux 3.17.6
Duplicity: 0.7.0
Python2: 2.7.9

And thanks for resharing the patches; I'll include them with the main post once we figure out what's wrong.

[vv221]

Okay, this time rdiffdir ran without problem, but I’m quite sure I’m not getting the expected result.

Wasteland 2 3.1 'game' dir before patching:

Code: Select all

dave@HAL9000:/media/stockage/dave/jeux/wasteland$ ls -1 wasteland-2_3.1/game/
WL2
WL2_Data

Wasteland 2 3.1 'game' dir after patching:

Code: Select all

dave@HAL9000:/media/stockage/dave/jeux/wasteland$ ls -1 wasteland-2_3.1to4.1/game/
level0
Localization
mainData
Managed
PlayerConnectionConfigFile
resources.assets
sharedassets0.assets
sharedassets1.assets
Streaming
WL2
WL2_Data

resources.assets after patching is unchanged:

Code: Select all

dave@HAL9000:~/jeux/wasteland$ md5sum wasteland-2_3.1to4.1/game/WL2_Data/resources.assets 
994b0fb6db3879377e7a3d3064347e00  wasteland-2_3.1to4.1/game/WL2_Data/resources.assets

I’m running a longer test to confirm this, but it looks like nothing in WL2_Data has been modified by rdiffdir.
EDIT: A recursive diff confirms this, nothing has been touched under WL2_Data.

I get similar results while trying to patch the 4.1 version to 5.1.
Tests ran with duplicity 0.7.0 and python2 2.7.9.

[tonurics]

At least we solved your initial problem!

Hmm, I tested the 3.1 -> 4.1 patch when I set everything up and it was working... Your final game directory should only have the WL2 binary and WL2_Data directory (where the other files live).

It shouldn't matter, but perhaps try running rdiffdir in the same working directory as the "game" directory (that's the only thing I see you doing differently).

Unfortunately, I don't have access to my patch stuff until next week (nor did I save the old game versions).

[vv221]

No luck here, running rdiffdir from the 'game' dir (with 'rdiffdir patch . /whatever/linux_patch3-patch4.delta') achieve the same result.

Could you describe the method you use to build the .delta files?
I’m going to try to build them, and see if I get something different from your patches.

[tonurics]

Sure, the process is pretty simple.

You first need to create a signature of a repository (i.e. game folder):

Code: Select all

rdiffdir signature [gog_wasteland_2_1.5.0.9 "/Wasteland 2/game" path] gog_wasteland_2_1.5.0.9.sig

Then using the signature, we create a delta of the differences between it and a newer repository:

Code: Select all

rdiffdir delta gog_wasteland_2_1.5.0.9.sig [gog_wasteland_2_1.6.0.10 "/Wasteland 2/game" path] patch1.5.0.9_1.6.0.10.delta

We can then use the delta to patch any repositories matching the original signature:

Code: Select all

rdiffdir patch [gog_wasteland_2_1.5.0.9 "/Wasteland 2/game" path] patch1.5.0.9_1.6.0.10.delta

[vv221]

Following your instructions I have been able to build a working .delta file for 4.1 -> 5.1. Sizes mismatch between the .delta I just created and the one you provide.
What do you think? Might there be something faulty in the .delta files you built for Linux? Or is there a difference between Steam and GOG versions of WL2? Or anything I just don’t think of?

Anyway, if you’re okay with this I should be able to build .delta files for every Linux GOG.com published version, and put them online for testing.
For the Windows and Mac OS X ones, you still are on your own yet
Well, actually I should be able to access Windows-powered toasters quite easily, but I kept backup of old GOG.com versions of WL2 only for Linux.

-----

Anyway, thank you for the short guide about rdiffdir, I can see quite a bunch of uses for this tool in my multiples projects

[tonurics]

What do you think? Might there be something faulty in the .delta files you built for Linux? Or is there a difference between Steam and GOG versions of WL2? Or anything I just don’t think of?

Ha! Well, I think I spotted the problem: you're using the Steam patches!

The GOG.com patches are here: viewtopic.php?f=34&t=11081#p141739 The data is largely the same, but the file hierarchy is slightly different (which would explain why things aren't showing up in the correct locations).

The Steam deltas are named: linux_patch2-patch3, linux_patch3-patch4, linux_patch4-patch5, etc. while the GOG ones are named: patch1.5.0.9_1.6.0.10, patch1.6.0.10_1.7.0.11, etc. (after the GOG version numbers).

Anyway, thank you for the short guide about rdiffdir, I can see quite a bunch of uses for this tool in my multiples projects

No problem! I haven't looked at the source, but I've long suspected rdiffdir is really just a wrapper for tar and bsdiff (your Traceback has confirmed tar is one of the components). It might be easily replaced with something simpler / more cross compatible.

[vv221]

Ha! Well, I think I spotted the problem: you're using the Steam patches!

Did I say anything wasn’t working? If you’re looking for me I’m probably hiding in a corner, crying…
Okay, now I’m going to re-run my tests with the right patches

-----

Here are the links to the GOG patches on my server:
http://www.dotslashplay.it/ressources/w ... .6.0.10.7z
http://www.dotslashplay.it/ressources/w ... .7.0.11.7z

[vv221]

Looks like we’re halfway to the goal: the delta 'patch1.6.0.10_1.7.0.11.delta' is working as expected, while 'patch1.5.0.9_1.6.0.10.delta' fails with the "AssertionError: snapshot" I posted earlier.

When building a new (working) delta for upgrading 1.5.0.9 to 1.6.0.10, sizes of yours and mines mismatch.
When building a new delta for upgrading 1.6.0.10 to 1.7.0.11, yours and mines are the same size (diff actually reports they differ, but it might only be due to different tar versions between our respective systems, or something like this).

-----

Phew, I think I might take a break now, it’s 5AM on this side of Earth, and I got one patch working.
I can upload my working patch for upgrading from 1.5.0.9 to 1.6.0.10 if you wish, as well as building a couple more for the older versions.

Just thinking about it: it would be no hassle for me to build patches for upgrading every published GOG.com version to the latest build (Linux only), now that you’ve showed me how to do it. It would at least lighten a bit your "burden"

[tonurics]

Hmm, they were created the same way... Did you check the md5sums of the 7z archive and resources.assets?

Another idea: before applying the delta, delete the PlayerConnectionConfigFile in the WL2_Data folder. A shot in the dark, but that might have done that when I created the signature originally.

I'd be happy to have you involved with creating the GOG.com Linux patches. But I'm not sure we can continue to call these "Semi-official" if they aren't coming from inXile directly; so I'll need to think on it.

Although, if you want to create patches for the older GOG versions: I don't see any harm in it. I don't have them myself and I can credit you with putting them together. But first: I'd like to figure out our patching issue before we put more work into these; you're my only beta tester.

[vv221]

Hmm, they were created the same way... Did you check the md5sums of the 7z archive and resources.assets?

Checked and double-checked.

Another idea: before applying the delta, delete the PlayerConnectionConfigFile in the WL2_Data folder.

Still getting the "AssertionError: snapshot". I wish it were a more easily understandable error message.

I'd be happy to have you involved with creating the GOG.com Linux patches. But I'm not sure we can continue to call these "Semi-official" if they aren't coming from inXile directly; so I'll need to think on it.

Like you said, there will be plenty of time to think about it when we’ll finally get the tricky patch working as expected…

I'd like to figure out our patching issue before we put more work into these; you're my only beta tester.

And we’re precisely in a situation where more testers would be a great help.
What if the version of WL2 I use as a base is corrupted, despite the right checksum for the resources.assets file?
I’m going to raise a call on the GOG forums to get some help sorting this out. (EDIT: Done!)

To any Linux GOGer reading this, if you still have the 1.5.0.9 archive, could you please tell me if you get the same md5 checksum I get?

Code: Select all

c1bba41693284ac7f72bec23598dd277  gog_wasteland_2_1.5.0.9.tar.gz

-----

For a week or two I should have access to other computers, running other versions of Debian (Wheezy i386, Wheezy amd64, and Jessie i386 or amd64, can’t remember the architecture of the last one). I’m going to run the same test on these in a couple days (very poor connection to transfer the 11G WL2 archive through), just to be sure I’m not experimenting a rdiffdir bug on my main computer (Debian Sid amd64).

[tonurics]

When I get back to the office next week, I'll md5sum the signatures I created. I assume that they should be the same on both our systems. That should help us figure out what has happened. (If the deltas are created with tar: it would record the owner and group ids of our files. Which could explain why our deltas are different, but still work.)

If the base is corrupted, then any output is going to be messed up as well. Based on what I read here I take it that the "AssertionError: snapshot" is because it hit a file it didn't like and gave up.

And thanks for making the post on the GOG forums!

[vv221]

Based on what I read here I take it that the "AssertionError: snapshot" is because it hit a file it didn't like and gave up.

While 141 files to test one by one would be an awful chore for any human being, it should be a piece of cake for the adequate shell script… Getting to work on it right now!

-----

It’s going to take a while, as the script needs to put back WL2 game dir in its original state after every failed attempt to patch it (once for every file in it until it achieve to patch it or it has tried once with each file removed in turn with no success). As I can’t find a way to see what has been modified before the patching fails, it means copying the whole game dir each time.
I let you guess how long this could take before every file has been tested individually… A chance I have a nearly idle server ready to see this through completion!

I’ll report here once it’s done (might take a dozen of hours), hopefully with the identity of the offending file.

As this test is a time-consuming task but very light on CPU and memory, I still can run other tests while this one is going on.

[kilobug]

As I said on the GOG forum, I can do tests in the future, but I didn't keep the old archives until now, so I can't do much help for now.

[tonurics]

I let you guess how long this could take before every file has been tested individually… A chance I have a nearly idle server ready to see this through completion!

I’ll report here once it’s done (might take a dozen of hours), hopefully with the identity of the offending file.

As this test is a time-consuming task but very light on CPU and memory, I still can run other tests while this one is going on.

Haha, I like your gusto! I was more or less content with knowing what was causing the cryptic error message. But it will be interesting to find out more.

As I said on the GOG forum, I can do tests in the future, but I didn't keep the old archives until now, so I can't do much help for now.

Thanks kilobug! Glad to have you aboard. Ideally, all you will need to do is confirm new patches work on your system.

[vv221]

Haha, I like your gusto! I was more or less content with knowing what was causing the cryptic error message. But it will be interesting to find out more.

I finally had no luck with this: I tried applying the patch with each time a different file removed from the game folder (minus resources.assets, it means 140 iterations), but got the same error message each and every time.
I don’t feel like trying it again for every possible two-files combination: it would take about three months on my computer with the method I used. I had an idea allowing to do it in only one day, but I’m missing 34 of the computers needed to put it to the trial of reality (well, one supercomputer could be enough, but I don’t know of anybody ready to lend one for the experience)

[tonurics]

Here are the md5sums of my GOG Linux signatures. Hopefully they match yours.

Code: Select all

11afd5fbf087a68539179b953420753a  gog_wasteland_2_1.5.0.9.sig
4a736f6a93eee53f90b5a99a52909705  gog_wasteland_2_1.6.0.10.sig
edf74e82ec79419f74a31777f64d6520  gog_wasteland_2_1.7.0.11.sig