Unsupported Semi-official Alt. Steam Patches [Lin/Mac/Win]

[tonurics]

Cool.

I replied with some extra information about rdiffdir. I'm not actually active on the GOG forum; so if anything comes up: feel free to reach out to me here.

[tomtom]

vv221 sugested that I crosspost this here from the GOG forums:

I wrote a script to create and apply patch sets, you can find it on github (add a : in the URL, I am not allowed to post URLs):

https://github.com/neovatar/patchmonger

It should make things a bit easier for Linux. Binary diffs are done with xdelta3 and LZMA compression, which should yield sufficient compression for the patches. The patchmonger script will create a tgz file with the patchset data and an embedded patchmonger, so that it is enough to download the patch tgz, extract it and run the embedded bash script.

The script is using bash, grep, sed (which are included in almost all Linux distros by default). The only thing you probably need to install is xdelta3 with LZMA support. Usually you can install it easily from your distros repository (e.g. via "sudo apt-get install xdelta3" on Ubuntu).

The script should work with all GOG Linux games tgz installers (e.g. Wasteland 2).

[tonurics]

tomtom: Thanks for reposting your script over here. I actually wrote reply with some feedback on the GOG forums.

[tonurics]

Here's some benchmarks with the current tools. In this test I created a GOG patch for: gog_wasteland_2_1.7.0.11 [WL2 5.1] -> gog_wasteland_2_1.9.0.13 [WL2 6.0]

My distribution does not have xdelta3 binaries with LZMA support. So I modified the patchmonger script to use the normal xdelta3 settings. Hopefully, since everything will be compressed with 7z afterwards it shouldn't make a big difference. But for full disclosure: I would expect these results to be smaller than the unmodified script; LZMA is applied upon all the data and gzip is not applied on top of LZMA (which would just make the files larger).

As bsdiff was the best tool during my early testing: I created a version of patchmonger that uses bsdiff in place of xdelta3 and tested that as well.

7z command used:

Code: Select all

7za a -t7z -m0=lzma -mx=9 -mfb=64 -md=32m -ms=on

Results:
Steam 5.1 -> 6.0 patch:
Size: 949.7MiB [baseline]

patchmonger - xdelta3:
Uncompressed size: 485.7MiB
7z compressed: 294.7MiB
Elapsed Time: 00:04:15
~3x smaller (68.9% reduction)

patchmonger - bsdiff:
Uncompressed size: 25.7MiB
7z compressed: 25.8MiB [size increased]
Elapsed Time: 02:37:05
~37x smaller (97.2% reduction)

rdiffdir - rsync:
Uncompressed size: 410.1MiB
7z compressed: 158.9MiB
Elapsed Time: 00:01:14
~6x smaller (83.2% reduction)

There were only 2 new files [totaling: 114.0KB]; so I was fully expecting xdelta3 to wipe the floor with rdiffdir. But it looks like rsync has a few tricks up it's sleeve. My guess: by searching all the data, it's able to find hash collisions at a greater rate; while the patchmonger process is limited to only files that share a path/name.

But the clear winner here is bsdiff. The reduction was an astounding 97.2%. It's delta was soo compact: 7z wasn't able to do anything with it. Too bad it took over 2 and a half hours to complete.

FYI: During testing, I didn't apply the xdelta3 patch. But I did test the bsdiff patch; it worked correctly and only took about 3 minutes to apply. The rdiffdir patch was tested previously.

[vv221]

Looks like patch 6 is the final patch for Wasteland 2 vanilla, new patches will be for the enhanced edition only. So I’m going to build a new round of patches for those who still have a non up-to-date version of WL2: those patches will allow to upgrade any GOG version to patch 6.
Well, not really "any" actually, as I didn’t keep all versions of the game. Here are the GOG releases that I plan to use as a base for these new patches:
1.0.0.4, 1.1.0.5, 1.3.0.7, 1.6.0.10, 1.7.0.11, 1.8.0.12

If some of you fellow Rangers use another version and would like to see a patch for it, please post here and I’ll explain what to do to help me build a patch for this version.

[vv221]

Done.

Cumulative patches can be found there (Linux/GOG only):
http://www.dotslashplay.it/ressources/wasteland-2/
Iterative patches can still be found there (Windows, OS X & Linux / Steam & GOG):
http://www.dotslashplay.it/ressources/w ... cs-builds/

As usual, all feedback is welcome, even if it is a simple "it works".

[tonurics]

Thanks for the new GOG patches vv221!

[vv221]

I see these patches as "learning by fun" more than tedious work, I just *had to* see if I could build cumulative patches of small size.
Looks like I can thanks to your method, so it would have been a shame to not share them!

Really, only ~730MiB for a patch from release version (actually 2.something I think) to the last 6.0 patch?
Steam is lucky I’m not (yet) launching my own platform of games distribution ;)

[tonurics]

My run through put me in a basic firefight with some raiders. I noticed right away that the environment seemed a bit different and, afterwards, Beekers shed light on why Wasteland 2 vets won’t feel right at home.

“A lot of the combat has been improved as well. You’ll have more cover, more height differentials, things like that,” he said. “We redid all the areas, we redid all the character models, we did a bunch of extra voice acting. On the system side we added perks and quirks.”

http://www.idigitaltimes.com/wasteland- ... aul-451722

I couldn't really comment before E3: But I don't think we'll be able to create very useful v6 -> GOTY patches. Rest assured, I'm still going to make one as a thought experiment.

[vv221]

GOG now provides brand new installers based on MojoSetup. One of the announced features is… incremental patches!
I’ll keep the cumulative patches I’ve built online anyway, just in case someone needs them at some point.

But it means this kind of custom-made patches might not be needed for the Director’s Cut edition that will be provided on GOG.
I’ll try to build some when it will get its first update to see if I get something significantly smaller in size than the GOG ones, but if I don’t it will probably mean my work here is done

[tonurics]

Thanks for the heads up; glad to hear positive change is afoot for Linux GOG.com gamers.

With the Director's Cut being such a huge overhaul: I'm not sure a meaningful upgrade path exists; it might spell the end of the semi-offical patches. My original goal was to add value to the physical copies of the game.

[vv221]

With the Director's Cut being such a huge overhaul: I'm not sure a meaningful upgrade path exists

Oh, I wasn’t thinking about a vanilla -> director’s cut patch. I will try to build one out of curiosity, but it will probably not end in anything useful to publish.
What I want to see is how well GOG is going to handle the patching for Linux, so I’ll build a "DC release -> DC patch 1" custom patch for comparison. If it doesn’t have a greatly reduced size (I hope it won’t) it will be time for me to give this project a (well-deserved) rest.

[tonurics]

To give everyone a little update:

rdiffdir doesn't create useful patches for: Wasteland 2 [Patch 6] -> Wasteland 2: Director's Cut

[vv221]

Well, unless anyone think a 15GiB patch could be "useful"

[tonurics]

I finally got around to writing a patching script of my own. It's multi-threaded, supports bsdiff, rdiff and xdelta3 delta encoding and transparently saves everything into a master "patch" zip file. At the moment it's only able to create patches (I haven't written anything to apply them yet). But the patch file format is human readable, so it should be simple for anyone to jump in and write a patching tool in their programming language of choice.

This is still just a pet project, so I don't know if it will see a release. But while playing around: I made some interesting benchmarks from on the Wasteland 2 "Classic" Vanilla Backer Edition 1.0 to a "Classic" Patch 6.0:

rdiffdir [baseline]
Uncompressed size: N/A (~9GiB?)
Zip compressed: N/A
7z compressed: 4.2GiB
Elapsed Time: N/A (~2 hours?)
Note: Original Unsupported Semi-official from last year (I'm not going to recreate it to find the real numbers).

----

Custom - bsdiff [FAILED]
Threads: 4
Uncompressed size: 4.5GiB*
Zip compressed: N/A
7z compressed: 4.5GiB*
Elapsed Time: 06:47:23*
Notes: I think one of the threads ran out of memory and locked up early on. As such, only 124 of the 125 total files were processed before I killed the task. I estimate the missing file to be 200-350MiB in size (based on what the other algorithms did with it).

Custom - rdiff
Threads: 12
Uncompressed size: 11.2GiB
Zip compressed: 5.7GiB
7z compressed: 4.3GiB
Elapsed Time: 00:16:52
Notes: Based on rsync; this is the same delta encoding used by rdiffdir.

Custom - xdelta
Threads: 12
Uncompressed size: 5.3GiB
Zip compressed: 5.3GiB
7z compressed: 5.3GiB
Elapsed Time: 00:12:33
Notes: VCDIFF level 9 compression with LZMA secondary compression. VCDIFF is probably the best choice for cross platform support.

----

I continue to be surprised by the results of rdiff and rdiffdir. I really expected bsdiff to mop the floor with everything. But once compressed in 7z, rdiff and rdiffdir continue to be the best options for patching WL2. I can only surmise that in working with such a large file set (~18.8GiB), the rsync algorithm does a really good job of creating delta regions that can be deduplicated/compressed by additional tools (i.e. 7z or zip).

The bsdiff story might change, if everything was put in a single tarfile, then patched [like how rdiffdir works]: that might allow it to do some deduping of it's own. But I estimate the required amount of RAM needed by bsdiff for that to be in the 150.4-319.6GiB range.

[vv221]

Hey, I’m interested
Anywhere we can download your patching script from, even if you think it’s not ready for a release?

[kilobug]

Now that I've fiber at home I'm not directly concerned anymore by those patches (but they did help me back then at WL2 release when I only had very slow DSL), but the geek in me still find this whole thing very interesting, so thanks for the detailed update

[tonurics]

Hey, I’m interested
Anywhere we can download your patching script from, even if you think it’s not ready for a release?

Not yet, I would need to make the script a little more portable and add support for command line switches first.

---

I did think up some more experiments over the weekend:

When I made the Non-Steam backer discs: I was able to get all three Patch 6 versions of the game [Lin/Mac/Win, totaling: 56.3GiB] squashed down to the size of 4GiB. Ever since then, it's annoyed me that the 1.0 -> 6.0 patches have been larger than the actual game! Well I'm happy to report that is finally no longer the case; using tar [recreating the rdiffdir process], srep and xdelta: I created a 1.0 -> 6.0 patch that is only ~2.8GiB.

I would guess the process to create the patch took about ~2 hours (which puts it in line with rdiffdir). I think the Steam 1.0 -> 6.0 patching process has you download something like ~18.8GiB! But while I'm excited about the new results, it's not worth putting any effort into releasing a new "Classic" patch in the tar/srep/xdelta format.

Less exciting, but also noteworthy:

• Using xdelta [without the srep step] to create a delta on the same tar files above, results: in a patch that is only 6MiB smaller than the non-tar version. I was hoping to see much larger gains (like the above).
• A version of my script that applies srep to the individuals files before using xdelta to create deltas, results: in a patch that is ~205MiB larger than the version without it.

[vv221]

It’s highly unlikely that inXile would publish new patches to Wasteland 2, and this project hasn’t seen a lot of activity lately… So I’m planning to take down the patches from the mirror I host in a not-so-far future.

Feel free to download them and host them anywhere if you would like to see them kept alive.