Thread: OpenRS2 Archive

I've been working on this service for a while now, and it's now ready to be announced publicly (though there's still lots of work to do!)

OpenRS2 Archive

It's currently a collection of caches and XTEA keys for all versions of RuneScape that use JS5, and its precursor, to manage assets (mainline RuneScape build 234+ and all builds of OSRS). Support for archiving clients/native libraries is planned in the future.

It automatically downloads new OSRS and RS3 caches when they are released and pulls OSRS XTEA keys from various sources.

Unlike the existing archives, it's backed by a database - not a collection of static files. The groups are stored in a content-addressable format (based on (archive, group, checksum, version) tuples), which provides several benefits:

* The disk usage is vastly reduced. Currently it uses 123 GB to store over 1,500 caches for a mixture of mainline RS and OSRS builds. The database backups (which are compressed and omit the indexes) are even smaller - around 48 GB at the moment. Storing all these caches individually would use over 8.3 TB!
* Missing groups and keys can be located from nearby caches, as long as the checksum and version number are identical.

The internal format is converted to .dat2/.idx on demand when a cache is downloaded. An alternative flat file format will be added for caches larger than 8GB (the max size of the .dat2 file) in the future.

XTEA keys are automatically verified and they can be downloaded as a JSON file for every cache present in the archive. I'll add support for downloading a ZIP of text files in the future too. As many of the old dumpers didn't record the region ID for dynamic regions correctly, the service verifies every candidate key against every encrypted group for every version of the cache (and all future caches when they are imported). This allows us to eke out as many validated keys as possible.

One known problem at the moment that I'm still fixing is that there are a small number of (archive, group, checksum, version) collisions (8 across the entire database at the time of writing). Some of the GZIP-compressed containers use the stored compression level (no compression). As GZIP has a CRC-32 trailer, any changes to the uncompressed data are actually cancelled out by the trailer (due to an unfortunate property of the CRC algorithm), rendering the overall CRC-32 checksum unchanged. Unfortunately, Jagex didn't increment the version number for these changes, and this would have in fact caused problems at the time: depending on when you first downloaded the cache, you might get an old or a new variant of the group. Users with the old variant of the group wouldn't receive updates. I can actually see this in the data I imported (e.g. some newer versions of the cache containing older variants of a colliding group, with others containing the new variant and no particular pattern to it). My suspicion is that Jagex only incremented the version number if they detected a change in the CRC-32 checksum, rather than comparing the uncompressed content (or a secure hash of it) directly. Luckily I haven't found any collisions in OSRS caches yet, so hopefully they've fixed it there. At the moment, the archive doesn't deal with collisions particularly well - for colliding groups, it prefers the first variant imported. I'm thinking of ways to improve this, but it is tricky - deciding which copy of the file is correct for a particular version of the cache probably can't be automated. What I might end up doing is showing a list of colliding groups in the web interface and punting the problem to the users.

A complete backup of the database is taken daily and made available over rsync/HTTP (rsync://archive.openrs2.org/pub/db/, Index of /pub/db/), in case anyone would like to maintain a mirror of the whole thing. This is much more efficient than downloading all the caches/keysets individually. The software you need to export caches from a mirrored copy is available at openrs2/archive at master - openrs2 - Gitea.

I might also allow read-only public access to the underlying Postgres database in the future, but I've got to think a bit carefully about security and this would be subject to nobody abusing it (e.g. deliberately DoSing it with inefficient queries). This would allow complex queries not supported by the web interface (some data isn't exposed yet) and replication.

I'll also be maintaining an archive of miscellaneous stuff (e.g. RS2 beta/RSC caches) here: Index of /pub/. It's also available over rsync. It's a little bare at the moment, but more files will be added over time.

So far in the main part of the archive I've imported OSRS/RS2 caches and keys present in Displee's/Polar's archives (so a lot of credit is due to them for doing such a good job of archiving stuff), as well as a few other sources. I'm still in the process of importing data (e.g. I haven't done any RS3 caches yet, but will soon, and I've got lots of old backup disks I need to sort through as well. I'll also trawl through old forum threads at some point too.)

I'm very keen on receiving any old copies of the cache (unless it's bit for bit identical to a copy in Displee's/Polar's archive - if you're not sure, send it anyway, as different copies might have slightly different sets of available groups) or XTEA keys anyone might have hanging around. For XTEA keys I don't need to know the region IDs, revisions, or whether they're valid - I'd prefer just to receive a massive dump of all candidate keys you have, as the service automatically handles verification across a wide range of caches without trusting region IDs. Ideally the caches should be unmodified, and it'd be nice to know the build number or approximate timestamp (but this is not a requirement). Send me a PM on the forum or DM on Discord with a download link if you do have something to contribute.

By pooling all our data in one place, the entire community will benefit from more complete caches and XTEA keysets, so I'm really hoping that people will see that and share rather than be selfish. For example, Sino was able to give me some XTEA keys for 550 I didn't have already, and in return my service had a few not present in the file he gave to me, so we both benefited from sharing keys.