kyuubi/ResoLocalStorage

Watch

Fork 0

Shared asset format: store each asset once #5

New issue

Open

opened 2026-09-25 12:30:48 +02:00 by kyuubi · 2 comments

kyuubi commented

2026-09-25 12:30:48 +02:00

Owner

An alternative to self-contained .resonitepackage files: each asset is stored once, no matter how many items or world versions use it, and a normal package can be rebuilt without Resonite or the mod.

Format

<name>.rlsitem: a .resonitepackage without the assets its records list in assetManifest. It keeps the records, the main object/world data and the thumbnails.
.assets/ in the storage folder: Assets/<signature>, Metadata/<signature>.<type>, Variants/<signature>/<variant>. That's the layout inside a package, unzipped and shared.
Rebuild rule: copy every entry of the .rlsitem, then add each manifest signature's files from .assets/ that it doesn't already contain.
Only SHA-256 signatures whose content matches go to the store; anything else stays in the item file.

Plan

Setting ShareAssets (default off): new saves become .rlsitem; existing files keep their format, including when a world is saved over
Split a written package into item file + store, under a lock shared with the clean-up
Load items and worlds from item file + store; cloud assets still fill Resonite's cache, nothing of Resonite's points into .assets/
Clean-up at start: store files no .rlsitem refers to (trash included) move to .trash/
tools/rebuild-package.py: standard library only, one file or a whole folder, verifies SHA-256
Tests: split/rebuild round trip (C# and Python), dedup, clean-up, loading from the store; round trip over the real packages
README, version 0.4.0, deploy (6b39a8e, hot-loaded into the running game, no errors)
In game: save an item and a world with ShareAssets on, spawn/open them, save the world over

Evidence

52 tests: 51 pass, the opt-in real-data round trip skips by default.
Real-data round trip (RLS_ROUNDTRIP_DIR = the storage folder): all 6 packages come back entry for entry from both the C# and the Python rebuild. The 1.68 GB world becomes a 3.8 MB item file; "localstorage test" (a re-save of "(2)") and the clipboard image add nothing new to the store.

An alternative to self-contained `.resonitepackage` files: each asset is stored once, no matter how many items or world versions use it, and a normal package can be rebuilt without Resonite or the mod. ## Format - `<name>.rlsitem`: a `.resonitepackage` without the assets its records list in `assetManifest`. It keeps the records, the main object/world data and the thumbnails. - `.assets/` in the storage folder: `Assets/<signature>`, `Metadata/<signature>.<type>`, `Variants/<signature>/<variant>`. That's the layout inside a package, unzipped and shared. - **Rebuild rule:** copy every entry of the `.rlsitem`, then add each manifest signature's files from `.assets/` that it doesn't already contain. - Only SHA-256 signatures whose content matches go to the store; anything else stays in the item file. ## Plan - [x] Setting `ShareAssets` (default off): new saves become `.rlsitem`; existing files keep their format, including when a world is saved over - [x] Split a written package into item file + store, under a lock shared with the clean-up - [x] Load items and worlds from item file + store; cloud assets still fill Resonite's cache, nothing of Resonite's points into `.assets/` - [x] Clean-up at start: store files no `.rlsitem` refers to (trash included) move to `.trash/` - [x] `tools/rebuild-package.py`: standard library only, one file or a whole folder, verifies SHA-256 - [x] Tests: split/rebuild round trip (C# and Python), dedup, clean-up, loading from the store; round trip over the real packages - [x] README, version 0.4.0, deploy (6b39a8e, hot-loaded into the running game, no errors) - [ ] In game: save an item and a world with `ShareAssets` on, spawn/open them, save the world over ## Evidence - 52 tests: 51 pass, the opt-in real-data round trip skips by default. - Real-data round trip (`RLS_ROUNDTRIP_DIR` = the storage folder): all 6 packages come back entry for entry from both the C# and the Python rebuild. The 1.68 GB world becomes a 3.8 MB item file; "localstorage test" (a re-save of "(2)") and the clipboard image add nothing new to the store.

kyuubi referenced this issue from a commit

2026-09-25 12:37:31 +02:00

Add shared assets: store each asset once, rebuild packages offline

kyuubi referenced this issue from a commit

2026-09-25 12:45:44 +02:00

Drop a deleted shared item's assets from the store

kyuubi commented

2026-09-25 12:45:49 +02:00

Author

Owner

In game: saving and loading shared items works. Found: assets of deleted items never left .assets/, because the clean-up counted items in .trash/ as users. Getting them out took emptying the trash by hand, a restart, and a second trip through the trash.

Changed in 0.4.1 (f7cb2ed): deleting a shared item turns it into a full .resonitepackage in the trash, then the store deletes what no remaining item needs. The same pass runs at startup, after every trash move and after saving over a shared world. Items that can't be rebuilt stay and keep their assets; an unreadable item stops the pass before anything is deleted.

Verified:

54 tests pass: tidy-up converts and drops assets, keeps assets of items it can't rebuild, and stops on an unreadable item.
A dry run of rebuild-package.py over the 6 real trashed items succeeded with all hashes checked.
The hot-loaded game logged "Turned 6 deleted items into packages in the trash, deleted 298 asset store files". The store went from 348 MB to 61 MB, which is exactly the 27 files of the live SM_Barrel.rlsitem. The 6 trash packages (1.29 GB) pass a zip integrity check.

Still open: save a world with ShareAssets on, open it, save over it.

**In game:** saving and loading shared items works. Found: assets of deleted items never left `.assets/`, because the clean-up counted items in `.trash/` as users. Getting them out took emptying the trash by hand, a restart, and a second trip through the trash. **Changed in 0.4.1 (f7cb2ed):** deleting a shared item turns it into a full `.resonitepackage` in the trash, then the store deletes what no remaining item needs. The same pass runs at startup, after every trash move and after saving over a shared world. Items that can't be rebuilt stay and keep their assets; an unreadable item stops the pass before anything is deleted. **Verified:** - 54 tests pass: tidy-up converts and drops assets, keeps assets of items it can't rebuild, and stops on an unreadable item. - A dry run of `rebuild-package.py` over the 6 real trashed items succeeded with all hashes checked. - The hot-loaded game logged "Turned 6 deleted items into packages in the trash, deleted 298 asset store files". The store went from 348 MB to 61 MB, which is exactly the 27 files of the live `SM_Barrel.rlsitem`. The 6 trash packages (1.29 GB) pass a zip integrity check. Still open: save a world with `ShareAssets` on, open it, save over it.

kyuubi referenced this issue from a commit

2026-09-25 12:53:44 +02:00

Delete for real and free shared assets through an index

kyuubi referenced this issue from a commit

2026-10-07 03:05:26 +02:00

Test rebuild-package.py on items deep in subfolders

kyuubi commented

2026-10-07 03:05:32 +02:00

Author

Owner

tools/rebuild-package.py checked against the real storage folder (3065 .rlsitem files, up to 6 folders below the root, 54 GB .assets). The storage folder was only read.

Folder mode on the root finds all 3065 items, none missed, none extra. Every item finds the one .assets at the root, and no two outputs share a path.
Folder mode on Fox V264/Gear (store four levels up): 57 of 57 rebuilt at their mirrored paths. Single-file mode from 6 folders down, run from another folder: works. Writing over an existing output without --force is refused, exit 1.
Every item rebuilt with the tool and checked, then deleted: 3065 passed, 0 failed, 160.8 GiB in 15 min. Each output matches SharedPackage.Rebuild entry for entry and byte for byte, keeps every entry of the item unchanged, and holds every manifest asset. All 75,823 Assets/ entries hash to their names.
Asset links in the object data: none use packdb; cloud assets keep their resdb link. 85,868 resdb links have their copy in the package. The 12 without one are a single .brson that the CloudSpawn avatars load from the cloud.
24,982 neosdb links in 1664 items have no copy, because vanilla PackageCreator.CollectAssets only packs resdb and local. Resonite can't load them anyway: SkyFrost.Base.GatherJob throws "Unsupported scheme" for anything but http(s), ftp and resdb. Nothing is lost.

The deep-subfolder cases are now in PythonToolRebuildsTheSamePackages (0db9817). Suite: 87 passed, 1 skipped; tool tests also pass on Python 3.8.20.

`tools/rebuild-package.py` checked against the real storage folder (3065 `.rlsitem` files, up to 6 folders below the root, 54 GB `.assets`). The storage folder was only read. - Folder mode on the root finds all 3065 items, none missed, none extra. Every item finds the one `.assets` at the root, and no two outputs share a path. - Folder mode on `Fox V264/Gear` (store four levels up): 57 of 57 rebuilt at their mirrored paths. Single-file mode from 6 folders down, run from another folder: works. Writing over an existing output without `--force` is refused, exit 1. - Every item rebuilt with the tool and checked, then deleted: **3065 passed, 0 failed**, 160.8 GiB in 15 min. Each output matches `SharedPackage.Rebuild` entry for entry and byte for byte, keeps every entry of the item unchanged, and holds every manifest asset. All 75,823 `Assets/` entries hash to their names. - Asset links in the object data: none use `packdb`; cloud assets keep their `resdb` link. 85,868 `resdb` links have their copy in the package. The 12 without one are a single `.brson` that the `CloudSpawn` avatars load from the cloud. - 24,982 `neosdb` links in 1664 items have no copy, because vanilla `PackageCreator.CollectAssets` only packs `resdb` and `local`. Resonite can't load them anyway: `SkyFrost.Base.GatherJob` throws "Unsupported scheme" for anything but http(s), ftp and `resdb`. Nothing is lost. The deep-subfolder cases are now in `PythonToolRebuildsTheSamePackages` (0db9817). Suite: 87 passed, 1 skipped; tool tests also pass on Python 3.8.20.

No labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set

Reference

kyuubi/ResoLocalStorage#5

No description provided.

Rows
Columns