Backups at Ludicrous Speed

[Renerabbitt]

 

---

I got Chief backups from 50 minutes down to 12 minutes. Chief devs, please don’t kill me.

First off, let me say this: Chief Architect devs, please don’t kill me. I’m sorry. This never would have happened if we had a simple tool to export toolbars.

With no toolbar export, the only real path I had was to create a backup. But my backups were taking forever, so naturally that sent me down a ridiculous rabbit hole.

The good news is that I went from backups taking around 50 minutes to backups now taking around 12 minutes.

That is the fun hook. Now here is the chaotic process that got me there.

After doing a bunch of monitoring, scripting, and testing, I discovered that when Chief boots up it creates a bunch of temporary directories inside the Windows temp folder. In managed mode, Chief is storing/encrypting managed resources. When you run a backup, Chief appears to decrypt/unpack those files into those temp directories. Once those are written to temp, Chief starts compiling the correct backup folder structure, caching/indexing that information, and then finally writing the backup zip.

At times, some managed resources also appear to pull textures from their original source locations on the hard drive. I have not fully figured out why or how to avoid that, but I know it is part of the process in my system. Because of that, I added an optional texture-folder redirect step as well, and that shaved another couple of minutes off my backup time.

For context, my original backup was around 24 to 25 GB.

A big part of my file bloat came from how my template/project system is built. I use an extra plan file where I store spec writing and Project Information headings so that I can quickly swap jurisdiction/reference information. That file had a bunch of materials in it that I did not actually need. So the first optimization was consolidating and reducing unnecessary materials in that warehouse/reference plan.

The next thing I started doing was exporting projects when I was done working on them, then reimporting them when needed instead of keeping everything active and bloated all the time.

The big asset step was building batch scripts to export textures from the Asset Manager. Then I used another script to identify secondary texture maps versus primary texture maps. By secondary maps, I mean things like opacity maps, bump maps, normal maps, AO maps, roughness maps, etc. My textures use a lot of these extra maps because I care about realism, but I also have thousands of textures in my libraries, so this adds up fast.

Back in X17, I batch-resized a lot of those secondary textures and then had to automate replacing them in Asset Manager. That was another whole script. Then in X18, since Chief allows WebP files, I did another full pass and converted a ton of PNGs to WebP.

All in, I reduced my backup from about 25 GB down to about 12 GB.

That alone was a huge improvement, but even at 12 GB, backups were still taking 30 to 45 minutes.

Then came the next round of craziness.

Once I understood more of the backup process, I built a set of PowerShell scripts that temporarily redirect three things to a RAM disk:

1. Chief’s managed resources folder

2. Chief’s temp directories

3. My external textures folder

Then I also write the backup output itself to the RAM disk.

The reason this matters is that Chief’s backup process is not just writing one big file. It is touching thousands and thousands of individual files. It is reading them, validating them, decrypting/unpacking them, indexing/caching them, restructuring them, and then writing them again. Sequential write speed alone is not the whole story here. A lot of this comes down to small-file read/write behavior, latency, and single-thread processing.

This is also why I started paying closer attention after being on calls with people running Apple Silicon Macs and seeing how much faster some of their systems were in certain Chief operations. I originally thought I just needed a drive with a higher sequential write speed, but this kind of workload is much more about small-file performance, latency, and how quickly the system can chew through tons of tiny file operations.

So, for my test, everything possible gets temporarily shoved into RAM:

• Managed resources: RAM disk

• Temp backup folders: RAM disk

• Textures: RAM disk

• Backup output: RAM disk

If I skip the texture redirect, I only need a RAM disk roughly three times the size of the backup. If I include textures, I need closer to four times the backup size. In my case, I have 128 GB of RAM, so creating a 60 GB RAM disk is not a big deal.

With all of that in place, my 12 GB backup is now taking about 12 minutes, so roughly one minute per gigabyte.

To be clear, I am not recommending that anyone casually do this. This is mostly just information for people who find this kind of thing interesting.

The bigger takeaway is what this taught me about building a very fast Windows machine for Chief.

If I were building a Chief-focused Windows workstation today, the things I would care about most are:

• Strong single-thread CPU performance

• A good amount of RAM, ideally 64 GB or more if you do large projects or multitask heavily

• An Nvidia GPU that supports current RTX/DLSS features

• Very fast NVMe storage, with a focus on latency and small-file behavior, not just peak sequential numbers

• Potentially RAID 0 for NVMe drives if you are comfortable with the risk and have a good backup strategy

For GPU, Nvidia’s current public DLSS branding is DLSS 4, not DLSS 4.5, and it is tied heavily to RTX 50-series features like Multi Frame Generation. (NVIDIA)

For storage, the newer PCIe 5 drives are getting pretty wild. Samsung’s 9100 PRO is listed up to 14,800 MB/s sequential read, and WD’s SN8100 is listed up to 14,900 MB/s sequential read. (Samsung) But again, for this kind of Chief backup workload, the headline sequential number is not the whole story.

I mistakenly thought pure sequential speed was going to get me closer to the performance I was seeing elsewhere, but the real issue is all the individual little file operations. That is why I am also moving toward a RAID 0 NVMe setup. The idea is to split the workload across drives and take better advantage of the individual NVMe controllers/caches when Chief is hammering thousands of tiny reads and writes.

One note on RAID 0: it is seen as one drive. So if you need 4 TB of usable storage, you use two 2 TB drives. Do not do what I did and buy one 4 TB drive, then realize you need a second 4 TB drive and accidentally build yourself an 8 TB monster you may not actually need.

If upgrading an existing system, my current thought process would be:

Move large files to an external drive or cloud storage, set cloud folders to online-only where possible, shrink the active system footprint, create a backup image, build the RAID, restore the image, then bring cloud/external files back as needed.

For RAM, I think 32 GB is okay, 64 GB is great, and 128 GB is useful if you are doing ridiculous stuff like this while also running Photoshop, Chrome with a pile of tabs, OBS, AI camera software, ChatGPT, Blender, Substance Painter, etc.

For CPU, single-core speed matters a lot in Chief. I am personally sticking with my current CPU for now because I run a ton of programs concurrently and I benefit from the larger core count and cache, but if all you mostly do is Chief, Chrome, and email, then single-thread speed should be a major priority.

Anyway, hopefully this is at least an interesting write-up. I do not recommend going full mad scientist unless you like breaking things and testing them. But I do think the broader lesson is useful:

Chief backup speed is not just about “fast storage.” It is about how Chief processes thousands of managed resources, textures, temp files, and small file operations.

Feel free to ask questions.

 

Edited Friday at 09:49 PM by Renerabbitt