October 17 2021
Technical Deep Dive on Object Lock, Ransomware Protection, and Immutable Backups in Retrospect Backup
Ransomware is a huge global threat to businesses around the world. The problem for companies is that their storage is always connected with full access for admins. When ransomware gets the administrative credentials, it has full access too. There is no policy to say that no one, not even the administrator, can change this file for a set amount of time.
Cloud Object Lock does just that. Because cloud storage providers like Amazon S3 control the API, they can add features like Object Lock. This lock is a retention policy for a specific version of a file that is locked from changes from every user, including the administrator. You can think of this as a virtual air-gap in the cloud because there is no way, barring to close the account, to delete that file before the retention date is passed.
Retrospect was one of the first data protection solutions to add ransomware protection using immutable backups: “Retrospect Backup 18: Ransomware Protection”. Retrospect utilizes Object Lock technology in major cloud storage providers to set a retention policy for cloud backups to ensure no one, not even the root user, can delete them during the retention window.
Retrospect User Interface
Creating an immutable backup set with Retrospect Backup is easy. There is a single checkbox in the user interface to enable it and a number of days to specify:
However, there is a lot of functionality underneath that checkbox to create immutable backups. Let’s dive into the technical details.
Forever-Incremental Backup’s Rolling Window
Retrospect Backup uses ProactiveAI for policy-driven scheduling and forever-incremental backup technology to minimize backup sizes while ensuring a point-in-time restore. The first backup is a full backup and every subsequent backup is called an incremental backup. Those incremental backups depend on previous backups. If a file doesn’t change, it doesn’t get backed up again.
Ordinarily, this workflow is a fantastic combination of minimizing storage while providing a backup that can perfectly recreate a point-in-time snapshot of the volume being protected. But that changes if you’re concerned the previous backups might be deleted. If a file is no longer locked, it can be deleted maliciously. Retrospect Backup needs to create backups where any backup within the rolling window of immutability are fully contained point-in-time snapshots of the volume.
Retrospect Backup accounts for the rolling window in two ways:
- File Matching: Retrospect adjusts its file matching to take into account retention policy for a given backed up file. A file that is outside of the retention policy is no longer considered to be backed up, and Retrospect will back it up again.
- ProactiveAI Scheduling: ProactiveAI determines the next date the script will run and backs up any file that will fall out of the retention policy by that date with forever-incremental backup, predicting into the future to ensure the file is protected at all times.
The consequence of this change is Retrospect will back up any file that is not protected in an immutable backup. Let’s say you back up every week and you set the retention policy for 4 weeks. Retrospect will back up every file every four weeks, regardless of whether it changed, because it needs to keep those files in the ransomware protection’s rolling window.
This process ensures that customers always have immutable backups with complete point-in-time restores. There is never a time when a backup depends on an out-of-policy file while preserving forever-incremental backups.
Cloud Storage Providers
There are two types of approaches from cloud storage providers: per-object policies and per-bucket policies. Per-object policies can be applied granularly to specific versions of an object at the time of creation, and they can vary within a bucket. Per-bucket policies are created for an entire bucket and are applied uniformly to every new version of any object in that bucket.
To compare with Retrospect:
- Per-Object Policy: You can create Backup Set A with an immutable retention policy of 2 days and Backup Set B with an immutable retention policy of 6 months in the same bucket, and the bucket does not need to have a bucket-wide policy.
- Per-Bucket Policy: You can only set a bucket-wide policy for immutable retention, so every new object is set to that retention period, regardless of what you have set in Retrospect.
Cloud storage providers with per-object policies are Amazon S3, Wasabi, Backblaze B2, MinIO, and Microsoft Azure Blob Storage (Preview - September 2021), while those with per-bucket policies are Google Cloud Storage and Microsoft Azure Blob Storage.
There are also different policy modes:
- Compliance Mode: The policy is time-based and enforced for every user, including administrators.
- Governance Mode: The policy is a legal hold, does not expire, and can be cancelled by a user with those permissions.
See Amazon S3 documentation for more information.
Retrospect Backup uses Retention Mode for its immutable backups. When you create an immutable backup, there is no permission level that will allow you to delete that version of the backup files. The root account cannot delete them. The only way to delete them is to close the account.
Because there is a way to ultimately delete the files, it’s important to use multi-factor authentication (MFA) for your root account on the cloud storage provider.
Retention Policy Dates
Let’s walk through the user interfaces for retention policy dates in the different cloud storage providers.
Amazon S3
Below is Amazon S3’s Retention Mode UI.
You’ll see it specifies the mode, the “Retain Until Date”, and the version of an object that you’re applying this to. Retrospect Backup does this step automatically when creating an immutable backup.
For Microsoft Azure Blob Storage and Google Cloud Storage, you will need to create the retention policy manually because they only support per-bucket policies.
Click on any object and scroll down on “Properties” to “Object Lock retention”.
Wasabi
Click on any object and “File Details” appears.
Backblaze B2
Click on any object’s blue link and “Details” appears.
Cyberduck
Select any object and click “Info” then “Metadata”.
Microsoft Azure Blob Storage
For a container, select “Access Policy”. Note: Per-object (blob) version locking in preview (September 2021)
Google Cloud Storage
When viewing a list of files, see the “Retention expiration date” column.
Viewing and Deleting Versions
One important nuance is how to view versions of a file. Only Amazon S3 and Cyberduck show versions. See below. Other interfaces choose to display a simplified version of the actual underlying content while preventing you from taking certain actions, like deletion.
One underlying feature is a delete marker. When you delete a object in a versioned bucket on Amazon S3, the file is not deleted. You are adding a delete marker as the next version of that file, and Amazon S3 understands it should not display that in the interface without “Show Versions” enabled.
Let’s look at the difference between deleting an object (“delete”) and deleting an object version (“permanently delete”):
In Wasabi or Backblaze, you don’t see versions, even though they are there for buckets with Object Lock enabled. Wasabi won’t let you delete files through their interface, but if an attacker added a delete marker to your file using an API, the file would no longer show up in Wasabi. You would have to use Cyberduck or other API to see that the locked files were indeed still there.
Video Walkthrough
I recorded a detailed video of the use cases and step-by-step walkthroughs on both Windows and Mac platforms as well as this technical deep dive into how Retrospect’s ransomware protection works and how Object Lock is implemented across cloud storage providers.
Note that much of this content is also available on the Retrospect website, under Ransomware Protection and Technical Deep Dive on Ransomware Protection, Object Lock, and Immutable Backups Ransomware. It took me a bit to compile, but the broad overview and then technical details have really helped clarify people’s understanding of ransomware, Object Lock, and Immutable backups–both inside the company and for our partners and customers.
