Contents
Apple knows better
It is sad that Apple doesn’t provide any backup migration wizard to move backups from locally attached disk to network share. The only option Apple provides is to start backup from the scratch on network share loosing all your backup history. This makes your network Time Machine backup absolutely useless since you can’t search for the file that you accidentally deleted a few years ago.
This instruction will help you to migrate your locally attached disk with Time Machine backups and whole backup history to Time Machine compatible network share.
There are different ways to do the same. All the ways have their pros and cons and I would like to share my observations and probably save a little bit of your time.
Configuration makes sense
This instruction is based on the configuration I have. It could not work on different configuration and could require additional tuning.
My configuration:
- MacBook Pro (Retina, 13-inch, Early 2013), 512GB SSD, macOS Mojave 10.14.6.
- (source disk) External USB3 backup magnetic disk 1TB with backups from 2013 year,
HFS+ case-insensitive, not encrypted. - (target disk) External USB3 backup magnetic disk 2TB with just one GPT partition, formatted with ext4, empty, needs to setup as new target.
- NAS with attached target disk running on ODROID C2 with Gentoo Linux.
NAS Samba v4.12.9 config:
[global]
map to guest = Bad User
disable netbios = yes
smb ports = 445
min protocol = SMB2
vfs objects = catia fruit streams_xattr
[media]
path = /media
read only = no
guest ok = yes
[TimeMachine]
path = /media/TimeMachine
read only = no
guest ok = yes
fruit:time machine = yes
NAS Avahi config:
<?xml version="1.0" standalone='no'?><!--*-nxml-*-->
<!DOCTYPE service-group SYSTEM "avahi-service.dtd">
<service-group>
<name replace-wildcards="yes">%h</name>
<service>
<type>_smb._tcp</type>
<port>445</port>
</service>
<service>
<type>_device-info._tcp</type>
<port>0</port>
<txt-record>model=Xserve</txt-record>
</service>
<service>
<type>_adisk._tcp</type>
<txt-record>sys=waMa=0,adVF=0x100</txt-record>
<txt-record>dk0=adVN=TimeMachine,adVF=0x82</txt-record>
</service>
</service-group>
For more details what these _adisk._tcp flags mean please reference to:
http://netatalk.sourceforge.net/wiki/index.php/Bonjour_record_adisk_adVF_values
This configuration makes Time Machine share discoverable and easy mountable using Finder in macOS and in Time Machine configuration tool.
I was not able to get exfat volume to work as Time Machine backup due to the lack of extended attributes support on exfat volumes. The fastest and most stable filesystem with extended attributes on Linux is ext4. This will make things easier on Linux side and… more complex on macOS side.
Recommendations
- Use wired connection for both NAS and your laptop to speedup backup migration in case if you would like to migrate over the network.
- Consider attaching both source backup disk and target backup disk (from NAS) to mac and do initial local copy (much faster write speeds if both disks are USB3). In my case local copy is around 90Mb/sec vs 12Mb/sec over network.
- Use larger sparse bundle band size to improve backup performance, but this could be a bad advice if your are planning to use encrypted sparse bundle.
- Consider having sparse bundle size the same as backup disk size. You won’t need to resize sparse bundle disk in that case and that will save time.
Step 1: Create network sparse bundle
Firstly you will need to create compatible sparse bundle file on network drive while your target backup disk is connected to hand made time capsule server. Later you can temporarily disconnect it and connect to your mac to do an initial copy.
Method 1: Let macOS to do that for you
You can create it manually but the easies way is to let Time Machine to create it for you.
Just open Time Machine and add new backup disk on network share but do not replace existing backup disk. Answer on question if backup need to be encrypted. If username will be asked, pick Guest (mine Samba doesn’t have enable auth for simplicity).
Once disk is added Time Machine will spend some time on creation of target sparse bundle file and then will start doing initial copy. Cancel the backup once Time Machine will start to show file copying progress. We will overwrite it anyway, so don’t need to complete initial copy.
Uncheck checkbox that enabled automatic backup. It could ruin the process! We will enable it back once target disk will be ready.
After that step is completed you should have sparse bundle {ComputerName}.sparsebundle
on your Time Machine network share.
Method 2: Manually
You can create sparse bundle image using Disk Utility on network share: {ComputerName}.sparsebundle
.
You can get know computer name by running: scutil --get ComputerName
There will be some manipulations needed with sparse bundle plist
files to make image be properly recognizable by Time Machine. Read more here: https://gist.github.com/martian111/e0d9885004eb56fd6abf3d1ba7671737
You can also create encrypted sparse bundle if you want.
I did not follow this method so there are no more details in this section.
(optional) Change band size on sparse bundle
This is OPTIONAL procedure but can significantly improve backup performance.
You can find more details in this article: https://edoardofederici.com/improve-time-machine-performance/
Below I would outline shorter version that doesn’t require too much scripting in Terminal.
If you are planning to use encrypted sparsed bundle then you should probably avoid changing band size as there are some reports that it could make performance even worse: https://gist.github.com/SebastianJ/b3e9af8a641df1dc73c0
Default sparse bundle band size is 8MB = 16384 blocks. Too small or too big band size could slow down the backup. Choose band size depending on how fast your network connection, how fast your target network and how big the hard disk you are backing up.
Command line band size for hdiutil
is specified in 512 byte blocks, so 8MB = (8 * 1024 * 1024 / 512) blocks = 16384 blocks
.
I played a little bit and found that local copying speed to sparse bundle with 32MB band size is twice higher than with 8MB band size (100Mb/s vs 50Mb/s).
Since I would like make target sparse bundle to be encrypted I decided to keep band size as it is and skip band size conversion.
By the way, here are migration steps:
- Unmount sparse bundle image using Disk Utility if already mounted (do not eject).
- Mount Time Machine volume from NAS.
- Convert band size using
hdiutil
command below. - If you will choose to use encryption then password will be asked.
- Move original sparse bundle somewhere as backup.
- Rename new sparse bundle to have the same filename as original.
Conversion without encryption:
hdiutil convert /Volumes/TimeMachine/{BundleName}.sparsebundle -format UDSB -tgtimagekey sparse-band-size={BandSize} -o /Volumes/TimeMachine/{TempBundleName}.sparsebundle
Conversion with encryption:
# Encryption could be AES-256 or AES-128
hdiutil convert /Volumes/TimeMachine/{BundleName}.sparsebundle -format UDSB -tgtimagekey sparse-band-size={BandSize} -encryption {Encryption} -o /Volumes/TimeMachine/{TempBundleName}.sparsebundle
There will be some manipulations needed with sparse bundle plist
files to make image be properly recognizable by Time Machine. Read more here: https://gist.github.com/martian111/e0d9885004eb56fd6abf3d1ba7671737
Step 2: Mount sparse bundle
Method 1: From local drive (complex, fast)
Local cloning should be the fastest way and faster than any network copying.
There is a special driver needed for macOS to be able to write to Linux ext4 formatted volume, that I have on Time Machine backup volume on the disk, connected to the NAS.
There is proprietary driver Paragon ExtFS that can read/write ext2/3/4 volumes. You can purchase it or install trial version (10 days should be more than enough): https://www.paragon-software.com/us/home/extfs-mac/
There is also fuse2fs application from e2fsprogs package that can do almost same and be installed using HomeBrew (did not test this path).
Steps:
- Install Paragon ExtFS and reboot.
- Detach hard drive that was connected to NAS and attach it your Mac. New disk should appear in Disk Utility and, if ext4 driver is installed, ext4 partition with Time Machine backup should be auto mounted and available in Finder.
- Open your target Time Machine volume and open target sparse bundle on it to mount it (the one that was previously created).
- Now you should have both source disk and target sparse bundle mounted and you can copy the contents between them.
Method 2: From network drive (easy, slow)
If you can connect your Mac over the wire or if your wireless network is fast and your Mac supports 802.11ac then it could be wise and convenient to do migration over the network: from backup disk connected to Mac to network drive connected to the NAS.
If you are not in rush and it is okay to have your Mac copying in background files for a few days then it could be a good option too, especially, if second method doesn’t work for some reason or looks too complex.
Unfortunately, my Mac is considered “vintage” these days (in terms of Apple) and the max network bandwidth is just around 200Mbit/s between NAS (wired) and Mac (wireless) using default wireless router provided by Verizon. With Samba and sparse bundle incapsulation total bandwidth averages 100Mbit/s. Copying 1TB of data as it is on 100Mbit/s should take around 20-30 hours. The longer it runs, the higher the chance that something will go wrong, so I decided to do faster local drive cloning.
Step 3: Copying contents
Method 1: Copy using disk cloning (fast, unstable)
There is an easier and faster way – clone external backup disk to sparse bundle disk using dd
as it is. No need to care about case sensitivity or ownership flag – your sparse bundle will have exactly the same byte to byte state as your external backup disk. Your sparse bundle volume must have the same or a little bit larger size than source disk. You can resize (grow) HFS+ partition on sparse bundle disk after but it is not fast.
Steps:
- Make sure that both source backup disk and target sparse bundle disk are visible in Disk Utility.
- Unmount partitions (but not eject volumes) on source disk and target image.
- Enable “Terminal” to have full disk access:
- Open “System Preferences”
- Then “Security and Privacy”
- Then “Privacy”
- Then “Full Disk Access”
- Press “+” and add “Terminal” application
- Quit “Terminal” if it is already started and start “Terminal” application.
- Start copy process
- Run
sudo dd if=/dev/r{SourceDisk} of=/dev/r{DestinationDisk} bs=2m
- Example with macOS dd:
sudo dd if=/dev/rdiskA of=/dev/rdiskB bs=2m
- Use
r
prefix, this will make process faster - Use disk ID, not partition ID
- Grab identifiers from
Disk Utility
or usingdiskutil list
.rdiskA
must be a physical disk where you are copying formrdiskB
must be a sparse bundle disk located on physical disk where you are copying to
- Block size
2m
should improve performance. You could experiment with different values. Terminate the process withControl + C
, unmount volume again (otherwise macOS will auto mount it) and test the speed. - Press
Control + T
to see the progress and the speed. - If you have HomeBrew and
coreutils
installed then you could use GNU dd that can automatically report progress- Run
sudo gdd if=/dev/r{SourceDisk} of=/dev/r{DestinationDisk} bs=2M status=progress
- Example:
sudo gdd if=/dev/rdiskA of=/dev/rdiskB bs=2M status=progress
- Block size units have slightly different format (
M
instead ofm
).
- Run
- Run
- Rename HFS+ volume on sparse bundle disk to
Time Machine Backups
after
(not sure if it is required, but macOS created sparse bundles have this name
for backup volume).- Open Disk Utility
- Select partition inside sparse bundle image
- Right click -> Rename
- Type the name
Time Machine Backups
- Sparse bundle won’t have exactly the same size as source disk, so need to fix that:
- You could skip this step if source backup disk and sparse bundle have the same or about the same size.
- Repair sparse bundle partition map (this will grow partition size in GPT)
- Run
sudo diskutil repairDisk {SparseBundleDiskID}
- Example:
sudo diskutil repairDisk diskX
- Run
- Grow backup HFS+ volume in sparse bundle (grow volume size to partition size)
- Run
sudo diskutil resizeVolume {TimeMachineBackupPartID} 0
- Example:
sudo diskutil resizeVolume diskXs2 0
- Resize will also trigger HFS+ file system check and it is good to confirm that partition after this hardcore migration is still not damaged
- This procedure will take a lot of time because file system check is slow.
- Run
Paragon ExtFS did not show perfect stability for me. It crashed macOS a few times until succeded.
The only time the process has succeded was with these steps and in ~3.5 hours:
- fresh rebooted macOS
- all applications besides Terminal were closed
- default band size was used
gdd
instead ofdd
- disabled macOS sleep using Amphetamine app
Let me know if you will find what exactly can crash the copying process, so I can share more details here for others.
Example:
Artems-MacBook-Pro:~ $ diskutil list
/dev/disk0 (internal, physical):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *500.3 GB disk0
1: EFI EFI 209.7 MB disk0s1
2: Apple_APFS Container disk1 500.1 GB disk0s2
/dev/disk1 (synthesized):
#: TYPE NAME SIZE IDENTIFIER
0: APFS Container Scheme - +500.1 GB disk1
Physical Store disk0s2
1: APFS Volume Macintosh HD 373.4 GB disk1s1
2: APFS Volume Preboot 47.2 MB disk1s2
3: APFS Volume Recovery 510.2 MB disk1s3
4: APFS Volume VM 1.1 GB disk1s4
/dev/disk2 (external, physical):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *1.0 TB disk2
1: EFI EFI 209.7 MB disk2s1
2: Apple_HFS Sormy's Backup 999.9 GB disk2s2
/dev/disk3 (external, physical):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme *2.0 TB disk3
1: Linux Filesystem tm-backup 2.0 TB disk3s1
/dev/disk4 (disk image):
#: TYPE NAME SIZE IDENTIFIER
0: GUID_partition_scheme +1.9 TB disk4
1: EFI EFI 209.7 MB disk4s1
2: Apple_HFS Sormy's Backup 999.9 GB disk4s2
Artems-MacBook-Pro:~ $ sudo gdd if=/dev/rdisk2 of=/dev/rdisk4 bs=2M status=progress
1000167440384 bytes (1.0 TB, 931 GiB) copied, 11989 s, 83.4 MB/s
476934+1 records in
476934+1 records out
1000204885504 bytes (1.0 TB, 932 GiB) copied, 11989.9 s, 83.4 MB/s
Method 2: Copy using Finder (slow, stable)
Your external backup disk most-likely has case-insensitive HFS+ and the sparse bundle will have by default case-sensitive HFS+ so copying Backups.backupdb
using Finder won’t work by default.
Steps:
- Reformat sparse bundle volume to case-insensitive HFS+ (same filesystem as external backup disk) using Disk Utility.
- Manually set ownership flag on it using Terminal and the remount:
sudo diskutil enableOwnership {PartID}
. - Only after that you can copy
Backups.backupdb
using Finder from one volume to another volume.
But it will take EXTREME amount of time. In my case even whole night was not enough to build just list of files to copy. I can only imagine how long will it take to also copy files. I gave up and decided to go forward with local cloning solution.
Step 4: Fix permissions
Eject NAS drive from your Mac and connect back to NAS if you chose option to locally copy files.
You will most-likely need to fix ownership on files to make them readable from network.
Run something like below on your NAS:
# in my example disk is mounted to /media and Samba is configured to use nobody
sudo chown -R nobody:nobody /media/TimeMachine
Step 5: Initiate new backup
Initiate backup using Time Machine menu on macOS.
Mac could probably ask about “The identity of the backup disk has changed since the previous backup”. This is fine for initial backup because this is exactly what we did. Click “Use This Disk” to continue.
The initial backup should be successfully completed.
You could enable automatic backups back after.
I would recommend to use both backup solutions (local disk and network disk) for some time before you could confirm that network backup works well in terms of stability and performance. After some time you can remove old local disk backup target from Time Machine and enjoy wireless backups.
Conclusing
I hope this article will save some of your time. Let me know please if there is anything I can fix to make this article better.
Thank you and happy migration š
Artem,
This is labtech from the HDDGuru forum.
Thanks for your research.
It was useful in helping with a Time Machine Sparsebundle from a Seagate Central NAS unit using the Paragon ExtFS Driver.
Although the Data volume containing the Time Machine Sparaebundle bands was cloned with 0 errors, when trying to mount the Time Machine.backupbundle using DiskImageMounter, Mac gives an error: “The following images couldn’t be opened”; “no mountable file systems”.
Does it sound like the Time Machine was not creating the backup bands correctly? Or is there a bug somewhere?
Many thanks.
Was copying performed on macOS?
Was sparse bundle mountable before copying?
Is sparse bundle mountable after copying?
How did you perform copying?
Did you try to mount ExtFS on macOS using mac Disk Utility? or using Paragon Mounter?
I don’t remember if macOS recognizes locally attached time machine backup on ExtFS volume.
You should mount it on NAS and expose over network and try to mount.
Thanks for the modern tutorial!
Do you have any changes from the last year?
Iām experimenting with new time machine from fresh install of Mac OS Monterey and using a FreeNAS/truenas box to backup over 10GBE wired connection.
I just purchased new MacBook Pro and there will be likely a new setup from the scratch. But likely not migration this time.