Time Machine backup disk migration to network drive

Published on Author Artem Butusov2 Comments

Apple knows better

It is sad that Apple doesn’t provide any backup migration wizard to move backups from locally attached disk to network share. The only option Apple provides is to start backup from the scratch on network share loosing all your backup history. This makes your network Time Machine backup absolutely useless since you can’t search for the file that you accidentally deleted a few years ago.

This instruction will help you to migrate your locally attached disk with Time Machine backups and whole backup history to Time Machine compatible network share.

There are different ways to do the same. All the ways have their pros and cons and I would like to share my observations and probably save a little bit of your time.

Configuration makes sense

This instruction is based on the configuration I have. If youth is different then you will probably need to slightly alter the plan.

My configuration:

  • MacBook Pro (Retina, 13-inch, Early 2013), 512GB SSD, macOS Mojave 10.14.6.
  • External USB3 backup magnetic disk 1TB with backups from 2013 year,
    HFS+ case-insensitive, not encrypted.
  • NAS with time machine Samba share pointing to another external USB3 magnetic
    disk formatted with ext4 (running on Odroid-XU4 with Arch Linux).

NAS Samba v4.10.10 config:

[global]

map to guest = Bad User

disable netbios = yes
smb ports = 445
min protocol = SMB2

vfs objects = catia fruit streams_xattr

[media]

path = /media
read only = no
guest ok = yes

[TimeMachine]

path = /media/TimeMachine
read only = no
guest ok = yes
fruit:time machine = yes

NAS Avahi config:

<?xml version="1.0" standalone='no'?><!--*-nxml-*-->
<!DOCTYPE service-group SYSTEM "avahi-service.dtd">
<service-group>
  <name replace-wildcards="yes">%h</name>
  <service>
    <type>_smb._tcp</type>
    <port>445</port>
  </service>
  <service>
    <type>_device-info._tcp</type>
    <port>0</port>
    <txt-record>model=Xserve</txt-record>
  </service>
</service-group>

This configuration makes Time Machine share discoverable and easy mountable using Finder in macOS.

The NAS disk I have is already partitioned and formatted using standard Linux tools. It has two GPT partitions:

  • 1TB for Time Machine backup, filesystem: ext4
  • remaining for files, filesystem: exfat

I was not able to get exfat volume to work as Time Machine backup due to the lack of extended attributes support on exfat volumes. The fastest and most stable filesystem with extended attributes on Linux is ext4. This will make things easier on Linux side and… more complex on macOS side.

I have also this option set but not sure if it is needed these days:

defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1

Recommendations

  • Use wired connection for both NAS and your laptop to speedup backup migration in case if you would like to migrate over the network.
  • Consider attaching both source backup disk and target backup disk (from NAS) to mac and do initial local copy (much faster write speeds if both disks are USB3). In my case local copy is around 90Mb/sec vs 12Mb/sec over network.
  • Use larger sparse bundle band size to improve backup performance, but this could be a bad advice if your are planning to use encrypted sparse bundle.
  • Consider having sparse bundle size the same as backup disk size. You won’t need to resize sparse bundle disk in that case and that will save time.

Create network sparse bundle

Firstly you will need to create compatible sparse bundle file on network drive.

Method 1: Manually

You can create sparse bundle image using Disk Utility on network share: {ComputerName}.sparsebundle.

You can get know computer name by running: scutil --get ComputerName

There will be some manipulations needed with sparse bundle plist files to make image be properly recognizable by Time Machine. Read more here: https://gist.github.com/martian111/e0d9885004eb56fd6abf3d1ba7671737

You can also create encrypted sparse bundle if you want.

I did not follow this method so there are no more details in this section.

Method 2: Let macOS to do that for you

You can create it manually but the easies way is to let Time Machine to create it for you. Just open Time Machine, add new backup disk on network share, do not replace existing backup disk, initiate backup manually and don’t forget to disable automatic backups. You can even choose option to encrypt backups. Cancel the backup once Time Machine will start to show file copying progress.

After that step is completed you should have sparse bundle {ComputerName}.sparsebundle on your Time Machine network share.

Change band size on sparse bundle

This is OPTIONAL procedure but can significantly improve backup performance.

You can find more details in this article: https://edoardofederici.com/improve-time-machine-performance/

Below I would outline shorter version that doesn’t require too much scripting in Terminal.

If you are planning to use encrypted sparsed bundle then you should probably avoid changing band size as there are some reports that it could make performance even worse: https://gist.github.com/SebastianJ/b3e9af8a641df1dc73c0

Default sparse bundle band size is 8MB = 16384 blocks. Too small or too big band size could slow down the backup. Choose band size depending on how fast your network connection, how fast your target network and how big the hard disk you are backing up.

Command line band size for hdiutil is specified in 512 byte blocks, so 8MB = (8 * 1024 * 1024 / 512) blocks = 16384 blocks.

I played a little bit and found that local copying speed to sparse bundle with 32MB band size is twice higher than with 8MB band size (100Mb/s vs 50Mb/s).

Since I would like make target sparse bundle to be encrypted I decided to keep band size as it is and skip band size conversion.

By the way, here are migration steps:

  • Unmount sparse bundle image using Disk Utility if already mounted (do not eject).
  • Mount Time Machine volume from NAS.
  • Convert band size using hdiutil command below.
  • If you will choose to use encryption then password will be asked.
  • Move original sparse bundle somewhere as backup.
  • Rename new sparse bundle to have the same filename as original.

Conversion without encryption:

hdiutil convert /Volumes/TimeMachine/{BundleName}.sparsebundle -format UDSB -tgtimagekey sparse-band-size={BandSize} -o /Volumes/TimeMachine/{TempBundleName}.sparsebundle

Conversion with encryption:

# Encryption could be AES-256 or AES-128
hdiutil convert /Volumes/TimeMachine/{BundleName}.sparsebundle -format UDSB -tgtimagekey sparse-band-size={BandSize} -encryption {Encryption} -o /Volumes/TimeMachine/{TempBundleName}.sparsebundle

There will be some manipulations needed with sparse bundle plist files to make image be properly recognizable by Time Machine. Read more here: https://gist.github.com/martian111/e0d9885004eb56fd6abf3d1ba7671737

Mount sparse bundle

Method 1: From network drive (easy, slow)

If you can connect your Mac over the wire or if your wireless network is fast and your Mac supports 802.11ac then it could be wise and convenient to do migration over the network: from backup disk connected to Mac to network drive connected to the NAS.

If you are not in rush and it is okay to have your Mac copying in background files for a few days then it could be a good option too, especially, if second method doesn’t work for some reason or looks too complex.

Unfortunately, my Mac is considered “vintage” these days (in terms of Apple) and the max network bandwidth is just around 200Mbit/s between NAS (wired) and Mac (wireless) using default wireless router provided by Verizon. With Samba and sparse bundle incapsulation total bandwidth averages 100Mbit/s. Copying 1TB of data as it is on 100Mbit/s should take around 20-30 hours. The longer it runs, the higher the chance that something will go wrong, so I decided to do faster local drive cloning.

Method 2: From local drive (complex, fast)

Local cloning should be the fastest way and faster than any network copying.

There is a special driver needed for macOS to be able to write to Linux ext4 formatted volume, that I have on Time Machine backup volume on the disk, connected to the NAS.

There is proprietary driver Paragon ExtFS that can read/write ext2/3/4 volumes. You can purchase it or install trial version (10 days should be more than enough): https://www.paragon-software.com/us/home/extfs-mac/

There is also fuse2fs application from e2fsprogs package that can do almost same and be installed using HomeBrew (did not test this path).

Steps:

  • Install Paragon ExtFS and reboot.
  • Detach hard drive that was connected to NAS and attach it your Mac. New disk should appear in Disk Utility and, if ext4 driver is installed, ext4 partition with Time Machine backup should be auto mounted and available in Finder.
  • Open your target Time Machine volume and open target sparse bundle on it to mount it (the one that was previously created).
  • Now you should have both source disk and target sparse bundle mounted and you can copy the contents between them.

Copying contents

Method 1: Copy using Finder (slow)

Your external backup disk most-likely has case-insensitive HFS+ and the sparse bundle will have by default case-sensitive HFS+ so copying Backups.backupdb using Finder won’t work by default.

Steps:

  • Reformat sparse bundle volume to case-insensitive HFS+ (same filesystem as external backup disk) using Disk Utility.
  • Manually set ownership flag on it using Terminal and the remount: sudo diskutil enableOwnership {PartID}.
  • Only after that you can copy Backups.backupdb using Finder from one volume to another volume.

But it will take EXTREME amount of time. In my case even whole night was not enough to build just list of files to copy. I can only imagine how long will it take to also copy files. I gave up and decided to go forward with local cloning solution.

Method 2: Copy using disk cloning (fast)

There is an easier and faster way – clone external backup disk to sparse bundle disk using dd as it is. No need to care about case sensitivity or ownership flag – your sparse bundle will have exactly the same byte to byte state as your external backup disk. Your sparse bundle volume must have the same or a little bit larger size than source disk. You can resize (grow) HFS+ partition on sparse bundle disk after but it is not fast.

Steps:

  • Make sure that both source backup disk and target sparse bundle disk are visible in Disk Utility.
  • Unmount partitions (but not eject volumes) on source disk and target image.
  • Enable “Terminal” to have full disk access:
    • Open “System Preferences”
    • Then “Security and Privacy”
    • Then “Privacy”
    • Then “Full Disk Access”
    • Press “+” and add “Terminal” application
  • Quit “Terminal” if it is already started and start “Terminal” application.
  • Start copy process
    • Run sudo dd if=/dev/r{SourceDisk} of=/dev/r{DestinationDisk} bs=2m
    • Example: sudo dd if=/dev/rdiskA of=/dev/rdiskB bs=1m
    • Use r prefix, this will make process faster
    • Use disk ID, not partition ID
    • Grab identifiers from Disk Utility or using diskutil list.
    • Block size 1m should improve performance. You could experiment with different values. Terminate the process with Control + C, unmount volume again (otherwise macOS will auto mount it) and test the speed.
    • Press Control + T to see the progress and the speed.
    • If you have HomeBrew and coreutils installed then you could use GNU dd that can automatically report progress
    • Run sudo gdd if=/dev/r{SourceDisk} of=/dev/r{DestinationDisk} bs=1M status=progress
    • Example:sudo gdd if=/dev/rdiskA of=/dev/rdiskB bs=1M status=progress
    • Block size units have slightly different format (M instead of m).
  • Rename HFS+ volume on sparse bundle disk to Time Machine Backups after
    (not sure if it is required, but macOS created sparse bundles have this name
    for backup volume).
  • Sparse bundle won’t have exactly the same size as source disk, so need to fix that:
    • You could skip this step if source backup disk and sparse bundle have the same or about the same size.
    • Repair sparse bundle partition map (this will grow partition size in GPT)
    • Run sudo diskutil repairDisk {SparseBundleDiskID}
    • Example: sudo diskutil repairDisk diskX
    • Grow backup HFS+ volume in sparse bundle (grow volume size to partition size)
    • Run sudo diskutil resizeVolume {TimeMachineBackupPartID} 0
    • Example: sudo diskutil resizeVolume diskXs2 0
    • Resize will also trigger HFS+ file system check and it is good to confirm that partition after this hardcore migration is still not damaged
    • This procedure will take a lot of time because file system check is slow.

Paragon ExtFS did not show perfect stability for me. It crashed macOS a few times until succeded.

The only time the process has succeded was with these steps and in ~5.5 hours:

  • fresh rebooted macOS
  • all applications besides Terminal were closed
  • 8MB band size was used (this slowed down copying process to 50MB/sec)
  • gdd instead of dd
  • disabled macOS sleep using Amphetamine app

Let me know if you will find what exactly can crash the copying process, so I can share more details here for others.

Testing network backup

Eject NAS drive from your Mac and connect back to NAS.

You will most-likely need to fix ownership on files to make them readable from network:

# in my example disk is mounted to /media and Samba is configured to use nobody
sudo chown -R nobody:nobody /media/TimeMachine

Initiate backup using Time Machine menu on macOS.

Mac could probably ask about “The identity of the backup disk has changed since the previous backup”. This is fine for initial backup because this is exactly what we did. Click “Use This Disk” to continue.

The initial backup should be successfully completed. You could enable automatic backups back after.

I would recommend to use both backup solutions (local disk and network disk) for some time before you could confirm that network backup works well in terms of stability and performance.

Conclusing

I hope this article will save some of your time. Let me know please if there is anything I can fix to make this article better.

Thank you and happy migration 😉

2 Responses to Time Machine backup disk migration to network drive

  1. Artem,

    This is labtech from the HDDGuru forum.

    Thanks for your research.
    It was useful in helping with a Time Machine Sparsebundle from a Seagate Central NAS unit using the Paragon ExtFS Driver.

    Although the Data volume containing the Time Machine Sparaebundle bands was cloned with 0 errors, when trying to mount the Time Machine.backupbundle using DiskImageMounter, Mac gives an error: “The following images couldn’t be opened”; “no mountable file systems”.

    Does it sound like the Time Machine was not creating the backup bands correctly? Or is there a bug somewhere?

    Many thanks.

    • Was copying performed on macOS?
      Was sparse bundle mountable before copying?
      Is sparse bundle mountable after copying?
      How did you perform copying?
      Did you try to mount ExtFS on macOS using mac Disk Utility? or using Paragon Mounter?
      I don’t remember if macOS recognizes locally attached time machine backup on ExtFS volume.
      You should mount it on NAS and expose over network and try to mount.

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.