Time Machine backup disk migration to network drive

Published on Author Artem ButusovLeave a comment

Apple knows better

It is sad that Apple doesn’t provide any backup migration wizard to move
backups from locally attached disk to network share. The only option Apple
provides is to start backup from the scratch on network share loosing all your
backup history. This makes your network Time Machine backup absolutely useless
since you can’t search for the file that you accedentally deleted a few years
ago.

This instruction will help you to migrate your locally attached disk with
Time Machine backups and whole backup history to Time Machine compatible network
share.

There are different ways to do the same. All the ways have their pros and cons
and I would like to share my observations and probably save a little bit of
your time.

Configuration makes sense

This instruction is based on the configuration I have. If yourth is different
then you will probably need to slightly alter the plan.

My configuration:

  • MacBook Pro (Retina, 13-inch, Early 2013), 512GB SSD, macOS Mojave 10.14.6.
  • External USB3 backup magnetic disk 1TB with backups from 2013 year,
    HFS+ case-insensitive, not encrypted.
  • NAS with time machine Samba share pointing to another external USB3 magnetic
    disk formatted with ext4 (running on Odroid-XU4 with Arch Linux).

NAS Samba v4.10.10 config:

[global]

map to guest = Bad User

disable netbios = yes
smb ports = 445
min protocol = SMB2

vfs objects = catia fruit streams_xattr

[media]

path = /media
read only = no
guest ok = yes

[TimeMachine]

path = /media/TimeMachine
read only = no
guest ok = yes
fruit:time machine = yes

NAS Avahi config:

<?xml version="1.0" standalone='no'?><!--*-nxml-*-->
<!DOCTYPE service-group SYSTEM "avahi-service.dtd">
<service-group>
  <name replace-wildcards="yes">%h</name>
  <service>
    <type>_smb._tcp</type>
    <port>445</port>
  </service>
  <service>
    <type>_device-info._tcp</type>
    <port>0</port>
    <txt-record>model=Xserve</txt-record>
  </service>
</service-group>

This configuration makes Time Machine share discoverable and easy mountable
using Finder in macOS.

The NAS disk I have is already partitioned and formatted using standard Linux
tools. It has two GPT partitions:

  • 1TB for Time Machine backup, filesystem: ext4
  • remaining for files, filesystem: exfat

I was not able to get exfat volume to work as Time Machine backup due to the
lack of extended attributes support on exfat volumes. The fastest and most
stable filesystem with extended attributes on Linux is ext4. This will make
things easier on Linux side and… more complex on macOS side.

I have also this option set but not sure if it is needed these days:

defaults write com.apple.systempreferences TMShowUnsupportedNetworkVolumes 1

Recommendations

  • Use wired connection for both NAS and your laptop to speedup backup migration
    in case if you would like to migrate over the network.
  • Consider attaching both source backup disk and target backup disk (from NAS)
    to mac and do initial local copy (much faster write speeds if both disks are
    USB3). In my case local copy is around 90Mb/sec vs 12Mb/sec over network.
  • Use larger sparse bundle band size to improve backup performance, but this
    could be a bad advice if your are planning to use encrypted sparse bundle.
  • Consider having sparse bundle size the same as backup disk size. You won’t
    need to resize sparse bundle disk in that case and that will save time.

Create network sparse bundle

Firstly you will need to create compatible sparse bundle file on network drive.

Method 1: Manually

You can create sparse bundle image using Disk Utility on network share:
{ComputerName}.sparsebundle.

You can get know computer name by running: scutil --get ComputerName

There will be some manipulations needed with sparse bundle plist files to
make image be properly recognizable by Time Machine. Read more here:
https://gist.github.com/martian111/e0d9885004eb56fd6abf3d1ba7671737

You can also create encrypted sparse bundle if you want.

I did not follow this method so there are no more details in this section.

Method 2: Let macOS to do that for you

You can create it manually but the easies way is to let Time Machine to create
it for you. Just open Time Machine, add new backup disk on network share, do not
replace existing backup disk, initiate backup manually and don’t forget to
disable automatic backups. You can even choose option to encrypt backups.
Cancel the backup once Time Machine will start to show file copying progress.

After that step is completed you should have sparse bundle
{ComputerName}.sparsebundle on your Time Machine network share.

Change band size on sparse bundle

This is OPTIONAL procedure but can significantly improve backup performance.

You can find more details in this article:
https://edoardofederici.com/improve-time-machine-performance/

Below I would outline shorter version that doesn’t require too much scripting
in Terminal.

If you are planning to use encrypted sparsed bundle then you should probably
avoid changing band size as there are some reports that it could make
performance even worse: https://gist.github.com/SebastianJ/b3e9af8a641df1dc73c0

Default sparse bundle band size is 8MB = 16384 blocks. Too small or too big
band size could slow down the backup. Choose band size depending on how fast
your network connection, how fast your target network and how big the hard disk
you are backing up.

Command line band size for hdiutil is specified in 512 byte blocks, so
8MB = (8 * 1024 * 1024 / 512) blocks = 16384 blocks.

I played a little bit and found that local copying speed to sparse bundle with
32MB band size is twice higher than with 8MB band size (100Mb/s vs 50Mb/s).

Since I would like make target sparse bundle to be encrypted I decided to keep
band size as it is and skip band size conversion.

By the way, here are migration steps:

  • Unmount sparse bundle image using Disk Utility if already mounted (do not eject).
  • Mount Time Machine volume from NAS.
  • Convert band size using hdiutil command below.
  • If you will choose to use encryption then password will be asked.
  • Move original sparse bundle somewhere as backup.
  • Rename new sparse bundle to have the same filename as original.

Conversion without encryption:

hdiutil convert /Volumes/TimeMachine/{BundleName}.sparsebundle -format UDSB -tgtimagekey sparse-band-size={BandSize} -o /Volumes/TimeMachine/{TempBundleName}.sparsebundle

Conversion with encryption:

# Encryption could be AES-256 or AES-128
hdiutil convert /Volumes/TimeMachine/{BundleName}.sparsebundle -format UDSB -tgtimagekey sparse-band-size={BandSize} -encryption {Encryption} -o /Volumes/TimeMachine/{TempBundleName}.sparsebundle

There will be some manipulations needed with sparse bundle plist files to
make image be properly recognizable by Time Machine. Read more here:
https://gist.github.com/martian111/e0d9885004eb56fd6abf3d1ba7671737

Mount sparse bundle

Method 1: From network drive (easy, slow)

If you can connect your Mac over the wire or if your wireless network is fast
and your Mac supports 802.11ac then it could be wise and convinient to do
migration over the network: from backup disk connected to Mac to network drive
connected to the NAS.

If you are not in rush and it is okay to have your Mac copying in background
files for a few days then it could be a good option too, especially, if second
method doesn’t work for some reason or looks too complex.

Unfortunately, my Mac is considered “vintage” these days (in terms of Apple) and
the max network bandwidth is just around 200Mbit/s between NAS (wired) and Mac
(wireless) using default wireless router provided by Verizon. With Samba and
sparse bundle incapsulation total bandwidth averages 100Mbit/s. Copying 1TB of
data as it is on 100Mbit/s should take around 20-30 hours. The longer it runs,
the higher the chance that something will go wrong, so I decided to do faster
local drive cloning.

Method 2: From local drive (complex, fast)

Local cloning should be the fastest way and faster than any network copying.

There is a special driver needed for macOS to be able to write to Linux ext4
formatted volume, that I have on Time Machine backup volume on the disk,
connected to the NAS.

There is proprietary driver Paragon ExtFS that can read/write ext2/3/4 volumes.
You can purchase it or install trial version (10 days should be more than enough):
https://www.paragon-software.com/us/home/extfs-mac/

There is also fuse2fs application from e2fsprogs package that can do almost same
and be installed using HomeBrew (did not test this path).

Steps:

  • Install Paragon ExtFS and reboot.
  • Detach hard drive that was connected to NAS and attach it your Mac. New disk
    should appear in Disk Utility and, if ext4 driver is installed, ext4 partition
    with Time Machine backup should be auto mounted and available in Finder.
  • Open your target Time Machine volume and open target sparse bundle on it to
    mount it (the one that was previously created).
  • Now you should have both source disk and target sparse bundle mounted and
    you can copy the contents between them.

Copying contents

Method 1: Copy using Finder (slow)

Your external backup disk most-likely has case-insensitive HFS+ and the sparse
bundle will have by default case-sensitive HFS+ so copying Backups.backupdb
using Finder won’t work by default.

Steps:

  • Reformat sparse bundle volume to case-insensitive HFS+ (same filesystem as
    external backup disk) using Disk Utility.
  • Manually set ownership flag on it using Terminal and the remount:
    sudo diskutil enableOwnership {PartID}.
  • Only after that you can copy Backups.backupdb using Finder from one volume
    to another volume.

But it will take EXTREME amount of time. In my case even whole night was not
enough to build just list of files to copy. I can only imagine how long will
it take to also copy files. I gave up and decided to go forward with local
cloning solution.

Method 2: Copy using disk cloning (fast)

There is an easier and faster way – clone external backup disk to sparse
bundle disk using dd as it is. No need to care about case sensitivity or
ownership flag – your sparse bundle will have exactly the same byte to byte
state as your external backup disk. Your sparse bundle volume must have the
same or a little bit larger size than source disk. You can resize (grow) HFS+
partition on sparse bundle disk after but it is not fast.

Steps:

  • Make sure that both source backup disk and target sparse bundle disk are
    visible in Disk Utility.
  • Unmount partitions (but not eject volumes) on source disk and target image.
  • Enable “Terminal” to have full disk access:
    • Open “System Preferences”
    • Then “Security and Privacy”
    • Then “Privacy”
    • Then “Full Disk Access”
    • Press “+” and add “Terminal” application
  • Quit “Terminal” if it is already started and start “Terminal” application.
  • Start copy process
    • Run sudo dd if=/dev/r{SourceDisk} of=/dev/r{DestinationDisk} bs=2m
    • Example: sudo dd if=/dev/rdiskA of=/dev/rdiskB bs=1m
    • Use r prefix, this will make process faster
    • Use disk ID, not partition ID
    • Grab identifiers from Disk Utility or using diskutil list.
    • Block size 1m should improve performance. You could experiment with
      different values. Terminate the process with Control + C, unmount
      volume again (otherwise macOS will auto mount it) and test the speed.
    • Press Control + T to see the progress and the speed.
    • If you have HomeBrew and coreutils installed then you could use GNU dd
      that can automatically report progress
    • Run sudo gdd if=/dev/r{SourceDisk} of=/dev/r{DestinationDisk} bs=1M status=progress
    • Example:sudo gdd if=/dev/rdiskA of=/dev/rdiskB bs=1M status=progress
    • Block size units have slightly different format (M instead of m).
  • Rename HFS+ volume on sparse bundle disk to Time Machine Backups after
    (not sure if it is required, but macOS created sparse bundles have this name
    for backup volume).
  • Sparse bundle won’t have exactly the same size as source disk, so need to fix that:
    • You could skip this step if source backup disk and sparse bundle have the
      same or about the same size.
    • Repair sparse bundle partition map (this will grow partition size in GPT)
    • Run sudo diskutil repairDisk {SparseBundleDiskID}
    • Example: sudo diskutil repairDisk diskX
    • Grow backup HFS+ volume in sparse bundle (grow volume size to partition size)
    • Run sudo diskutil resizeVolume {TimeMachineBackupPartID} 0
    • Example: sudo diskutil resizeVolume diskXs2 0
    • Resize will also trigger HFS+ file system check and it is good to confirm
      that partition after this hardcore migration is still not damaged
    • This procedure will take a lot of time because file system check is slow.

Paragon ExtFS did not show perfect stability for me. It crashed macOS a few
times until succeded.

The only time the process has succeded was with these steps and in ~5.5 hours:

  • fresh rebooted macOS
  • all applications besides Terminal were closed
  • 8MB band size was used (this slowed down copying process to 50MB/sec)
  • gdd instead of dd
  • disabled macOS sleep using Amphetamine app

Let me know if you will find what exactly can crash the copying process, so
I can share more details here for others.

Testing network backup

Eject NAS drive from your Mac and connect back to NAS.

You will most-likely need to fix ownership on files to make them readable
from network:

# in my example disk is mounted to /media and Samba is configured to use nobody
sudo chown -R nobody:nobody /media/TimeMachine

Initiate backup using Time Machine menu on macOS.

Mac could probably ask about “The identity of the backup disk has changed since
the previous backup”. This is fine for initial backup because this is exactly
what we did. Click “Use This Disk” to continue.

The initial backup should be successfully completed. You could enable automatic
backups back after.

I would recommend to use both backup solutions (local disk and network disk)
for some time before you could confirm that network backup works well in terms
of stability and performance.

Conclusing

I hope this article will save some of your time. Let me know please if there is
anything I can fix to make this article better.

Thank you and happy migration 😉

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.