Drowning in email? Try Inbox Zero to learn sane tips for dealing with high-volume email. And don’t miss the free Inbox Zero video. »
Register for free on 43 Folders to comment on articles, post to our forum, customize your visits, and much more. Current users can login now.
Geek Throwdown: How to sync two or more Macs?
Merlin Mann | Oct 12 2007
 Enter the OctagonHere’s an experimental new feature: The Throwdown. Take a problem that lots of people face and tell us your personal favorite way to deal with it — in as much detail and with as much persuasion as you can muster. Today, a lot of us are living on two or more Macs — which is great, except for the challenge of keeping the contents and settings of multiple machines effortlessly in sync. Now before you pop in, holler “dot mac,” and jump back on your Segway®, consider that many folks (including your author) are looking for a lot more than simple document syncing and perfunctory preference sharing. How about if your needs are more nuanced:
You get the idea. You have a system; now tell us about it. Bow to your sensei, then spare no detail. How do you sync your Macs?rsync? ChronoSync? Synchronize? Unison? Something you made yourself? What are using to sync your Macs, and how are you using it? POSTED IN:
|
|
| EXPLORE 43Folders | THE GOOD STUFF |
Popular |
|
Inbox ZeroThe original 43 Folders series looking at the skills, tools, and attitude needed to empty your email inbox — and then keep it that way. Don’t miss the free video of Merlin’s Inbox Zero presentation. Get Started with ‘GTD’David Allen’s popular productivity book and the system on which it’s based help turn ‘stuff’ into actions that support valuable outcomes. |
43 Folders is powered by Drupal, which rules. Our site was designed by a bearded fella from Wire and Twine and has been lovingly skinned for Drupal by the code ninjas at RoopleTheme. Ben Durbin is the sine qua non and our personal consigliere. 43f’s web hosting is sponsored by A2.
43 Folders® is a registered trademark of Merlin Mann. All posts and comments © their original owners. Everything else (including the design) is © Merlin Mann, 2004-2008. Merlin’s posts can be reused or republished solely in accordance with the BY-NC-ND Creative Commons License. Please don’t be a douche with my stuff. Here’s our Legal and Ground Rules. 43 Folders loves you. Namasté.
Re: Geek Throwdown: How to sync two or more Macs?
I use .Mac for keychains, calendars & bookmarks, etc, and Unison for everything else. I would say that the goal of smart comparison and the goal of full automation are somewhat in conflict. If you know that you will always require the most recent version of a file across all machines then you can automate fully. But I use Unison manually precisely because this isn’t always the case. This is also the reason I keep my backup system separate from the sync stuff.
Unison FTW
I imagine that the apps mentioned in the post will tip people to what I have tried. :-)
So far, I’ve found the most success with Unison, which I setup on two machines with great assistance from Ethan Schoonover. It was working pretty well for me, but three pilot errors I had were frustrating:
I’ve temporarily abandoned Unison while using [thing under NDA], but I plan to return soon.
I contend that if someone comes up a truly easy-to-use GUI front end to Unison, it might be game over — the feature set is just so powerful, especially when using a remote server + something to automate backups.
In the end, I think all these apps are awesome and powerful, but they’re also almost all a huge pain to configure and update. That’s why a lot of people satisfice with .Mac, IMO.
Re: Unison FTW
Oh, sure. Ask us questions that get us dreaming of the perfect app.
Introduce Unison (which I love, combined with the surplus storage at DreamHost makes a good backup/sync) as a great tool, profess your love.
Tell us you’ve temporarily abandoned it for something better, but then be unable to tell us anything about it due to NDA.
Cruel, cruel.
Subversion works pretty well.
I’m a fan of subversion. I’m a student at college now, and I find it very useful to subversion each of my class directories. I have a desktop that I use dynamic dns to access (sadly not a Mac) on which I set up each of the subversion repositories.
Not only do I get version control on every file for my classes, but I can check out the repository onto any computer with subversion and make edits (which I do on my Macbook). And when the semester’s over and I no longer need immediate access to my files, I just delete my local copy on my Macbook - instant space savings, and the repository is still safe on my desktop.
I have a weekly backup script to tarball my repository directory and back it up to an external drive, so my data stays pretty safe.
If I ever need to give a friend access to my class files, it’s no big deal - just adjust the permissions for that repository. No more than a few seconds of work.
I realize that subversion may not work for everybody (especially if you wanted to sync your entire user directory, or if you don’t like using the terminal) but for my needs it has worked wonderfully.
Automating Subversion?
How much can Subversion be automated? I’ve always wanted to learn more about it, but I feel like I get stuck at manually checking stuff in (from the command line).
Got a good tip for Subversion noobs?
Re: Automating Subversion?
People new to subversion (or who don’t want to do much tinkering with the command line) can take advantage of this Finder plug-in to have full access to the commands from within the comfort of the Finder. I haven’t personally tried it out, but I’ve heard good things about it. If you’re really new, you’ll probably want to read up on just what it is with this subversion book.
This makes it easy to use subversion from outside of the command line, but I don’t know if it has some sort of Applescript interface (I’m pretty sure it doesn’t) for automating updates or commits.
I find it useful to update and commit manually rather than at regular intervals (just helps to reduce conflicts if I’m editing the file on more than one machine), but I can’t imagine it to be too hard to automate the process using a shell script.
If you have a repository checked out at ~/college101/, then you can use this small shell script to automate the update and commit from the server:
Save this script, and add it to your crontab, and you should be set. My word of caution is that if you check out the repository onto multiple machines, and tend to edit similar files, you may start running into conflicts on those files. Remember to commit and update regularly without the script as well.
Re: Re: Automating Subversion?
SCPlugin works great, PathFinder also has pretty decent Subversion support built in. Not sure about using SVN to sync two Macs… but as a way to store working files, it’s awesome.
Re: Automating Subversion?
I concur completely, but I confess that it’s because as a software developer I think in terms of the programmer’s work cycle:
In svn, step 1 looks like
svn up, and step 3 looks likesvn commit; the exact incantation changes for git or p4 or what have you, but this is basically the workflow.Of course, if this isn’t your workflow, working with a version-control-style system will seem awkward and painful.
I bet, however, that you already go through all these steps if you currently use a mirroring solution (rsync/Unison):
The trick is that there’s an assumed step 2.5 in which your work magically gets synchronized elsewhere. If you’re using an always-on computer, you can hire a house-elf to take care of this for you (scheduled backup); laptop users have to insert this step manually by invoking Unison or what have you.
So, all this is a long way of saying, “if you’re afraid of the checkin/checkout work style, you may already be using it (and in denial).”
SVN hooks
You can also use cron or hook scripts with SVN to do various powerful backup-related tasks. For example, store a hot-backup every hour, or dump the latest revision in text form with each and every commit. Semi-painful to set up, but if that ain’t automation…
The main thing I don’t like about syncing with Subversion is that it forces you to put up with .svn folders everywhere. It is a bit unwieldy for this type of application.
subversion (SVN) is a poor choice for generic synching
I have used subversion for years. It’s great for automated backup and synching of a tightly integrated group of files that don’t change too much, like all the files in a coding project. But it’s a bad choice for general synching because of its fussy handling of file creation/deletion and because of its incompatibilities with subtleties of the OSX file system. There are three main problems:
1) SVN requires special workflow for adding and deleting files.
As long as you’re only modifying files in a predefined set, SVN requires little handholding. You just do an “update” command at the beginning of your work session and a “commit” command at the end. But if you want to create a new file, or delete an old one, it is not enough to create or delete it in the Finder. You need to issue an explicit “add” or “delete” command to subversion. The same is true every time you create or delete a directory.
2) SVN isn’t aware of bundles.
The last issue becomes a real pain when you also consider that SVN is unaware of file bundles. A lot of files in OS X are in fact directories, which the Finder presents as a single file. But SVN just sees it as a directory. So what happens when a file within the pseudo-file bundle is created or deleted? Then SVN requires you to go inside the file bundle and do the explicit add or delete command on the files within. This is a common problem with apps like OmniOutliner, that use file bundles for their document format.
The workaround is to always compress the bundle into a zip file or a dmg, and then handle that through SVN, but this is a nuisance.
3) SVN obliterates metadata
SVN was designed for a standard unix filesystem like ext2, and so it loses a lot of the metadata that is unique to the HFS+ filesystem used by OS X — extended attributes (used by apps like Skim.app), resource forks (I think), ACL’s, etc.. This isn’t always a big deal. But it’s one more thing to worry about.
Mercurial
I’ve been using Mercurial, a subversion replacement, for awhile. I do currently use it primarily for source code, but it doesn’t limit you to just that.
What I like about it is that it handles offline checkins very, very well.
My personal spec was: I have a laptop. At the time I was in Germany where cellular data was very expensive. I want to be able to checkpoint my work. When I’m about to rip apart some code, I want to check in the working version on my laptop without an Internet link.
I also work on code that runs on servers - primarily Python. I do the testing on my local laptop, then I push it to the server. Sometimes I find bugs on the server side code. I could edit on the laptop and push back, but I usually prefer to edit on the server. Now I have another version of the code…
Mercurial handles this all extremely well. I can push code changes back and forth, and I keep a full history of the changes.
When I type “hg push” on my laptop, all my changes are pushed onto my remote server, including the full history of those changes. When I type “hg pull” on my laptop, I get the changes back from the server. If I’ve made conflicting changes it does inform me and I have to fix that, but if you don’t make conflicting changes it’s virtually effortless.
There is a bit of a learning curve, but if you read their quick start tutorials, and create a dummy repository with a few text files it will all start to make sense. I think it’s the best solution out there if you want to have serious support for disconnected operation.
Second that.
I’ve been using Mercurial to manage my site and a bunch of documentation across both my Macs and a Windows laptop, and it is now pretty much brainless.
Sure, it needs a GUI, but I’m confident one will emerge given time. Considering the advantages of being able to edit things in a fully decentralized fashion and having it all mostly glom together without any hassle, it is well worth the trouble of typing “hg pull” and “hg push” now and then.
IMAP is the other thing I can’t live without (all my e-mail - and RSS as e-mail - goes to the same box, and gets read in a single place via whatever I’m using).
Two more wishes
Merlin, I’d like to add three wishes to your list:
Network optimized: Use the speed of the local network when both machines are on it, and be able to avoid firewall issues by falling back to any web (port 80) connection.
End-to-end strong encryption: I don’t want my data stored unencrypted on any intermediary servers that can be hacked.
Peer-to-peer: Let me set up any number of shared folders and decide who I want to share it with, instead of one centrally located “account” I have to “manage”.
When I was using Windows, I used Groove and it was pretty much the ideal file sync tool, addressing all the needs you mentioned (and my three extra ones). I still miss it terribly (Groove, not Windows).
Foldershare
I’ve been using Foldershare for 3 years now - though a non-Universal App - it works great. I hate the fact they are now a MS property - but that has not affected the service one bit.
At my peak, I was syncing my projects folder over 5 different Macs at once. These days, I rely on Foldershare to serve as backup functionality only - as I have it running on an offsite Xserve.
Either way, it’s a great service. One I’m surprised hasn’t been challenged.
Foldershare
I don't know how well it would work on preferences and the like. It works on a folder-by-folder basis; you can't specify or exclude particular files within the folders.
That said, I use it constantly on a work PC, a home PC, and a home Mac, synchronizing nearly 2GB of files spread over hundreds of folders, and it has that wonderful Mac-ish quality of being easy to set up and just plain working, with almost no fuss.
One part Foldershare, One part Mozy
I’ve got an iMac, a Macbook and a PC that I use as part of my law practice. Foldershare just works. Throw in a little Mozy for backup purposes, and it’s what .Mac should have been.
Just a 160gb iPod away?
I haven’t tried this but wouldn’t a portable Home folder on a HDD based iPod, preferably 160 gigs, do the trick?
No sync required.
Re: Just a 160gb iPod away?
I think this is pretty good for files, but what functionality could actually sync stuff like your preferences? AppleScript maybe, I guess.
For me the challenge isn’t in finding media to move stuff around — I love thumb drives — it’s more in trying to find smart functionality for doing the comparing, moving, and updating (without input or work from me). I think that’s the trickier part.
more on Portable Home folder
Correct me if I’m wrong. Aren’t the vast majority of preferences located in the Library of your Home folder? They would travel with the iPod as well.
The following is from Apple.com Leopard Server “More New Features. The Next Level.” page…
Portable Home Directories 2
External Accounts is a new Portable Home Directory feature that allows you to have a home directory on an external FireWire or USB portable drive. The account can be fully managed using Workgroup Manager and can be protected using FileVault. This new feature is ideal for student use in computer labs or for users with multiple workstations.
http://www.apple.com/server/macosx/leopard/more.html
Re: more on Portable Home folder
Ok, I see. So theoretically, this is something I could use if I ran Leopard server on my main, always-on Mac?
Single Point of Failure
This looks really cool. I would love to try it.
However, you would have a new problem. If that iPod goes down, you are toast.
If there is a way to have Leopard automatically backup the home directory to machines it is connected to, that would help a lot. Encrypted maybe?
If not, you could be back where you started. You would need to connect, and remember to back up. Not ideal, but there is great potential.
iPod Toast & Home folder
Backup, backup, backup.
Actually...
You can just store files on an external device (iPod or otherwise) and create symlinks on the workstation that point to the device.
I’m a sysadmin in a research lab, and our lab’s primary scientist keeps all her data on an iPod; however, the “Documents” and other folders on her home and work Macs are merely symbolic links to locations on the iPod. Unix-fu FTW!
It would be trivial to symlink “Library” as well to a location on external storage so that your prefs are mobile as well. However, as others have mentioned, that also sets up the external device as a single point of failure. It is also a good idea, if using such a system, to periodically back up your mobile storage somewhere else; even on the local hard drive of one or the other primary workstation. This is currently done manually by the scientist herself, though I’m working on an Applescript solution to make it more “set-it-and-forget-it” automatic.
Basically what this comes down to is keeping your data under sysadmin control (e.g. on a server) or under user control (e.g. on your own portable hard drive). Always opt for the former unless you’ve got a user who’s persnickety and also powerful within your organization.
(P.S. I’ve only been running this lab for 6 weeks; believe me, if I had been the one to set up this “mobile home folder” thing, there would already be an automatic solution in place…)
Using Portable Home Folder since OS X 10.1
You don’t have to run Leopard to have a portable Home folder. I’ve been using a portable Home folder since 2002 (i.e., since OS X 10.1).
All you have to do is use NetInfo Manager to change the location of your home folder from /Users/[username] to /Volumes/[diskname], where [diskname] is the name of your removable disk. Then repeat all these same steps exactly on the other Mac. (I think it might also be best to use the same UID numbers for the user accounts on both machines, although I’m not sure whether it makes a difference.) The entire removable disk then becomes your Home folder.
For years I have carried my little disk back and forth from the office Powerbook to the home iMac. At home, I plug in the hard drive and I am immediately working with the originals of all my office files — no syncing, no updating, and everything is exactly as I left it at the office. All the windows are in the same places, all my web browser bookmarks are there, my e-mail address book is completely up to date, even mundane things like highlight color and toolbar position are preserved. No synchronization, no waiting, no comparing older and newer versions of files. Just plug it in, and boom, there’s your workplace desktop, just as you left it. Try doing that in Windows!
For some reason, this idea of using an external drive seems to freak some people out. (Note the comment here entitled “Single Point of Failure” which notes that “If that iPod goes down, you are toast.”) However, my approach actually makes your backup strategy even more robust because you are now backing up in two geographically separate locations. I use 2.5-inch disks (160GB each) and the outstanding bus-powered MacAlly PHR-250CC enclosures. I have five of them — the main one which is my Home folder, and four clones updated several times a day with SuperDuper and/or rotated out every month or so. I also do incremental backups to big fat 3.5-inch drives. These clones and backups are located both at home and at work, so even an earthquake probably wouldn’t wipe me out. (I work in Tokyo and live in Saitama prefecture, many miles away, so even a big quake like the one in 1923 probably wouldn’t destroy both work and home.)
Also, when I killed my work Powerbook in July, by spilling Coke all over it, the Home folder wasn’t affected. I just took out an old Titanium Powerbook, plugged in my Home folder, and kept on working. Total downtime was about ten minutes. Then a few days later I bought a new Mac Pro and did the same thing. Indeed, I now have four computers that are all configured to use the same Home folder, without any syncing between them.
some more info please
Can you detail some more how exactly you are doing things?
When you update your external disks, I assume you daisy-chain two external drives and clone one to the other, right? How would you go about the airplane problem, i.e. how would you clone the external to the internal harddrive? Is its configuration exactly the same or did you set up any symlinks or the like that might pose a problem here?
BTW I think that choosing the same UserID etc is vital to preserve permissions. Correct me if I’m wrong.
Many thanks for sharing your experience
Prion
Portable Home Folder and the Airplane Problem
I do indeed daisy-chain my external disks together. I have seven external Firewire disks attached at the moment. Two are clones of my Home folder, two consist of my video library and its clone, and three are staggered incremental backups of the Home folder and the video library. I also have another drive that I plug in about once a month to clone my internal boot drive.
My external drive with my Home folder has actually died twice (nothing lasts forever!), but each time I just plugged in a clone and kept going. The first time I didn’t lose any data, the second time I lost about ten e-mail messages that arrived between the backup time and the drive failure.
On airplanes I just plug in my external drive with the portable Home folder and use it like I do everywhere else. It consumes a little extra power but I pack extra batteries.
If you didn’t want to do that, then you could clone the external drive with your portable Home folder to a disk image on the internal hard drive with the same name as your external drive. (You could log into a different user account in order to mount the disk image, then switch into your regular user account.) When you got to your hotel room you could then clone the disk image back to your external drive.
And no, there are no symlinks necessary with these ideas.
putting everything to sleep
Hi Keith
many thanks for your reply!
How do you log off before unplugging the external HDD? Is putting the Mac to sleep sufficient or do you need to log in as another user?
When I am on the road I’d rather work on the internal HDD only so your suggestion to clone it to a disk image sounds promising. theoretically this should even work with symlinks as it is pointing to the same target (disk image and external HDD sharing the same name).
But it seems to me that setting this up may be more difficult. Do you set the external drive to ignore file persmissions? If not, how do you clone the external to the internal HDD (or in fact any two external HDDs)? Correct me if I am wwrong but when they are mounted simultaneously they should be called differently at least under the hood. Or do you log in as a neutral administrator and do the cloning from there?
Also, do you have any information if this system is going to work with Leopard equally well? Rumour is that some of the functionality of NetinfoManager will be moved to the Directory Service. No idea if that is going to make things more difficult or just a little different. Perhaps I am more confused than is actually necessary, so I’d welcome your feedback.
Prion
PS: have you posted a similar thread on the kGTD forum back in March 2006? Nothing goes unnoticed ;-)
Mounting and unmounting portable Home folder drives
Prion,
Yeah, that was me on the kGTD forum! Sorry, I never checked back to see your question. Ethan’s Kinkless installer is one of about four programs I have ever used that mistakenly assume that everyone’s Home folder will be in the /users directory. I have always found a workaround.
To answer your kGTD question, yes, under Panther it used to be that If you accidentally logged into the account without the external drive attached, OS X would create a home folder for that account in a folder with the name of the external disk in the /Volumes subdirectory on the startup disk. For example, if your account was called “Prion” and your external disk was called “External,” then OSX would create a folder in /Volumes called “External” and would put a “Prion” home folder into it. You then wouldn’t be able to use the external drive as a home folder for the “Prion” account until that automatically-created “External” folder was deleted, which was a pain because of permissions. (In 10.1 days, I used to boot into OS 9 to do it.) But Tiger got smart, and it doesn’t let you do that any more. If you try to log in without your external drive mounted, it will ponder for a bit and then tell you that you can’t do that right now.
I always log in as another user to eject the external drive after logging out of the account that has its home folder on the external drive. This might not be necessary but I do it anyway. I figure that until a drive is ejected, it’s probably still mounted, even if no one is logged in. I never tried your sleep idea.
You may also find that sometimes you need to log into a dummy account in order to mount your Home folder drive before logging into the account that uses it. My Powerbooks started making me do this then when Tiger came along, but then I got a Mac Pro and for some reason it doesn’t. Perhaps in Tiger it was decided that Powerbooks would not mount external drives until login, in order to save power, while no one worries about power usage with a Mac Pro, so it’s configured to mount all the drives immediately at startup? I have no idea. But logging into a dummy account never took more than a few seconds anyway.
I do not set up the external drive to ignore permissions because SuperDuper preserves all the permissions. (Indeed, I believe it complains if you have the external drive set to ignore permissions.)
You are correct that each volume must have a unique name, but this won’t stop you from plugging in two disks with identical names. Try it sometime — if you mount two disks named “Prion,” you’ll find that OS X invisibly calls one of the disks “Prion-1.” You can see this in the /Volumes folder. Just be sure that this invisible renaming is terminated before you try to log in using your Home folder on the external drive. If your Home folder is on “Prion” but OS X has temporarily decided to call it “Prion-1” because another “Prion” was already present when your Home folder drive mounted, then you won’t be able to log in until you get OS X to call “Prion” by its original name again. The easiest way to do this is to eject it and then remount it. (I eject, turn it off, then turn it back on again.) This should work just the same with an external drive and a disk image, I suppose.
I have no reason to suspect that any of this will change with Leopard. So far it’s worked since 10.1 and the only changes have been improvements, like when Tiger got smart enough to refuse to let me log in when I hadn’t yet plugged in my Home folder drive. I guess we’ll see soon enough, though!
Keith
great stuff
Sorry to interrupt, but thanks Merlin for the question to get things flowing and thank you prion and keith for dishing up the follow-up questions and detailed answers. Totally web show material! I am so tired of the basic stuff covered, we need an advanced genius level web show with topics like this.
Merlin, whenever the Merlin show gets up and running again you should totally do a show about this stuff (or anyone else with the skill, resources, and time). Syncing or having your most current data available at all times should be the key to computing in the future.
I learned so much from you guys, thanks to all of you for sharing.
this is where symbolic links come into play
Keith
many Carbon programs and some other programs like the infamous Installer expect to find the home folder including your library of things that keep them happy on the local harddrive. The reason I asked about symlinks with your solution was that they can be incredibly handy to solve this problem. Indeed this would be possible with your suggestion of letting the home folder reside in a disk image bearing the same name as the external harddrive for when you need the maximum mobility.
Disclaimer: I have only played around with this because my Powerbook is the latest generation G4 while my desktop is an Intel iMac so I cannot “go real” with this. But intial testing looks reallly well.
What you do: You copy your home folder to its new location preserving its resourceFork with ditto (while this user is logged OUT) and then you create a symbolic link pointing from the old location to the new one. The syntax is: sudo ln -s /Volumes/NewLocation/Prion /Users/Prion
This was figured out by people more knowledgeable than myself but it works very well in testing. As soon as my trusty Powerbook fades into nothingness I hope this will give me the best of both worlds: Give me the a home folder with all my programs, all my files, all everything that I just need to plug into wherever I may be without having to lug my Powerbook with me and without being restricted to a laptop in the first place. (By then Pussycat - the successor to Leopard - will be out and perhaps all this will be irrelevant: eeek).
I was surprised that almost all of the mobility I requested when I bought the Powerbook was actually moving about between exactly two locations: my office and my study at home. rarely I would take the laptop elsewhere in my house and when I did I would just surf the net or watch pictures etc, not something I needed all my data with me. Carrying my home folder with library etc. instead sounds very compelling though and seems to work. Caveat: Testing works really well, for real life experience I need another intelMac.
Prion
PS: I really enjoy this discusssion myself, I hope others do, too, and share their experience.