Syncing project state across multiple machines

This article, and the discussion over at hackernews, reminded me that I wanted to write a short note about how I keep all of my project files in sync when developing on more than one machine.

Turns out this is non-trivial. Well, the solution is trivial but getting there wasn't. There are a few different ways to approach this but until I alighted on Syncthing, none of them were satisfying.

My dev setup

I have a home dev machine (running Ubuntu 20.04 LTS), but I also have a Macbook Pro (running Mojave) that I use when travelling or if I just want to move around while working or get out of the house.

My IDEs of choice are VS Code and Pycharm. All of my project files are in my home directory. Most of the time I'm in front of the Ubuntu machine, but when I sit down to use the Mac I want the state of all project files to match that on the desktop.

False Starts

My initial idea was to just sync everything using Github. Push my work before switching machines then pull down the latest commit. The problem with this is two-fold. First, every time you switch machines, you have to do a commit, a push and a pull. This quickly becomes tedious. Second, it results in a git log full of commits with no purpose other than that you were syncing the state of your files. There are ways to streamline it for example, using a post-commit hook to auto-push to the remote repo but it is clearly square peg, round hole. As I'm sure plenty of more experienced devs could tell you: Github is not a syncing or backup service and if you're using it as such you're pushing uphill.

VS Code and Pycharm both have remote development modalities (although the approach is quite different). However, the further you dig into these the more you realise that they are probably not what you want if your circumstances are like mine namely, two local machines with equal primacy in the development process.

VS Code has an excellent SSH extension, but this is really for different circumstances where you have a single remote development instance (e.g., VPS, container) and you want to do all your development in that environment, treating any local machines as dumb terminals. There are lots of reasons why that modality might be exactly what you want, but it wasn't what I wanted. I'm usually developing for both platforms and regularly working on both helps to ensure that platform-specific bugs are caught early. Using both platforms means I remain in touch with the stack as a whole, too. There are plenty of other reasons but they're not directly relevant only you know if these are your circumstances, too.

Pycharm's take is a "remote deployment" model (Tools -> Deployment). You first specify an SFTP connection to the remote host. Your Python project files are then uploaded to the remote and kept in sync each time a save is triggered (you can see the upload activity in the console). Note that this is not a wholesale two-way directory sync. First, a number of file types (for example, any SQLite databases) are not sent to the remote, so you do not have an exact duplicate of project state across machines. Second, the sync is one-way. You can't just sit down in front of the "remote" machine (in this case, your second dev machine), go to your home folder and work on your synced project files they are tucked away in a folder created and managed by Pycharm specifically for the purpose of hosting the SFTP connection from another machine. The directory structure in no way mirrors a normal local home folder structure. The "remote" folders really aren't designed to be accessed in this way. That setup falls far short of what I wanted, which is step-in, step-out perfect parity between both machines including sane directory structure.

I've seen it mentioned a few times that an alternative to using the native remote development capabilities of VS Code or Pycharm is to mount a remote filesystem containing your project repos on your local machine, creating a single source of truth. There are significant practical issues with this approach, however the machine hosting the repository has to be awake when you're working on the other machine; mounting and unmounting the remote share cleanly through the rebooting (planned or unplanned) of either machine throws up issues; you can't work from the secondary machine without a network connection.

Syncthing

Instead, the solution that works best for me is robust file sync using Syncthing. See the article linked at the top of this post for a good look at getting everything set up and linked I'm only going to focus on how it works in the context of two (or more)-machine development setups. Note that unlike the linked post I run Syncthing in a Docker container for convenience it allows me to bundle it up along with other services into a single, version-controlled docker-compose.yml file. YMMV.

All project files are put into folders monitored by Syncthing (in my case, I use a single ~/dev folder and a bunch of subfolders under that). Syncing isn't instantaneous, but it's plenty quick (I believe you can change the polling rate if you're so inclined, although I haven't found it to be necessary). In the case of text and database files that make up much of my work, I haven't had any issues with file locking: I can have any given file open on both machines and if I make a saved change to that file on one machine, I can walk into the other room and find the file already updated on the other, the most recent change visible in the open VS Code / Pycharm instance.

Clearly in order for the machines to stay in sync, any machine on which you make a change must be awake and connected to the network when you switch on the second machine. That second machine will then connect to the first and and bring its state up to date. Note that this is a less onerous requirement than for an SSHFS setup, where the machine hosting the repository must be awake even if you're working exclusively on the other machine at the time.

As with any sync protocol, if you make changes on one machine that are not synced to the other (if, for example, you switch off the machine hosting the changes before the other machine has pulled the sync), you will have to deal with file conflicts. This is an inherent limitation of any file syncing paradigm, though it's not Syncthing-specific. The same problem would arise if two machines syncing files using Dropbox found themselves in a similar situation. Syncthing has excellent documentation and a graceful approach to resolving sync conflicts.

While in pratice both of my dev machines are usually switched on during the work day, for convenience and redundancy I added a third Syncthing node to the network: a power-sipping Raspberry Pi 4-powered, Ubuntu 20.04 LTS-based NAS to ensure the availability of up-to-date files at all times. The Raspberry Pi is on 24 hours a day, 7 days a week and costs just a few dollars a month to run. It also hosts a bunch of other ARM64-architected homelab containers in addition to Syncthing (Plex Media Server, Pihole, etc.) which also require 24-7 availability so the additional marginal trouble is neglible in my case. I would highly recommend this setup as cheap and reliable, but it's by no means required for Syncthing to sync your project files effectively. In practice, adding additional nodes to your Syncthing network is trivial there's no reason you couldn't scale up to 3, 4 or even more dev machines if your workflow somehow requires it.

A note on using Syncthing and iCloud together

Finally, I just want to add a small note about using Syncthing on folders that are also synced using Apple iCloud. I have found them to work very well together no syncing conflicts so far. Obviously iCloud isn't needed at all to sync files as between two (or more) machines in a dev environment regardless of how many of them are Macs. The main reason you might want to use this dual-sync setup is if you need access to your project files on your iOS devices (which cannot run a Syncthing client). Pro-tip: the MacOS iCloud folder is not easy to find from the command line, which means it's not easy to add to Syncthing. Here's the correct way to enter it:

~/Library/Mobile Documents/com~apple~CloudDocs/Documents