A look at how sync works in beorg

beorg stores all of your projects, tasks, and notes in plaintext files. You may have one text file or hundreds. These text files are usually synchronized with a service such as iCloud or Dropbox. In this article I’ll be giving a little more detail about how this sync process works, and what to do if it isn’t working properly.

When you start beorg, or finish using the app, it will sync your files. You can choose to have a sync performed each time you modify a file. To do this make sure “Auto sync on modify” is enabled on the Settings tab. For users’ of iCloud a sync may be triggered by iCloud spotting an update to a file.

When a sync is in progress beorg blocks you from making changes. You’ll see this if you tap an item on your agenda, make a change and then try to save it. Usually a sync is fairly quick so this shouldn’t be an issue.

There is going to be some work over the next few months on improving how long you are locked from making changes. 🤞

Here is what happens when beorg syncs your files:

1. Your chosen sync method authorization is checked.
2. beorg tells iOS that it is beginning a background task. That way it can continue syncing even if you are using another app or put your phone in your pocket.
3. Notifications are scheduled for any upcoming tasks with dates.
4. A simplified version of your agenda is saved to the cache for use by the beorg widget and Apple Watch app.
5. If you aren’t using a remote sync service (i.e. you are keeping your files only in beorg itself) then anything captured by the share sheet is imported and the sync finishes.
6. If your sync service supports push updates (currently only iCloud) these push updates are disabled for the remainder of the sync.
7. If you are synchronizing the Apple calendar to an org file, then this org file is updated from the calendar. (You aren’t supposed to manually modify this file, or the reminders file.)
8. If you are synchronizing the Apple reminders to an org file, then this org file is updated from your reminders.
9. If there are any files you deleted locally then those files are deleted in the cloud (or another app).
10. The remote source is now asked for a list of all the files in your org folder. For some services this may require that multiple requests are made - for example on Dropbox you can only get information about 100 files are once. (Sync services such as Dropbox unfortunately can’t be asked just for a list of files which have been modified since a particular date. That would make sync much quicker.)
11. The list of files is filtered to remove any files which should be ignored for the purposes of the sync. This is determined by the Scheme function sync-ignore-file.
12. For each file on the remote source (e.g. Dropbox):
13. If the remote file isn’t newer than a corresponding local file or if it is the generated calendar org file then the process continues to the next file.
14. If there is a corresponding local file then a snapshot is taken by the version manager (up to 7 days worth of versions of each file are stored locally in beorg).
15. If there is a corresponding local file that has been modified since it was last uploaded, but the remote file also looks like it has been modified, then flag it as a potential conflict. Since a snapshot of the local file has been taken the user can revert the file if needed.
16. The file is now downloaded.
17. Once download a snapshot is taken by the version manager (the version manager generates a hash for each file so only takes a snapshot if there are any actual changes).
18. The process now continues with the next file.
19. Anything captured via the share sheet is now inserted into the relevant files. This ensures that you can capture without worrying about if you’ve modified the file outside of beorg.
20. Locally modified files are now uploaded.
21. If files have been deleted on the server (and weren’t just uploaded) they are deleted locally.
22. If your sync service supports push updates then this is re-enabled.
23. Any errors are the sync are shown to the user.
24. Finally notifications are scheduled and the cached agenda (used by the widget and Apple Watch app) is updated.
25. iOS is told that this process is completed and that if the app is in the background it can be fully suspended.
26. Now the sync has finished you are free to modify your files again.

As you can see there is a lot that happens when beorg synchronizes your files.

Sometimes you may find that syncing isn’t working 😥. If that happens you can take advantage of the beorg debug log. For most sync actions beorg will add to the log, so that in the event sync fails it is often possible to work out what has gone wrong. You can view the debug log by either:

• Going to the Settings tab, scrolling to the bottom and tapping “Show Log”
• Also on the Settings tab is the REPL (a sort of command line interface to the Scheme scripting used by beorg). See https://www.beorg.app/support/debug/ for more details on how to get the log as text you can copy/paste.

To look out for:

• If you are using WebDAV: SSL issues/getting the URL/folder right can be fiddly. If you are using HTTPS for WebDAV (highly recommended, otherwise you are transmitting your files in plaintext potentially over the internet) then you’ll need to be using a domain name and not an IP address.
• iCloud can sometimes (hopefully not too often) get ‘stuck’ - i.e. you’ll make a change on your Mac but it isn’t being picked up on your iPhone/iPad. A restart of your mobile device should fix the issue.
• The “Choose Folder” sync method only works if you the app providing the files to beorg allows you to select a folder. Many apps, such as OneDrive, don’t allow folders to be selected unfortunately.

I hope this deep dive into beorg’s file synchronization has given you a clear understanding of how it works and how to troubleshoot common issues. If after reading this you have some ideas on how I could improve sync let me know — your feedback helps make beorg even better. For any questions or further support, feel free to reach out on Mastodon or Bluesky. Happy syncing!