Syncing & reacting to file changes

Sync

This docs page is for you if you

• Are trying to keep the state of your drive file tree in sync with something else

  • "Something else" being a database, a spreadsheet, a local file system, or a remote file system

• Want to make some automation off of a created/deleted/moved file or folder

This page is NOT for you if

• You're not a developer — we'll be making an easier-to-use version of this soon

Theory

The most reliable way to sync two systems is to give references to the changes over time, allowing the client trying to perform the sync to acknowledge, replay, apply the diff before ensuring database integrity and moving onto the next change

An example of this is like in git VCS where there are a series of commits and the difference between two commits is given by a "patch" if you want to end up with the current state of the tree you apply these patches to your local system which will add together to end up with the resultant state (ok not exactly like that, but it works in this example).

The sync is similar though in this case where you query our servers for a patch, if there's a new patch you apply that patch to your system — at the end of the day when you sum everything it will be correct.

How to

Getting the patch diff

1. Get an API key

2. Set up your listener system to periodically use that key to redeem a ShadeFS JWT using the route GET https://api.shade.inc/workspaces/drives/{drive_id}/shade-fs-token

   • This will return a JWT that is valid for the next 5 minutes

3. Now you can use that JWT to see the current "head" for your drive, this is a hash that represents the current state

   • Do this by calling GET https://fs.shade.inc/{drive_id}/heads/current, with your JWT as the "Authorization" header — this will return a hash like 972d06bb83b2979f7bd5d2e1e98aa0da2b763ffa7a5a408fee27c1d561e45754

4. To get your first patch diff you can get the lifetime history of the drive

   • Do this by calling GET https://fs.shade.inc/{drive_id}/heads/compare?to=your_hash_you_got

   • This will give back a JSON

5. As heads come in, the result returned by heads/current, compare that against the previous patch head you applied, if they're different, then perform the patch diff

Applying the patch diff

Now that you have a diff it can be parsed into changes fairly easily An entry for the "blob state" must be kept as well if its "null" "draft" or "main" Null means an empty file, draft means a file thats still being uploaded and is inaccessible and main means a fully uploaded and accessible file

1. First apply the "deleted files" in the deletedFiles entry of the diff

2. Next apply the renames in the "renames" entry of the diff by renaming files in the "from" to "to"

3. Create the files as main if they're in the "createdMainFiles"

4. Upsert the files as draft if they're in the "mainOrNullToDraftFiles"

5. Upsert the files as null if they're in the "draftOrMainToNullFiles"

6. Delete and reinsert the asset if it's in the "modifiedMainFiles" or in "modifiedDraftFiles"

You should end up with all of your files in sync

Dealing with folder creation is a little more difficult, we recommend just creating the folder if it doesn't exist and after delete if a folder goes empty then you can delete the folder. Sometimes order matters for folders like if you move a folder out then delete the parent if you apply the operations wrong its possible to accidentally delete all files under the directory and then "attempt and fail" to move the files in it out of it — so avoiding the problem all together is generally better by never having empty folders.