| Both sides previous revisionPrevious revisionNext revision | Previous revisionNext revisionBoth sides next revision |
| dev:web_api:v3:syncing [2017/11/27 04:08] – Remove Zotero 5 warning bwiernik | dev:web_api:v3:syncing [2019/03/10 04:22] – [Full-Library Syncing] dstillman |
|---|
| TODO: | TODO: |
| |
| * WebSocket notifications (planned feature) | * WebSocket notifications (implemented, but not documented) |
| |
| ===== Sync Properties ===== | ===== Sync Properties ===== |
| The following steps are for complete syncing of Zotero libraries, such as to enable full offline usage. For tips on alternative syncing methods, see [[#partial_library_syncing|Partial-Library Syncing]]. | The following steps are for complete syncing of Zotero libraries, such as to enable full offline usage. For tips on alternative syncing methods, see [[#partial_library_syncing|Partial-Library Syncing]]. |
| |
| ==== 1) Get updated group metadata ==== | ==== 1) Verify key access ==== |
| | |
| | GET /keys/current |
| | |
| | ''200'' Response: |
| | |
| | <code>{ |
| | { |
| | "userID": 12345 |
| | "username": "Z User" |
| | "access": { |
| | "user": { |
| | "library": true |
| | "files": true |
| | "notes": true |
| | "write": true |
| | } |
| | "groups": { |
| | "all": { |
| | "library": true |
| | "write": true |
| | } |
| | } |
| | } |
| | } |
| | }</code> |
| | |
| | ''/keys/current'' returns information on the API key provided in the ''Zotero-API-Key'' header. Use this response to verify that the key has the expected access to the library you're trying to access. |
| | |
| | ==== 2) Get updated group metadata ==== |
| |
| Group metadata includes group titles and descriptions as well as member/role/permissions information. It is separate from group library data. | Group metadata includes group titles and descriptions as well as member/role/permissions information. It is separate from group library data. |
| GET /users/<userID>/groups?format=versions | GET /users/<userID>/groups?format=versions |
| |
| ''200 '' Response: | ''200'' Response: |
| |
| <code>{ | <code>{ |
| }</code> | }</code> |
| |
| Delete any local groups not in the list. Optionally, if data has been modified locally in any remotely deleted groups, offer the user the ability to cancel and transfer modified data elsewhere before continuing. | Delete any local groups not in the list. If data has been modified locally in any remotely deleted groups, offer the user the ability to cancel and transfer modified data elsewhere before continuing. |
| |
| For each group that doesn't exist locally or that has a different version number, retrieve the group metadata: | For each group that doesn't exist locally or that has a different version number, retrieve the group metadata: |
| Update the local group metadata and version number. | Update the local group metadata and version number. |
| |
| ==== 2) Sync library data ==== | ==== 3) Sync library data ==== |
| |
| Perform the following steps for each library: | Perform the following steps for each library: |
| Retrieve the versions of all objects changed since the last check for that object type, using the appropriate request for each object type: | Retrieve the versions of all objects changed since the last check for that object type, using the appropriate request for each object type: |
| |
| GET <userOrGroupPrefix>/collections?since=<last collections version>&format=versions | GET <userOrGroupPrefix>/collections?since=<last saved library version>&format=versions |
| GET <userOrGroupPrefix>/searches?since=<last searches version>&format=versions | GET <userOrGroupPrefix>/searches?since=<last saved library version>&format=versions |
| GET <userOrGroupPrefix>/items?since=<last items version>&format=versions | GET <userOrGroupPrefix>/items/top?since=<last saved library version>&format=versions&includeTrashed=1 |
| GET <userOrGroupPrefix>/items/trash?since=<last items version>&format=versions | GET <userOrGroupPrefix>/items?since=<last saved library version>&format=versions&includeTrashed=1 |
| |
| If-Modified-Since-Version: <current local library version> | ''<last saved library version>'' is the ''[[#last-modified-version|Last-Modified-Version]]'' returned from the API for the last successfully completed sync process, or ''0'' when syncing a library for the first time. |
| |
| (The ''since'' parameter can also be used on ''.../tags'' requests (without ''format=versions'') by clients that don't download all items and wish to keep a list of all tags in a library up-to-date. It isn't necessary for clients that download all items to request updated tags directly, as item objects contain all associated tags.) | (The ''since'' parameter can also be used on ''.../tags'' requests (without ''format=versions'') by clients that don't download all items and wish to keep a list of all tags in a library up-to-date. It isn't necessary for clients that download all items to request updated tags directly, as item objects contain all associated tags.) |
| |
| If the API returns ''304 Not Modified'', no library data of any object type has changed since the version specified. If you are tracking a single library version for all object types, skip ahead to [[#iv_upload_modified_data|uploading modified data]]; otherwise, skip to the next object type with a lower stored library version. | The first request — e.g., for collection versions — can also include an ''If-Modified-Since-Version: <last saved library version>'' header. If the API returns ''304 Not Modified'', no library data of any object type has changed since the version specified and no further requests need to be made to retrieve data. |
| |
| ''200'' response: | ''200'' response: |
| ]</code> | ]</code> |
| |
| For each returned object, compare the version to the local version of the object. If the remote version doesn't match, queue the object for download. Generally all returned objects should have newer version numbers, but there are some situations, such as full syncs (i.e., since=0) or interrupted syncs, where clients may retrieve versions for objects that are already up-to-date locally. | For each returned object, compare the version to the local version of the object. If the remote version doesn't match, queue the object for download. Generally all returned objects should have newer version numbers, but there are some situations, such as full syncs (i.e., since=0) or interrupted syncs, where clients may retrieve versions for objects that are already up-to-date locally. The version will also match for top-level items on the second, non-'/top' ''items'' request, since top-level items will have already been processed. |
| |
| Retrieve the queued objects by key, up to 50 at a time, using the appropriate request for each object type: | Retrieve the queued objects by key, up to 50 at a time, using the appropriate request for each object type: |
| GET <userOrGroupPrefix>/collections?collectionKey=<key>,<key>,<key>,<key> | GET <userOrGroupPrefix>/collections?collectionKey=<key>,<key>,<key>,<key> |
| GET <userOrGroupPrefix>/searches?searchKey=<key>,<key>,<key>,<key> | GET <userOrGroupPrefix>/searches?searchKey=<key>,<key>,<key>,<key> |
| GET <userOrGroupPrefix>/items?itemKey=<key>,<key>,<key>,<key> | GET <userOrGroupPrefix>/items?itemKey=<key>,<key>,<key>,<key>&includeTrashed=1 |
| GET <userOrGroupPrefix>/items/trash?itemKey=<key>,<key>,<key>,<key> | |
| |
| Item responses include creators, tags, collection associations, and relations. | Item responses include creators, tags, collection associations, and relations. |
| for each updated object: | for each updated object: |
| if object doesn't exist locally: | if object doesn't exist locally: |
| create local object with version and set synced = true | create local object with version = Last-Modified-Version and set synced = true |
| continue | continue |
| | |
| if object hasn't been modified locally (synced == true): | if object hasn't been modified locally (synced == true): |
| if version number matches: | overwrite with synced = true and version = Last-Modified-Version |
| continue | |
| | |
| else: | |
| overwrite with synced = true and new version number | |
| | |
| else: | else: |
| if different: | perform conflict resolution |
| perform conflict resolution | if object hasn't changed: |
| | set synced = true and version = Last-Modified-Version |
| | |
| if user chooses remote copy: | else if changes can be automatically merged: |
| overwrite with synced = true and new version number | apply changes from each side and set synced = true and version = Last-Modified-Version |
| | |
| if user chooses local copy: | else: |
| synced = false | prompt user to choose a side or merge conflicts |
| | if user chooses remote copy: |
| else: | overwrite with synced = true and version = Last-Modified-Version |
| Update version and set synced = true | |
| | else if user chooses local copy: |
| | synced = false and set a flag to restart the sync when finished |
| | |
| |
| When modifying objects locally, set ''synced = false'' unless the write is a result of syncing. | If an error occurs while processing an object (e.g., due to a foreign-key constraint in a database), it can be handled one of two ways: |
| | |
| | - Treat the error as fatal and stop the sync without updating the local library version |
| | - Mark the object as needing to be downloaded later and continue with the sync, updating the local library version at the end as if the sync had succeeded. In a future sync, add objects with this flag to the set of objects returned from the ''versions'' request so that their data is requested again even if the remote version is lower than the library version specified in ''?since=''. |
| | |
| | When processing a set of objects, it may be helpful to maintain an object queue and add failing objects to the end of the queue in case they depend on other objects to succeed. (In some cases, it's also possible to sort objects beforehand to avoid such errors, such as by sorting parent collections before subcollections.) |
| | |
| | When objects are created or modified locally by the user during regular usage, set ''synced = false'' to indicate that the object needs to be uploaded on the next sync. Give new objects version 0. Do not change the version when objects are modified outside of the sync process. |
| |
| === ii. Get deleted data === | === ii. Get deleted data === |
| |
| GET <userOrGroupPrefix>/deleted?since=<last deleted version> | GET <userOrGroupPrefix>/deleted?since=<version> |
| |
| Response: | Response: |
| if user chooses deletion, delete local object, skipping delete log | if user chooses deletion, delete local object, skipping delete log |
| | |
| if user chooses local modification, keep object and set synced = true | if user chooses local modification, keep object and set synced = true and version = ''Last-Modified-Version' |
| |
| Tags removed from all items are not necessarily deleted, hence the separate tag deletion mechanism. | Tags removed from all items are not necessarily deleted, hence the separate tag deletion mechanism. |
| === iii. Check for concurrent remote updates === | === iii. Check for concurrent remote updates === |
| |
| When done updating local data, compare the ''Last-Modified-Version'' returned from the ''collections?since'' request (i.e., the first request for changed data) to ''Last-Modified-Version'' from the ''/deleted'' request (i.e., the last request for changed data). If the version hasn't changed, server data hasn't changed in that library while downloading changes and the version can be stored locally as the current version for that library. If the version has changed, repeat the above steps to retrieve updated and deleted data. The ''Last-Modified-Version'' from each ''?since'' request can optionally be stored in memory to avoid having to download and compare the same keys if the requests need to be repeated. | For each response from the API, check the ''Last-Modified-Version'' to see if it has changed since the ''Last-Modified-Version'' returned from the first request (i.e., ''collections?since=''). If it has, restart the process of retrieving updated and deleted data, waiting increasing amounts of time between restarts to give the other client the opportunity to finish. |
| | |
| | After saving all remote changes, save ''Last-Modified-Version'' from the last set of requests as the new local library version. |
| |
| === iv. Upload modified data === | === iv. Upload modified data === |
| On a ''200'' response, set ''synced = true'' and ''version = Last-Modified-Version'' for each successfully uploaded Zotero object and store ''Last-Modified-Version'' as the current library version to be passed with the next write request. Do not update the version of Zotero objects in the ''unchanged'' object. Retry non-fatal failures. | On a ''200'' response, set ''synced = true'' and ''version = Last-Modified-Version'' for each successfully uploaded Zotero object and store ''Last-Modified-Version'' as the current library version to be passed with the next write request. Do not update the version of Zotero objects in the ''unchanged'' object. Retry non-fatal failures. |
| |
| On a ''412 Precondition Failed'' response, return to the beginning of the sync process for that library. | On a ''412 Precondition Failed'' response, return to the beginning of the sync process for that library, waiting increasing amounts of time between restarts. |
| |
| |
| === v. Upload local deletions === | === v. Upload local deletions === |
| |
| See [[write_requests#deleting_multiple_collections|Deleting Multiple Collections]], [[write_requests#deleting_multiple_searches|Deleting Multiple Searches]], [[write_requests#deleting_multiple_items|Deleting Multiple Items]], and [[write_requests#deleting_multiple_tags|Deleting Multiple Tags]]. Pass the current library version as ''If-Unmodified-Since-Version''. | See [[write_requests#deleting_multiple_collections|Deleting Multiple Collections]], [[write_requests#deleting_multiple_searches|Deleting Multiple Searches]], and [[write_requests#deleting_multiple_items|Deleting Multiple Items]]. Pass the current library version as ''If-Unmodified-Since-Version''. |
| |
| Example request: | Example request: |
Upgrade Storage