Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
kb:mendeley_import [2018/08/28 21:37]
dstillman [Extracting Mendeley Annotations and Highlights]
kb:mendeley_import [2019/02/28 21:11] (current)
dstillman [Preserving Mendeley Annotations and Highlights] Update Mendeley data directory link
Line 5: Line 5:
 ===== Mendeley Database Encryption ===== ===== Mendeley Database Encryption =====
  
-Mendeley 1.19 and later have begun encrypting the local database, making it unreadable by Zotero and other standard database tools. Elsevier made this change a few months after Zotero publicly announced work on an importer, despite having long touted the openness of its database format as a guarantee against lock-in and [[https://​service.elsevier.com/​app/​answers/​detail/​a_id/​22098/​kw/​database/​supporthub/​mendeley/​|erroneously continuing to state in its documentation that the database can be accessed using standard tools]]). At the same time, Mendeley continues to import data from Zotero’s own open database, as it has since 2009.+Mendeley 1.19 and later have begun encrypting the local database, making it unreadable by Zotero and other standard database tools. Elsevier made this change a few months after Zotero publicly announced work on an importer, despite having long touted the openness of its database format as a guarantee against lock-in and [[https://​service.elsevier.com/​app/​answers/​detail/​a_id/​22098/​kw/​database/​supporthub/​mendeley/​|erroneously continuing to state]] in its documentation that the database can be accessed using standard tools. At the same time, Mendeley continues to import data from Zotero’s own open database, as it has since 2009.
  
 The [[https://​www.mendeley.com/​release-notes/​v1_19|Mendeley 1.19 release notes]] claimed that the encryption was for “improved security” on shared machines, yet applications rarely encrypt their local data files, as file protections are generally handled by the operating system with account permissions and full-disk encryption, and anyone using the same operating system account or an admin account can already install a keylogger to capture passwords. Elsevier later [[https://​twitter.com/​mendeley_com/​status/​1006915998841221120|stated]] that the change was required by new European privacy regulations — a bizarre claim, given that those regulations are designed to give people control over their data and guarantee data portability,​ not the opposite — and continued to assert, falsely, that full local export was still possible, while [[https://​twitter.com/​mendeley_com/​status/​1006919608471818240|repeatedly]] [[https://​twitter.com/​MendeleySupport/​status/​1006920802120470528|dismissing]] reports of the change as "#​fakenews"​. The [[https://​www.mendeley.com/​release-notes/​v1_19|Mendeley 1.19 release notes]] claimed that the encryption was for “improved security” on shared machines, yet applications rarely encrypt their local data files, as file protections are generally handled by the operating system with account permissions and full-disk encryption, and anyone using the same operating system account or an admin account can already install a keylogger to capture passwords. Elsevier later [[https://​twitter.com/​mendeley_com/​status/​1006915998841221120|stated]] that the change was required by new European privacy regulations — a bizarre claim, given that those regulations are designed to give people control over their data and guarantee data portability,​ not the opposite — and continued to assert, falsely, that full local export was still possible, while [[https://​twitter.com/​mendeley_com/​status/​1006919608471818240|repeatedly]] [[https://​twitter.com/​MendeleySupport/​status/​1006920802120470528|dismissing]] reports of the change as "#​fakenews"​.
Line 16: Line 16:
  
   * Mendeley stores PDF annotations and highlights in its database rather than storing them in the file where they would be accessible to other PDF readers, so Zotero extracts annotations to a note stored under the item, with links back to the original page. Highlights are not imported. To embed annotations and highlights into the PDF files themselves so they can be imported into Zotero, you have a couple options:   * Mendeley stores PDF annotations and highlights in its database rather than storing them in the file where they would be accessible to other PDF readers, so Zotero extracts annotations to a note stored under the item, with links back to the original page. Highlights are not imported. To embed annotations and highlights into the PDF files themselves so they can be imported into Zotero, you have a couple options:
-    - Mendeley provides an option to generate a PDF with annotations and highlights added back to the file, but it can only be run for one file at a time. If you’d like to import PDFs with annotations and highlights, you could use that option for select ​files and then replace the imported files in Zotero (right-click on the item and choose “Show File”) with the annotated versions. +    - Mendeley provides an option to generate a PDF with annotations and highlights added back to the file. Prior to Mendeley 1.19.3that option could only be run for one file at a time; in 1.19.3, it can be run for multiple files but doesn’t include highlights. In our testing, exported PDFs also don’t always contain the text of annotationsBut if you’re able to satisfactorily export ​annotations and/or highlights, you could use that option for one or more files and then replace the imported files in Zotero (right-click on the item and choose “Show File”) with the annotated versions. 
-    - You can also try using the third-party [[https://​github.com/​cycomanic/​Menextract2pdf|Menextract2pdf]] script to write annotations and highlights back to the PDFs in Mendeley before importing into Zotero. See the [[#Extracting_Mendeley_Annotations_and_Highlights|instructions below]].+    - You can also try using the third-party [[https://​github.com/​cycomanic/​Menextract2pdf|Menextract2pdf]] script to write annotations and highlights back to the PDFs in Mendeley before importing into Zotero. See the [[#preserving_mendeley_annotations_and_highlights|instructions below]].
   * It’s not possible to directly import group libraries. To import items in group libraries, simply copy the group items to a collection in your Mendeley library before importing. You can then create a Zotero group and drag imported collections or items to that group.   * It’s not possible to directly import group libraries. To import items in group libraries, simply copy the group items to a collection in your Mendeley library before importing. You can then create a Zotero group and drag imported collections or items to that group.
   * Mendeley allows any field to be added to any type. When importing into Zotero, if a field isn’t valid for a given item type, the field is placed into the Extra field. When possible, those will be used automatically in citations (e.g., Original Date), and future versions of Zotero will automatically convert those to any real fields that become available.   * Mendeley allows any field to be added to any type. When importing into Zotero, if a field isn’t valid for a given item type, the field is placed into the Extra field. When possible, those will be used automatically in citations (e.g., Original Date), and future versions of Zotero will automatically convert those to any real fields that become available.
-  * When using the Zotero word processor plugins, document citations created with Mendeley won’t currently be linked to imported citations in your Zotero database. ​We may be able to add support for this in a future version.+  * When using the Zotero word processor plugins, document citations created with Mendeley won’t currently be linked to imported citations in your Zotero database. ​Zotero'​s word processor plugins can, however, read Mendeley citations and their embedded metadata, so you can continue using the same documents with Zotero.
  
 ===== The Import Process ===== ===== The Import Process =====
Line 34: Line 34:
 ** If you’ve already upgraded to 1.19 and have an automatic backup of your Mendeley SQLite database in the Mendeley data directory:​** ** If you’ve already upgraded to 1.19 and have an automatic backup of your Mendeley SQLite database in the Mendeley data directory:​**
  
-Locate your [[https://​service.elsevier.com/​app/​answers/​detail/​a_id/​22098/​kw/​database/​supporthub/​mendeley/|Mendeley data directory]] and look for a backupSlot1 (or higher) folder. If you see a database in the form %%<​email>​@www.mendeley.com.sqlite%% within one of the backup directories,​ copy it into the data directory one level up. Then start the import in Zotero by going to File → “Import…”,​ choosing the “Mendeley” option, and selecting that database.+Locate your [[https://​service.elsevier.com/​app/​answers/​detail/​a_id/​18153/|Mendeley data directory]] and look for a backupSlot1 (or higher) folder. If you see a database in the form %%<​email>​@www.mendeley.com.sqlite%% within one of the backup directories,​ copy it into the data directory one level up. Then start the import in Zotero by going to File → “Import…”,​ choosing the “Mendeley” option, and selecting that database.
  
 If the backupSlot//​N//​ folder contains an .sqlite filename with a long string of random characters, the backup has already been overwritten by the encrypted database. Check another backupSlot folder or use one of the methods below. If the backupSlot//​N//​ folder contains an .sqlite filename with a long string of random characters, the backup has already been overwritten by the encrypted database. Check another backupSlot folder or use one of the methods below.
Line 56: Line 56:
  
   - Make sure you've synced all data and — if you want to avoid reassociating files manually — files to Elsevier'​s servers.   - Make sure you've synced all data and — if you want to avoid reassociating files manually — files to Elsevier'​s servers.
-  - Move your database out of the [[https://​service.elsevier.com/​app/​answers/​detail/​a_id/​22098/​kw/​database/​supporthub/​mendeley/|Mendeley data directory]] to a temporary location.+  - Move your database out of the [[https://​service.elsevier.com/​app/​answers/​detail/​a_id/​18153/|Mendeley data directory]] to a temporary location.
   - Download Mendeley 1.18.   - Download Mendeley 1.18.
   - Perform a fresh sync to pull down your Mendeley data from the Elsevier servers.   - Perform a fresh sync to pull down your Mendeley data from the Elsevier servers.
Line 84: Line 84:
 As [[#​known_issues|noted above]], Mendeley stores PDF annotations and highlights in its own database rather than in the PDF itself where they would be accessible to other PDF readers, and while Zotero will extract annotations to child notes with links back to the PDF, embedded annotations and highlights will not be imported. As [[#​known_issues|noted above]], Mendeley stores PDF annotations and highlights in its own database rather than in the PDF itself where they would be accessible to other PDF readers, and while Zotero will extract annotations to child notes with links back to the PDF, embedded annotations and highlights will not be imported.
  
-To use the third-party Menextract2pdf script to save annotations and highlights back to all of the PDFs in your library before importing into Zotero, follow these steps:+To use the third-party ​[[https://​github.com/​cycomanic/​Menextract2pdf|menextract2pdf]] ​script to save annotations and highlights back to all of the PDFs in your library before importing into Zotero, follow these steps:
  
 ==== macOS or Linux ==== ==== macOS or Linux ====
  
   - Close Mendeley   - Close Mendeley
-  - Make a backup of your [[https://​service.elsevier.com/​app/​answers/​detail/​a_id/​22098/​kw/​database/​supporthub/​mendeley/|Mendeley data directory]],​ including the ''​Downloaded''​ folder that holds your PDF files (and other locations on your computer where you have PDF files you wish to save Mendeley annotations to) +  - Make a backup of your [[https://​service.elsevier.com/​app/​answers/​detail/​a_id/​18153/|Mendeley data directory]],​ including the ''​Downloaded''​ folder that holds your PDF files (and other locations on your computer where you have PDF files you wish to save Mendeley annotations to) 
-  - Download and unzip the Mextract2pdf ​script from [[https://​github.com/​bwiernik/​Menextract2pdf/​archive/​master.zip|here]] +  - Download and unzip the menextract2pdf ​script from [[https://​github.com/​cycomanic/​Menextract2pdf/​archive/​master.zip|here]] 
-  - Open Terminal, navigate to the unzipped ​Mextract2pdf ​folder, and run the following command (filling in the correct file paths):+  - Open Terminal, navigate to the unzipped ​Menextract2pdf ​folder, and run the following command (filling in the correct file paths):
     * <​code>​menextract2pdf_overwrite.sh "/​Path/​To/​Mendeley/​Data Folder/"​ "/​Path/​To/​Mendeley/​PDF/​Folder/"</​code>​     * <​code>​menextract2pdf_overwrite.sh "/​Path/​To/​Mendeley/​Data Folder/"​ "/​Path/​To/​Mendeley/​PDF/​Folder/"</​code>​
-  - For example, on macOS, if you have your Mendeley database in the [[https://​service.elsevier.com/​app/​answers/​detail/​a_id/​22098/​kw/​database/​supporthub/​mendeley/|default location]] and your PDFs stored in Mendeley'​s "​Downloaded"​ folder, you would enter:+  - For example, on macOS, if you have your Mendeley database in the [[https://​service.elsevier.com/​app/​answers/​detail/​a_id/​18153/|default location]] and your PDFs stored in Mendeley'​s "​Downloaded"​ folder, you would enter:
     * <​code>​menextract2pdf_overwrite.sh "/​Users/"​Your Name"/​Library/​Application Support/​Mendeley Desktop"​ "/​Users/"​Your Name"/​Library/​Application Support/​Mendeley Desktop/​Downloaded/"</​code>​     * <​code>​menextract2pdf_overwrite.sh "/​Users/"​Your Name"/​Library/​Application Support/​Mendeley Desktop"​ "/​Users/"​Your Name"/​Library/​Application Support/​Mendeley Desktop/​Downloaded/"</​code>​
  
Line 99: Line 99:
   - Install [[https://​www.python.org/​downloads/​|Python]]   - Install [[https://​www.python.org/​downloads/​|Python]]
   - Close Mendeley   - Close Mendeley
-  - Make a backup of your [[https://​service.elsevier.com/​app/​answers/​detail/​a_id/​22098/​kw/​database/​supporthub/​mendeley/|Mendeley data directory]],​ including the ''​Downloaded''​ folder that holds your PDF files (and other locations on your computer where you have PDF files you wish to save Mendeley annotations to) +  - Make a backup of your [[https://​service.elsevier.com/​app/​answers/​detail/​a_id/​18153/|Mendeley data directory]],​ including the ''​Downloaded''​ folder that holds your PDF files (and other locations on your computer where you have PDF files you wish to save Mendeley annotations to) 
-  - Download and unzip the Mextract2pdf ​script from [[https://​github.com/​bwiernik/​Menextract2pdf/​archive/​master.zip|here]] +  - Download and unzip the menextract2pdf ​script from [[https://​github.com/​cycomanic/​Menextract2pdf/​archive/​master.zip|here]] 
-  - Open Command Prompt, navigate to the unzipped ​Mextract2pdf ​folder, and run the following command (filling in the correct file paths):+  - Open Command Prompt, navigate to the unzipped ​Menextract2pdf ​folder, and run the following command (filling in the correct file paths):
     * <​code>​menextract2pdf_overwrite.bat "​C:​\Path\To\Mendeley\Data Folder\"​ "​C:​\Path\To\Mendeley\PDF\Folder\"</​code>​     * <​code>​menextract2pdf_overwrite.bat "​C:​\Path\To\Mendeley\Data Folder\"​ "​C:​\Path\To\Mendeley\PDF\Folder\"</​code>​
   - For example, if you have your Mendeley database in the [[https://​service.elsevier.com/​app/​answers/​detail/​a_id/​22098/​kw/​database/​supporthub/​mendeley/​|default location]] and your PDFs stored in Mendeley'​s "​Downloaded"​ folder, you would enter:   - For example, if you have your Mendeley database in the [[https://​service.elsevier.com/​app/​answers/​detail/​a_id/​22098/​kw/​database/​supporthub/​mendeley/​|default location]] and your PDFs stored in Mendeley'​s "​Downloaded"​ folder, you would enter:
kb/mendeley_import.1535506632.txt.gz · Last modified: 2018/08/28 21:37 by dstillman