Translations of this page:

We’re in the process of updating the documentation for Zotero 5.0. Some documentation may be outdated in the meantime. Thanks for your understanding.

A very brief introduction to some commonly used technologies used in translator development.

XPath

XPath provides a way to refer to specific parts of HTML or XML documents. It's usually the best way to extract data from webpages when writing a translator.

An XPath expression is a chain of pieces that specify the path to a node of the document. The main pieces of expressions are:

  • / Separator between parts of the path
  • * match any tag
  • // one or more levels deeper
  • .. go up one level
  • [] match a tag that has this (the contents of the brackets)
  • @key an attribute named key
  • text() match a text node
  • [2] match the second matching node
  • [last()] match the last matching node
  • div[@class=“important”] match a <div> with the attribute class, with the value important.
  • td[contains(text(),“Expect”)] match a <td> which contains text that contains “Expect”
  • Plus much more. See the XPath specification and the XPath documentation of the Mozilla Developer Network.

The best introduction to XPath for use in translators is Mozilla's Introduction to using XPath in JavaScript, but it may be even easier to model your code off of the logic in existing translators, which provide a wide array of XPath techniques to pick apart fussy sites.

Examples

<html>
 <body>
 <div id="names">
   <span class="editor">George Spelvin</span>,
   <span class="translator">Andrea Johnson</span>
 </div>
 <table>
  <tr class="odd">
   <td>Great Expectations</td>
   <td>Mediocre Plans</td>
  </tr>
 </table>
 </body>
</html>

For the sample document above, these expressions would refer to…

  • //tr[@class="odd"]/td: a result set with the nodes <td>Great Expectations</td> and <td>Mediocre Plans</td>
  • //table//td: Same as previous
  • //table//td[last()]: a result set with the single node <td>Mediocre Plans</td>
  • //span[@class="editor"]: a result set with the single node <span class=“editor”>George Spelvin</span>

In Translators

When writing Zotero translators, XPath expressions can be used with the builtin evaluate(..) method on DOM document objects:

var xpath = '//div[@class="tableofcontents-row"][div/input[@type="checkbox"][@name="toView"]]';
var tableRows = doc.evaluate(xpath, doc, nsResolver, XPathResult.ANY_TYPE, null);

Then iterate through the rows using tableRows.iterateNext():

while (var row = tableRows.iterateNext()) {
    // do something with each row
}

Starting with Zotero 2.1.9, there are two new utility functions for XPath, ZU.xpath(..) and ZU.xpathText(..). See the utility function documentation for more information.

var xpath = '//div[@class="tableofcontents-row"][div/input[@type="checkbox"][@name="toView"]]';
var tableRows = Zotero.Utilities.xpath(doc, xpath);
for (var row in tableRows) {
    // do something with each row
}

Using Firefox Inspect Element to design XPath or CSS Selector expressions

In Firefox, you can right click on an element and select “Inspect Element”. A pane with the relevant bit of the site's code highlighted will open. Right-click on the highlighted part and select Copy –> Xpath (or CSS Selector/CSS Path)

Then we can paste the XPath expression into the editor (i.e., the code pane of Scaffold):

/html/body/table/tbody/tr[5]/td[3]/div/table/tbody/tr/td/div/table[3]/tbody/tr[4]/td/a[2]

XPath like this, however, is very fragile. It's better to latch onto some identifiers and make an expression relative to them:

//div/table/tbody/tr/td/div/table[3]/tbody/tr[4]/td/a[2]

and even better to rely on specific attributes in a much shorter Xpath such as

//td[@type="article-link"]/a[2]

Regular Expressions

Regular expressions are a way of matching and extracting pieces of text from a larger body of text. In translator development, they can be very useful for pulling out important pieces of text from parts of a webpage that aren't meaningfully marked up in HTML. They are also the only way to work with imported data that isn't in HTML or XML. Finally, every web translator uses regular expressions in its target expression.

  • . matches any character
  • [a-z01] matches any of the lowercase English letters and the numbers 0 and 1
  • () surround a match expression
  • + Match one or more of the preceding expression
  • * Match 0 or more of the preceding expression
  • ? Match 0 or 1 of the preceding expression

For a complete description of how regular expressions are used in JavaScript, with examples of their use, see Mozilla's Regular Expressions Guide.

There are many regular expressions guides on the web, and entire books on the topic. While you can and should consult these guides, keep in mind that the details of how they work will depend a little on the environment they're running in– JavaScript does not support everything that you might read about regular expressions in, say, a book for Perl or Java. The Mozilla guide above describes everything that can be done in JavaScript.

dev/technologies.txt · Last modified: 2017/11/18 16:29 by adamsmith