Monday, May 8, 2023

Ingesting Structured Descriptive Data into Pandoc

Ever since I found out about Structured Descriptive Data (SDD) I keep coming back to it. Right now I am planning to make two identification keys, one translated and modernized from a 1961 publication for which I have not found a good alternative, and one completely de novo (well, nearly). In both cases, I started writing out the key in a word processor with a numbered list and tabs and manual alignment etc., but in both cases that experience was pretty bad so far.

If the keys were interactive, multi-access keys there would be plenty of alternatives to word processors, including Xper3. However, these are both single-access, simple, dichotomous keys. There are CTAN packages to display such keys in LaTeX (biokey, dichokey), but those packages seem limited in several ways, and given how difficult LaTeX can be to parse, I would be locked into that choice. Normally I would encode the key data in some simple JSON format and generate HTML followed by a PDF, but at that point it makes as much sense to encode the data in SDD instead. One problem: how do I go from an SDD file to a PDF? Or HTML for a webpage, for that matter?


That is were Pandoc comes in. By creating a custom reader that transforms SDD files into an Pandoc AST, files can be generated as PDF, HTML, LaTeX, Word, ODT, and more. These custom readers allow for external dependencies so a pure-Lua XML parser library like xml2lua is easily imported. The challenge then becomes twofold:

  1. ‘Implementing’ the SDD format itself, and
  2. Building a layout that looks good in the various different formats, like HTML and PDF (which is LaTeX-based in Pandoc).

Firstly, SDD is in many places under-specified, showing only some suggested ways of encoding data, and not showing any examples in other places. This means that I personally need to decide on how I want certain data to be encoded in SDD (for the Pandoc reader), if multiple ways are possible.

Secondly, where in HTML (+ CSS) I could use float: right; or display: grid; to make pretty, dichotomous keys, or in LaTeX and for the PDF the environment longtblr from tabularray, a single design that works for both is more difficult. This is especially the case if you want to include numbered back-references, as it eliminates the possibility of using simple numbered lists.

Well, it turns out there is a trick for that. The AST supports RawInline, with which a reader can insert markup specific to the output format. pandoc.RawInline('html', '<span style="float: right;">) for HTML and pandoc.RawInline('tex', '\hfill') for LaTeX (and PDF), et cetera. It is still better to use regular Pandoc features as much as possible to allow for more flexibility in selecting output formats, but I feel like progressive enhancements like these are definitely “okay” to apply.

SDD limitations

In spite of the freedom and under-specification that SDD gives in some places, the format (as documented) is limited in other places. Apart from not being able to incorporate related formats from TDWG such as Darwin Core (other than through non-standard extensions), some of features are missing:

  • Endpoints of identification keys (<Lead>) can only have a <TaxonName> or a <Subkey>, not both, meaning that the many publications that list the key to genera separate to each genus subkey to species, cannot be encoded in an optimal way.
  • Endpoints of identification keys (<Lead>) are only shown to have one <TaxonName>, so keys leading to a species pair (or a differently-sized, non-taxonomic group of species) cannot be encoded. (At the same time, descriptions can be linked to multiple taxa with <Scope>.)
  • Descriptions (<NaturalLanguageData>) and identification keys are not shown to allow markup. Confusingly, the term “markup” is used, but as far as I can tell only for annotating the text with concepts and publications; not for putting taxon names in italics or other styling.


After quite some time I was finally content with the key that I was testing it with (Fig. 1). The resulting filter is available at, and will probably see continued development as I try to use it for other keys. A major bottleneck is still getting the SDD in the first place, but I hope that becomes easier as I continue developing scripts and tooling. Overall though, the Pandoc reader has been a great success.

Screenshot of page 17 and 18 of a document showing a key to the genera of Scutelleridae, a drawing of Odontoscelis fuliginosus, and the start of a key to Odontoscelis.
Figure 1: Page 17 and 18 of the PDF, made with Pandoc from a Structured Descriptive Data XML file. Translated, original text and figures from Puchkov, 1961 (Fauna of Ukraine, vol. 21 (1),

Wednesday, February 1, 2023

On the modelling of the content of identification keys

Last november I wrote a blog post about how to model the taxonomic coverage of identification keys. I wanted to model this coverage to be able to determine to what extent an identification key applies to a given observation or specimen, for use in my Library of Identification Resources project. For the same project I also find it useful to be able to archive identification keys. Many keys can be downloaded as PDFs or saved in the Internet Archive. However, some keys cannot be archived so easily:

  • Matrix-type keys (also known as multi-access keys) are usually presented as interactive apps. Sometimes these work from the Wayback Machine, but sometimes they dynamically load additional web resources in which case they do not work. Although this is less common, it also occurs for single-access keys.
  • Even non-interactive dichotomous keys can be spread out over multiple web pages. For example, in B477 each step has its own page. This can be crawled normally (although it is not yet in this example) but the key still is not in one place.
  • Some keys only exist in obscure print (or CD) sources or outdated or otherwise obscure file formats, where it may be useful to have a more standard format to archive or re-publish the data in (taking copyright into account of course).

In these cases it may be useful to archive the keys in a different format, especially if conserving the source is not feasible. In this blog post I will examine how I think I can best express these keys in more standard formats.

Metallic blue-green moth with feathered antennae, sitting on a vertical flower stem. In the out-of-focus background there is grass, other stems, and pink flowers.
Adscita statices, observed June 28th, 2020

The Library of Identification Resources supports 7 types of resource types: collection, checklist, gallery, reference, key, matrix, and supplement. The first one, collection, is used for pages that link to a variety of different resource that are not all in the same series, and can be modelled with schema:Collection or something similar. checklist is also relatively simple, and can be modelled with a list of entities dwc:Taxon, which could be linked to a schema:ItemList.

gallery, reference, key, and matrix are a bit more difficult to model. gallery and reference are both supersets of checklists, that respectively have just an image, or a description and optionally an image, a distribution range, and/or more info for each taxon; key are single-access keys; and matrix are multi-access keys. Regarding gallery: sure, has a type for schema:ImageGallery but that is specifically for websites, whereas gallery also includes booklets, flyers, and CD-based programs, and the schema:Collection type from before does not capture the cohesiveness in my opinion. reference, key, and matrix are even more difficult to capture accurately.

For these types, I would like to introduce Structured Descriptive Data (SDD)¹, a standard from TDWG from 2005 that defines an format to encode descriptions and single-access and multi-access keys in XML files. gallery, reference, key, and matrix can all be expressed in SDD files. However, I think it is missing some features.

  1. It is not linked data, so it is difficult to relate the content of the key with the metadata.
  2. SDD has a <Taxon> element but it would be great if SDD files were instead interoperable with the definitions from dwc:Taxon.
  3. For reference and gallery resources, it should be possible to specify the relation between an image and a taxon.
  4. For matrix keys, SDD has qualitative/discrete characters and quantitative/continuous characters. Characters relating to temporal, seasonal, and spatial distributions are common in matrix keys, but are difficult to express in SDD. I suggest temporal, seasonal, and spatial characters for both discrete and continuous characters. The former because old keys may for example have divided the continent into regions, but the user should be able to enter coordinates (so a regular discrete character does not suffice); the latter because newer keys may use the future to include more detailed distribution ranges.
  5. For matrix keys it may be useful to assign probabilities to discrete character states per taxon.

Lastly, supplement remains. This is a difficult one, even when just listing the taxonomic coverage. The goal when modelling this would be to list the differences between the original source and the “fixed” version in a standard format. If all original sources (i.e., the above types) could be expressed with linked data, the standard format could be the Linked Data Patch Format. Of course this would mean that unlike the other types, supplement would not be expressed in linked data, which is a bit unfortunate.

¹ At the time of writing, TDWG only hosts SDD Primer in the twiki source format on GitHub. I have taken the liberty to parse the twiki source and produce HTML:

Saturday, December 31, 2022

Citation.js: 2022 in review

Following up on the previous two updates this year (Version 0.5 and a 2022 update and Version 0.6: CSL v1.0.2), here are some updates from the second half of 2022, as well as some statistics.

Sapygina decemguttata, a small wasp, observed July 8th, 2021


  • The mappings of the Wikidata plugin were updated, especially to accomodate for software.
  • The default-locale setting of some CSL styles is now respected.

New plugins

New users

  • WikiPathways is in the process of moving to a new site which uses Citation.js to generate citations.
  • Page, R. D. (2022). Wikidata and the bibliography of life. PeerJ, 10, e13712. 10.7717/peerj.13712
  • Boer, D. (2021). A Novel Data Platform for Low-Temperature Plasma Physics. [Master’s thesis] Radboud University, Nijmegen, The Netherlands. URL


@citation-js/core was downloaded approximately 240,000 times in 2022, against 205,000 times in 2021. The legacy package citation-js was downloaded approximately 160,000 times (compared to 190,000 times in 2021). This is good, as it indicates people are starting to use the new system more often. Note that most downloads of citation-js also lead to a download of @citation-js/core so most people still use the legacy package. The shift to stop using citation-js seemed to have started in October 2021, which coincides with the time time I forgot to update that package for 2 months after releasing v0.5.2 of @citation-js/core.

Since February 2022 a relative decrease in downloads of @citation-js/plugin-wikidata compared to core is also visible. In December the Wikidata plugin was downloaded more than 50% fewer times, in fact. This is also good and the exact point of the modularisation introduced back in 2018: to let users choose which formats to include. The DOI plugin was similarly less popular, while the BibTeX and CSL plugins were — as expected — almost always included. Surprisingly, BibJSON, a very niche format, only had a 25% reduction (maybe due to confusion with BibTeX and the internal BibTeX JSON format), while RIS, after BibTeX the most common format, had a 50% reduction as well.

New Year’s Eve tradition

After the releases on New Year’s Eve of 2016, 2017, and 2021, this New Year’s Eve also brings the new v0.6.5 release which changes the priority of some RIS fields in very specific situations, most notably the date field of conference papers (now using DA instead of C2).

Happy New Year!

Friday, December 23, 2022

Explore identification keys on the world map

The website of the Library of Identification Resources has a new feature: a map view. The resources in the catalog are associated with a geographic scope to approximate which species in a taxonomic group can be identified. These geographic scopes are usually countries or continents but can also be subdivisions within countries, multinational ecological regions like mountain ranges, and biogeographical realms.

The scopes are also linked to Wikidata identifiers, and the Wikidata identifiers are linked to iNaturalist identifiers. This makes it possible to get GeoJSON of these places from the iNaturalist API. Then the places can be displayed with the Leaflet library, and shaded according to the number of resources associated with it.

Map of the world with certain continents, countries, and lower divisions with black outlines and colored in various shades of red depending on the number of resources. Europe, especially western Europe is the darkest red.
World map with countries, continents and other regions shaded according to the number of associated resources.

There are a number of places that are not displayed:

  1. The biogeographical realms are not all in iNaturalist (e.g. Indomalayan realm).
  2. Some places like have ambiguous definitions (e.g. the Middle East) making it difficult to decide whether to link their Wikidata & iNaturalist entities if different definitions are used.
  3. Other places are historical and are difficult to express in modern borders (e.g. Japanese Empire in ca. 1930).
  4. A number of resources have scopes that have never been expressed in political borders (e.g. “the coast of Italy”).

All in all 347 of the current 1286 resources are not displayed, though it should be noted that 72 of those have no scope (e.g. keys for all globally known species).

Either way, the map clearly shows the geographical bias: Western Europe, especially the Netherlands, is overrepresented. This is because I have been mostly adding Dutch resources and various series from the UK (RES Handbooks), France (Faune de France), Denmark (Danmarks Fauna), Sweden (online keys and Nationalnyckeln), and Norway (Norske Insekttabeller).

The map is available from the home page and at

Wednesday, November 9, 2022

On the modelling and application of the taxonomic coverage of identification keys

The main feature of the Library of Identification Resources is the description of the identification key (or matrix, reference, etc.). This description should on its basis specify when the key can or should be used. I have initially split this description into the taxonomic coverage and the ‘scope’. The latter includes life stage and sex but also some restrictions on the taxonomic coverage that are more difficult to characterize, like “species that cause galls on plants in the Rosa genus” or “species that live in aquatic environments”.

Hairy, black moth with long antennae with a clubbed end, and narrow, blue-speckled wings with bright red spots, with a bright yellowish green background consisting of out-of-focus grass
Zygaena filipendulae (Linnaeus, 1758)

Simple solution

For the taxonomic coverage sensu stricto, I started with the parent taxon, e.g. the family or genus. However, many keys do not treat all the species in a family, and are instead limited by a geographic scope. This geographic scope should clearly also be included. Then there are more casual keys that can be very useful but may not be complete even at the time of publication, either excluding some rare species or only including some common species. This can be detailed (to some extent) with an incomplete/complete switch. Finally, although many keys are for species, there are some keys primarily for identifying genera, families, or other ranks. Below are some example works where these aspects do or do not apply.

Title Parent taxon Geographic scope Incomplete Target rank
B460: A revision of the world Embolemidae (Hymenoptera Chrysidoidea) Embolemidae
B1: Identification Key to the European Species of the Bee Genus Nomada Scopoli, 1770 (Hymenoptera: Apidae), Including 23 New Species Nomada Scopoli, 1770 Europe
B81: Key to some European species of Xylomyidae Xylomyidae Europe Yes
B63: MOSCHweb - Interactive key to the genera of the Palaearctic Tachinidae (Insecta: Diptera) Tachinidae Palaearctic realm genus

If only it were that simple

It became clear quite quickly that this is not enough. For one, parent taxon, geographic scope, and target rank should be able to contain multiple values. Additionally, as we saw before, some taxonomic coverages like “gallers on Rosa sp.” cannot be captured with these parameters unless the “Parent taxon” list gets very long and detailed.

Another, more common problem is that even the combination of a parent taxon and a geographic scope is hardly specific enough to be able to say whether an identification key is reliable and complete for a situation. Species are discovered, migrate, emigrate, and become extinct. Taxa are moved to different genera, families, and orders. In B659: Orthoptêres et Dermaptêres (Faune de France 3) from 1922, the order Orthoptera also contains Dictyoptera, currently a superorder consisting of cockroaches, stick insects, praying mantises, and more. This is a big problem too. Key questions that should be answerable are “How well does a British key apply to a Dutch observation?” and “How well does a British key from 1950 apply to a British observation from 2020?”

To account for the changes within higher-level taxa you might want to make a list of species that are included in the keys. For B81 for example, that could look like this:

Screenshot of a webpage titled "Identification resource B81:1", linked below. The relevant part of the webpage is a tree of taxa, consisting of one family containing two genera, containing 2 and 1 species respectively.

This gives a very clear image of the taxonomic coverage of the key: it includes three species, Salva marginata, S. varia and Xylomya maculata. The inclusion of the species in this key has a permalink, and the taxon is linked to a GBIF identifier. The list of species can then be matched (especially with the identifiers) to current checklists for the region that the observation was made in. More on that later.

This solution can easily be used for more complex taxonomic coverages, like the aforementioned key for species that cause galls on Rosa species. The ‘species’ list can also be a taxon list and have e.g. genera as the lowest rank. Another advantage is that there is no longer a need to explicitly specify the geographic scope, whether or not the key was known to be incomplete at the time of writing, and what the target taxon rank of the key is. In addition, this specific implementation also allows for multiple keys per work, which has various uses.

Two purposes

One problem that still comes up is that these taxon lists have two purposes:

  1. They describe which taxa can be distinguished with the key.
  2. They describe which taxa are considered when making the key.

These two are at odds. In a key to genera it would be simple to make a list of these genera, fulfilling (1) but not (2): if a species in of those genera gets moved to a different one, or if an additional species in one of the existing genera appears, the key ‘breaks’. The latter could be fixed by just listing all species but at that point you might as well fix both problems with (2) by grouping the species by genus. To simultaneously fulfill purpose (1) and (2) it is necessary to divide the taxonomic tree into units that can be distinguished by the key.

Even when fulfilling purpose (1) it is still possible to partially accomplish (2) in keys for e.g. families that key out to species. If, when making the species list, higher ranks such as genera are included, the key to genera can be validated according to (2) without having to go into more detail than the key. Therefore it is important to capture the entire taxonomic tree as presented in the work.

Another thing that might be useful to record is the synonyms. Many of the more academic works publish along with the key a series of descriptions including synonyms. If the status of that synonymic taxon is now different from the status in the key, this may be of importance to the results of the identification. It comes down to the same split of purposes: (1) no distinction is made in the key between the two taxa but (2) both taxa are, in theory, considered. Either way it is important to list both in some way.

If only it were that simple

At this point it became clear which taxa are necessary to include in such a list. But how to describe the taxa in those lists? This is an (enormous) rabbit hole in and of itself, and if I had more experience I could make a Falsehoods Programmers Believe About Names-style post about it. It starts with the normal parts: every taxon has a name, an author, a year, and a rank. That rank can be kingdom, phylum, class, order, family, genus, or species. But there is a seemingly endless list of increasingly obscure ranks, sometimes only in use by one or a few authors (what is a “stirps”??); taxa can have multiple authors and different ways of presenting this (et/and/& and et al. or listing everyone); the names themselves can be spelled in different ways and include or omit initials; the year can be very unclear, especially for taxa published in older works published over multiple years; and the scientific names themselves can have different spellings, capitalizations, and hyphenations as well.

As with personal names, the ‘trick’ is to trust the source to some extent, and avoid focussing on the specifics of the scientific name. After all, there is no reason to make the computer understand the scientific names. The goal is to allow for matching (and there are ways to do that without trying to standardize everything) and, secondarily, to present the names in such a way that humans can understand them (and the original key already did that).

Measuring applicability (more problems)

Now, with a complete and structured taxonomic tree, the question becomes how to check whether it applies to a certain observation. The idea is to compare the taxon list to a list of all the species that a certain observation could be. But where does such a list come from? The simple solution would be a checklist: a professional, thorough list of all the species that are known to occur in a certain area.

If such a checklist is not (easily) available (I am not aware of a global database of checklists, let alone in a standardised format), it might be tempting to create one from a map of observations from e.g. GBIF. The advantage of this is that you have, to some extent, global coverage and a standardised format. However, this has some caveats. The correctness of the resulting checklist is entirely dependent on the quality and quantity of the observations in the region. To get a higher sensitivity it may be useful to instead make a checklist of a larger, encompassing region, but this lowers the specificity. A species occurring in Belgium might only have recorded observations in the Netherlands on GBIF. The same goes for the time scale. There might be museum specimens of species that are now extinct in the region, but there might also be rare species that are only seen every 50 years or so.

A good addition (or a risky alternative) to a checklist would be a measure of the (relative) abundance of species in the region, and weighing the taxon lists of the keys according to that. This prioritises keys that include more common species over keys that include more rare species. Of course, the key of common species can be wrong about an observation of a rare species. Another problem is how to determine the (relative) abundance. Again it is tempting to derive this from GBIF observations, but a species with a lot of observations is not necessarily more common. It might be the focus of an observation campaign, or national attention of a different kind, it might sit still more often, it might be easier to recognize, or it might even be easier to identify to a species level.

An additional possibility of measuring applicability is bringing the scope restrictions mentioned at the start into this. Apart from belonging to a certain taxon, the observed organism also has a biological sex, a life stage, and more characteristics that may restrict which keys can be used. To compare these characteristics between keys and organisms they need to be described with a common, consistent vocabulary. iNaturalist has such a vocabulary (the “Annotations”), but there might be more suitable ones out there somewhere.

(For this it might also be useful to map keys that distinguish castes of ants, males/females of solitary bees, or life stages of shield bugs. How do you model that? Not in the same way as described above, that is for sure.)

Using the data

As I teased in the previous blog post about the Library of Identification Resources, I have started to collect this data, and attempted to recruit some others to contribute as well. All works which have their keys (and matrices, checklists, descriptions) mapped out can be found by searching for “TRUE” in the “Tax. data extr.” field. To get from the work page to a taxa list, look for the row titled “Resources” in the first table on the page. If available it lists the individual resources in the work for which the taxonomic data is extracted.

The pages for the taxonomic data of the individual resources contains some basic information about the work (as well as a link), as well as the page numbers of the resource within the work if available. There is also some info on the resource, derived from the same info in the work unless different for that resource. Then there is a list of taxa, displayed as a tree. Each taxon has a rank, an anchor, and if available a link to the corresponding taxon in GBIF.

The data is also used in the proof-of-concept app that I introduced in the previous blog post. Searching for a taxon and coordinates will now query GBIF for observations in that taxon in the country encompassing the coordinates, and match this with the GBIF identifiers in the taxon lists. It displays the relative amount of taxa in the ‘checklist’ that are also in the key as a percentage and a small pie chart. It does not yet deal with synonyms.

Saturday, August 6, 2022

Library of Identification Resources

Since around this time last year, I have been working on creating a library of identification resources. Here, “identification resources” are identification keys, multi-access (matrix) keys, other works that can aid in the identification of species. The project is managed on GitHub:

Digital drawing of a black moth with red spots and light speckles on the forewings and red hindwings. Below the moth is a grey bar signifying a label, and the whole picture has a light beige background.
The logo: Zygaena filipendulae (Linnaeus, 1758)

About the project

Right now the project mainly consists of a catalog, a list of works (or rather “manifestations” in the FRBR model) that contain identification resources. The catalog contains bibliographic information, but also a summary of the identification resource(s) that they contain.

This summary consists of the starting taxon or taxa (e.g. Chrysididae), a region that the resource applies to (e.g. Europe), an optional scope (e.g. adults, males and females), the taxonomic rank(s) that the resource distinguishes (usually species), and whether the key was supposed to be complete for the region-scope combination at the time of publication.

This data is similar to that provided by BioInfo UK (example). They actually go in more detail, with more nuance in completeness, the equipment required for using the key (e.g. stereo microscope), whether and what specimen preparation is required, and expert reviews. The Library of Identification Resources does not include this information, but has a broader geographical scope¹ and can be searched in a more structural way.

The places (for the geographical scope of keys) and the authors and publishers linked to works as well as the works themselves are linked to Wikidata entities wherever possible. Journals and book series are linked to ISSNs.

Using the data

On top of this catalog, I built a website: This provides a user interface, allowing for structured searches through the catalog, linking multiple versions (FRBR “manifestation”) of the same work (FRBR “work”), a few graphs exploring the distribution of languages, licenses, etc., and some additional functionality like exporting citations.

To improve searching, I am also working on an app that lists the most relevant resources for a given situation. In the proof-of-concept (GitHub) users can enter a parent taxon and a coordinate location and get (possibly) applicable resources for that situation. This uses the GBIF API for taxon searches, and the iNaturalist API for looking up places around coordinates.

Results are sorted according to a few heuristics, including:

  • The availability of full text.
  • How close the key taxon is: a key for Vespidae is likely better than one for Insecta, when identifiying Vespa.
  • How recent the resource is.
  • Whether the level of detail indicates that the key can actually improve on the parent taxon.
  • Whether the resource was considered complete during publication.
  • Whether the resource is a (matrix) key, just a reference or a photo gallery, or even a just collection of other resources (which are not guaranteed to contain anything relevant, and if they did, they would likely show up in the results already).

The data is currently biased towards the Netherlands and the rest of western Europe, as well as towards insects and mainly Hymenoptera. This is because I have mainly worked on adding resources that I was using, or resources referenced therein.

Screenshot of a website. The header has a logo of a moth, the text "Library of Identification Resources" and three links, "Catalog", "Statistics", and "GitHub". In the main part: in the top left is a paragraph of text; in the top right two input fields labeled "Taxon" and "Location" and a button labeled "Search"; on the left is a table listing items and on the right is an embedded PDF viewer
The proof-of-concept app in action.

Future plans

Improving the interface

I want to improve the interface of the proof-of-concept app and I have a number of ideas for this:

  • Allow users to “upload” photos. These would not be uploaded to a server but rather just displayed locally, but could serve a couple of purposes:
    • Automatic extraction of coordinates from EXIF data
    • Automatic determination of a parent taxon using computer vision [needed: a computer vision model :)]
    • Viewing the key side-by-side with the photos.
  • Same, but by entering a GBIF/iNaturalist observation ID.

Improving the data model

One of the main improvements that I want to make is modeling of individual keys/resources within works for two reasons:

  1. Works can contain multiple resources with completely different properties. If a work contains a key of Family A to Genus B and C, a key to the species of Genus B, and a key to the species of Genus C, those can be modeled as a single key to the species of Family A (assuming B and C are the only relevant genera). But this is not always the case. In addition, a work can contain resources of different types, like a key and a checklist (B159).
  2. Listing the taxa in the resource. I am aware of a few problems with this, but a list of species in a key can be matched to a modern checklist. This gives a better idea about how well a British key could apply to a Dutch observation (or an older British key to a more recent British observation, with all the species migration going on).

A screenshot of a collapsible taxonomic tree, where each taxon has a taxon name, a rank in grey text, a link icon and usually a GBIF icon
Example of what this would look like.

Contributing data

You are welcome to contribute data to the catalog. I plan to make this easier in the future, as currently the master copy of the data is still in the non-version-controlled spreadsheet that I started the work in.

¹ Note that the current data has a geographical bias, but this is not the same as a strict scope in my opinion.

Tuesday, May 31, 2022

Citation.js Version 0.6: CSL v1.0.2

Since the citation-js npm package was first published, version 0.6 is the first major version of Citation.js that did not start out as a pre-release. Version 0.3 itself spent almost 6 months in pre-release, but only received updates for less than half a month. Version 0.4 spent more than a year in pre-release and received updates for about 4 months. Version 0.5 takes the cake with one and a half years in pre-release, receiving updates for a year, also making it the best-maintained version.

Yellow flowers with lots of little "rays" on greenish brown stems

Tussilago farfara, March 27th, 2022

Version 0.6 is a major version bump because it introduces a number of breaking changes, including raising the minimal Node.js version to 14. Since April 2022, Node.js 12 is End-Of-Life, which led to a lot of dependencies dropping support. Now, Citation.js does so too. Other changes include the following:

Update data format to CSL v1.0.2

The internal data format is now updated from CSL v1.0.1 to v1.0.2. This introduces the software type and the generic document type, as well as some other types, and some new fields. The event field is also renamed to event-title. That, and software replacing book, makes it so that CSL v1.0.2 is not compatible with CSL v1.0.1 styles, making it a breaking change.

  • CSL data is now automatically upgraded to v1.0.2 on input.
  • Cite#data ((new Cite()).data) now contains CSL v1.0.2 data.
  • Output formatters of plugins now receive CSL v1.0.2 data as input.
  • util (import { util } from '@citation-js/core') now has two functions, downgradeCsl and upgradeCsl, to convert data between the two versions.
  • The data formatter (.format('data')) now takes a version option. When set to '1.0.1', this downgrades the CSL data before outputting.
  • @citation-js/plugin-csl already automatically downgrades CSL to v1.0.1 for compatibility with the style files.
  • Custom fields are now generally put in the custom object, instead of prefixing an underscore to the field name.

The mappings are also updated. Especially the RIS and BibLaTeX mappings were made more complete by the increased capabilities of the CSL data schema. Non-core plugins are also being updated, mainly affecting @citation-js/plugin-software-formats and @citation-js/plugin-zotero-translation-server.

Test coverage

While updating the plugin mappings, the test suites of the plugins were also expanded. This led to the identification of a number of bugs, that were also fixed in this release:

  • BibLaTeX
    • handling of CSL entries without a type
    • handling of bookpagination
    • handling of masterthesis
  • RIS
    • RegExp pattern for ISSNs
    • Name parsing of single-component names

Closing issues

A number of issues were also fixed in this release:

  • Adding full support for the Bib(La)TeX crossref field
  • Mapping BibLaTeX eid to number instead of page
  • Adding a mapping for the custom BibLaTeX s2id field
  • In Wikidata, getting issue/volume/page/publication date info from qualifiers as well as top-level properties.

CSL styles

The bundled styles (apa, vancouver, and harvard1) were updated. Note that harvard1 is now an alias for harvard-cite-them-right. Quoting the documentation:

The original “harvard1.csl” independent CSL style was not based on any style guide, but it nevertheless remained popular because it was included by default in Mendeley Desktop. We have now taken one step to remove this style from the central CSL repository by turning it into a dependent style that uses the Harvard author-date format specified by the popular “Cite Them Right” guide. This dependent style will likely be removed from the CSL repository entirely at some point in the future., CC-BY-SA-3.0

Looking forward

Some breaking changes are still pending, mainly changes to the plugin API and the removal of some left-over APIs. However, I also want to work on a more comprehensive format for machine-readable mappings, a format for mappings for linked data, and of course implementing more mappings in general!