This post is a follow on from a previous one where I discuss mapping the Rhododendron monographs to EoL. It is an opinionated rant but I offer it in the hope that it will be of some use.

When monographs/floras/faunas are mentioned in the context of digitization people usually chirp up with PDF or, if they are more clued up on biodiversity informatics,  TaXMLit and INOTAXA (Hi to Anna if you are reading) or TaxonX and Plazi.org (Hi to Donat).  The point I am going to make in this blog post is not against these ways of marking up taxonomic literature but more the nature of the monographic/floristic/faunistic taxonomic product itself. I am far more familiar with the botanical side of things so apologies to zoologists in advance.

The problem is what I call the “narrative form” of monographic data not whether it is available in print, pdf, ebook or lovingly marked up XML. These publications are arranged hierarchically. There is introductory material, family descriptions, generic descriptions, species descriptions, subspecies descriptions. These descriptions are nested within each other and it isn’t always clear what information in one level of the hierarchy is repeated at lower levels. Descriptions are diagnostic within the frame of reference of that treatment i.e. they provided enough detail to separate that taxon from the others at that level in that particular hierarchy. Differentiating the taxon from other taxa in other treatments of the same or overlapping groups is usually relegated to notes.

Today we talk of monographs having this form because they reflect phylogeny. Previously they reflected a more ill-defined ‘affinity’ or natural ordering. Originally hierarchies were used as an aide-mémoire. This results in a mishmash of  concepts that it is difficult to decode. Phylogenies do not have ranks or a linear order yet monographs do. So what does the order and rank in a phylogenetically based monograph represent? If these things exist as aide-mémoire then why aren’t they totally arbitrary – merely picking out the most easy to remember characteristics of different groups and making no attempt to represent evolutionary history.

Imagine approaching a monograph/flora/fauna with a species name. You look it up in the index. Turn to the right page and then have to assemble a description of the taxon by reading back up the taxonomic hierarchy – unless of course the author has redundantly repeated all the descriptive data in the species level account which begs the question of what the higher level accounts are for.

Now suppose you have in your hand an unknown specimen. First thing you need to do is to know the family and possibly the genus so you can find the right work to look it up in. There is rarely a multi access approach to getting you near to a taxon such as “deciduous trees with palmate leaves”. You have to be a taxonomist, have fertile material and more or less know what the thing is before you even start. By definition the monographs are not optimized for identification of specimens.

This means that these works are mainly of use to taxonomists who are familiar with the groups concerned. But what do they use them for?

If a taxonomist is working on a new revision they won’t be consulting current, extant monographs very much. That work has been done and shouldn’t need revising for decades. They will be working on material that hasn’t been monographed for decades if ever and needs to be classified. If they are finding new species within a recently monographed group then they will be turning over the apple cart because the descriptions in that monograph are now out of date because the monographic form is designed to be comprehensive.

What is more likely is that a taxonomist will use existing monographs to produce secondary taxonomic products such as field guides – and this is where my key point comes in.

Remixing

Suppose you want to produce a secondary taxonomic product. Say a guide to the lowland trees of a country. Even if you had a checklist of all the species of the country how would you know which were lowland trees? That kind of habit character is likely to be buried in descriptions. Even if you had your list of the species how would you build your guide? How would you pull together free standing descriptions of each taxon? The only way at the moment is to roll your sleeves up an become a taxonomist. Start writing new descriptions based on the contents of monographs (in which the descriptions are designed to differentiate your target taxa from taxa that will not be included in your guide). This kind of thing should really be done automatically. We should be able to do a search for all the species that are considered trees and occur below a certain altitude and find free standing descriptions of these species that we can load on our phone or tablet or print in a booklet and take into the field. The stuff that taxonomists currently produce does not support this kind of behaviour.

What about putting the monograph on the web? If someone links to a species what do we show on the page that is displayed? Do we include the genus description as well? What if the species description doesn’t mention the generic characteristics? Do we include the subspecific taxa? What if the subspecific taxon is only defined in terms of its minor differences to the main species – “var. alba”

Two years ago I discussed how difficult it is was to handle hierarchies in Synonyms Are SubClasses And Higher Taxa Are Just Tags which is a little more technical than this piece but makes similar points.

Tough Love

Producing electronic versions of narrative monographic works is OK  from a political point of view and, if you are doing a print copy you may as well do an ebook and pdf but from the point of view of a non-taxonomist it is of little value and we shouldn’t kid ourselves that we are increasing accessibility very much. It may even be counter productive because people think they have produced an electronic resource when all they have produced is a facsimile of the paper one that is probably slower to use.

My Suggestion

Taxonomy needs to move to a One Species Per Publication model – I call this a fact sheet based approach. Instead of producing monographs of groups taxonomists should produce single free standing publications, one per single species with a global scope. If their primary interest is in phylogeny then they produce separate papers that only discuss the relationships between species that are already described in the free standing publications. This approach is far more appropriate for this digital age for the following reasons:

• Referable - Single species can be used and referenced like other scientific or web resources. It is possible to refer to the use of a species in a study or in legislation and reference a single source that just describes that species. A lawyer can right a document that says we want to conserve species X as described in publication Y and that statement is not entailed with all the other taxa and data that is presented in publication Y.
• Remixable - It is possible to pull together a set of species descriptions to form a new resource. This may be done either automatically, say from a list of occurrence records for a region or habitat, or on a pick’n'mix basis (see taggable below).
• Granular Versionability (= Stability) - It is possible to replace individual species definitions in a set of definitions without having to reversion the whole lot. A new phylogeny or new species in a genus need not change other species in the genus that may be subjects of legal protection etc.
• Data transparent - In a typical monograph the data is of varying quality. One species may be based on five hundred specimens and another on only five. This isn’t always clear from casual use of the monograph where specimens examined and data analysis are typically presented separately from the main treatment. If all the data used to define a species is presented in a single publication then things become a great deal clearer.
• Granular Peer Review - Not all monographs are peer reviewed. Those that are are taken all or nothing. Suppose a monograph of twenty species is presented. It may be very good and have a good phylogenetic analysis etc. Perhaps two of the species are not particularly well defined but it is of high enough merit as a whole to be published. The result is that 10% of species are not particularly well defined! It would have been better to pass eighteen species and reject two. Taxonomy is riddled with such species. You only need to read a monograph that is sinking ill-defined species from the previous monograph that probably shouldn’t have been published in the first place – whilst creating new ill-defined species of its own.
• Taggable - Anything that can be reliably referenced can be tagged. This means that it becomes possible to build meaningful lists of species that can be pulled together into useable products. The tagging does not have to be done by the authors. For example IUCN tags species with conservation status and a group working on functional ecology may tag them with their functions in the environment. It is then possible to pull together a list (with descriptions) of endangered species that perform a certain role in the environment. Currently this process only leads to a list of names that can be handed off for someone to work on trying to establish what the different sources meant by those names.
• Faster More Agile Development - We can’t describe all the species on earth in the way we have been doing with the resource available in a reasonable time. This is not an unusual problem. All domains are faced with challenges that can’t be addressed by the resources available. In software engineering the ‘agile’ approach to this problem is to prioritize development of important, doable things to build an initial working system and then revisit and re-prioritize what needs doing next. Publish results quickly and often. In taxonomy the opposite approach is often taken. A group is selected for monograph and worked on until resources are exhausted and the monograph is then published. By adopting a One Species Per Publication approach the ‘easy’ species would be published as soon as the researcher was sure they were ‘good’ taxa making their work available for others to use and give feed back on years sooner than is traditionally the case and whilst resources are still available to respond. Should the project stall or fail to complete then possibly the most valuable results will already be in circulation and not lost to science. Those enormous genera that are a life times work for someone could be chipped away at by the army of short term employees who are replacing career scientists!
• It would make the job of aggregators like EoL much, much easier! If we accept the fact that we need projects like EoL (which I think we all do) then we must also accept that we need to produce data in a form that they can use.

Summary

This is a long post so to summarize my proposal

1. Stop writing monographs or  floristic or faunistic regional accounts of taxonomic groups.
2. Produce individual, self contained fact sheets of single species that are global in scope.
3. Use ‘Agile’ development techniques to produce and update these rapidly.
4. Treat phylogenies as separate products that handle the arrangement of the entities described in fact sheets.

I am sure this will put a lot of peoples backs up. Please leave a comment if you agree and well as if you want to see my lynched.

1. Text is obtained from the source monograph either via OCR or access to the original word processor documents.
2. The text is topped-and-tailed to remove the introduction and any appendices and indexes.
3. Text is converted to UTF-8 if it isn’t already.
4. An XML header and foot are put in place and any non-XML characters are escaped  – this actually came down to just replacing & with &
5. The text is now in a well formed XML document.
6. A series of custom regular expression based replacements are carried out to put XML tags at the start of each of the recognizable ‘fields’ in the species accounts. These have to be find tuned to the document as the styles of the monographs are subtly different. Even the monographs published in the same journal had some differences. It is not possible to identify the start and end of each document element automatically. This is for three reasons:
   1. OCR errors mean the punctuation, some letters and line breaks are inconsistent.
   2. Original documents have typos in them. A classic is a period appearing inside or outside or inside and outside a closing parenthesis.
   3. There are no consistent markers in the source documents structure for some fields. For example the final sentence of the description may  contain a description of the habitat, frequency and altitude but the order and style may vary presumably to make the text more pleasant to read. The only way to resolve this is by human intervention.
7. The text is no longer in a well formed XML document!
8. The text is manually edited whilst consulting the published hard copy to insert missing XML tags and correct really obvious OCR errors. In some places actual editing of the text is needed to get it to fit a uniform document structure as in the habitat example above.
9. The text is now back to being a well formed XML document.
10. An XSL transformation is carried out on the XML to turn it into ‘clean’ species accounts and alter the structure slightly.
11. An XSL transformation is carried out to convert the clean species accounts into SQL insert statements for a simple MySQL database. The structure of this database is very like an RDF triple store (actually a quad store as there is a column for source). A canonical, simplified taxon name (without authority or rank) is used as the equivalent of the URI to identify each ‘object’ in the database. Putting the data in a database makes it much easier to clean up and to extract some additional data. An alternative would be to have a single large XML document and write XPath queries.

By writing queries that join the Rhododendron database to institutional databases I can create lists of living and dead specimens at Royal Botanic Garden Edinburgh and extract images from the herbarium digitisation project. Previously I extracted images from BHL that I can also join in. I can do things like ‘tag’  species with the ISO country codes, whether they are epiphytes, their altitude range – all interesting facts. I can imagine someone asking a real question such as “Give me accounts for all the rhododendrons that occur above X meters in Thailand”.

I could write a bespoke front end to the database that enables this functionality but this wouldn’t help someone answer the question “Give me accounts for all the XYZs that occur above X meters in Thailand”. Let’s face it only a small bunch of taxonomists and enthusiasts are interested in data that only includes rhododendrons. For most people this data will  never be the whole answer. I am being funded to do this work so that we can get the information from the Edinburgh Rhododendron monographs into the Encyclopedia of Life. There it can be mixed in with data from many other sources and so move towards answering the questions “most people” are likely to ask.

‘Properties’ I Have Captured

From the workflow described above you can see that the  properties I have in my database have to represent the document structure of the monographs – plus some tags extracted by very simple data mining.  The properties are:

• altitude (around, max, min, range) – Many accounts include a range of numbers or maybe a single ‘circa’ number.
• description – Diagnostic description. Importantly this may or may not include characters that have been mentioned higher up the taxonomy in a group, subsection, section or subgenus description.
• distribution – Usually the country and province. Sometimes individual mountains or parks.
• habitat – a sentence describing the habitat but often including whether it is epiphytic or terrestrial (shouldn’t this be habit?) and also whether it is common or not.
• icon-ref – a citation of where an image can be found in the literature.
• image – a link to a Curtis image.
• name (author, cite, formatted) – Three properties breaking down the name
• type - The type citation string for the accepted name of the taxon.
• note – All sorts of things in here! Almost all accounts have some comment varying from “Known only from type collection” to several paragraphs of text. May include derivation of name. May occur multiple times  for a species.
• occursInCountryIso – this was extracted by simply looking for country names in the distribution field
• rank - the database contains facts about the subspecies, varieties and even forma that occur in the monographs (see below)
• subgenus – a single word indicating which subgenus the species is in. Subgenera in Rhododendron could be thought of as genera …
• synonyms – this is a block of text representing all the names, types and citations that came in the synonyms paragraph.

To get these properties into EoL I need to squeeze them into the EoL Transfer Schema . (Here I need to have a declaration of interest in that I think I was in on the original design of this at a workshop at GBIF as few years ago.) The basic structure is like this:

• Document
  • Taxon
    • Name
    • Source
    • Other Metadata…
    • DataObject
      • Type
      • Source
      • Other metadata…
      • Value
    • DataObject
      • ..
  • Taxon
    • …

So a document contains a number of taxa and each taxon contains some metadata plus a number of DataObjects. Each DataObject is of a ‘type’ and has its own metadata plus a value of some kind – such as text or a link to an object. This is a very generic data structure that allows for expansion by adding new types of DataObject.

All I need to do is hack together a PHP script to map my properties to the DataObject types and I can go back to trying to clean up the data. This is what the types look like:

Associations, Behaviour, Biology, Conservation, ConservationStatus, Cyclicity, Cytology, Description, DiagnosticDescription, Diseases, Dispersal, Distribution, Ecology, Evolution, GeneralDescription, Genetics, Growth, Habitat, Key, Legislation, LifeCycle, LifeExpectancy, LookAlikes, Management, Migration, MolecularBiology, Morphology, Physiology, PopulationBiology, Procedures, Reproduction, RiskStatement, Size, TaxonBiology, Threats, Trends, TrophicStrategy, Uses

These map to subject types on taxon pages within EoL. There is a description of these on the EoL help pages.

This is where I run into a problem. My properties don’t map to these subject types. The only matches I really have are Distribution, Description and Habitat. The advice is to put “note” type data under “Description” so probably 90% of what I have goes into “Description” DataObjects. Why have I just spent the last umpteen weeks marking all this stuff up?

There are interesting and important questions here:

• How important is semantic mark up of this kind of data? What advantages are gained over just treating each species treatment as a single block of text? I could still pull out all the ones that occur in China etc.
• If the EoL Subject Types are a list of the kinds of information people want to see on species pages and they don’t match the data that is captured in a monograph should we continue to produce monographs in their current form? Who is driving the production of data, the users or tradition?

I have been working with Sencha Touch and the Ext.util.Geolocation object but am having problems with accuracy. I have noted the following behaviour.

When I call for a location on iPhone (3G) and iPad (v1) I get a one with around 1.3km accuracy. Basically it places me at one of two spots about 1km apart. If I switch to the native maps app then it places my position within 10m of where I am standing – that “wow it knows where I am” accuracy . Switch back to my web app and the first call to the GeoLocation returns similar accuracy. Any subsequent calls return the old inaccurate positions.

I have tested this briefly on Android (Google’s first phone) and have similar behaviour.

I am setting allowHighAccuracy=true and I have experimented with different age and time out durations to no effect. I have also tried calling the underlying JavaScript methods with similar results so I don’t believe it is the Touch libraries – but would welcome your thoughts.

My conclusion is that both iOS and Android only pass cell tower level location accuracy to the browser and do not ever use GPS – at least not in an urban environment. Basically enableHighAccuracy from the Geolocation API spec is ignored. Is this a correct assumption? I hope it isn’t because it effectively cripples the HTML5+GPS application market.

I wrote a test application that you can use to check this behaviour for yourself. Remember this is a mobile app. You can test it in Chrome or Safari on the desk top but will need to load it on your phone and be outside to benefit from GPS! Dink the QR Code with your phone and take a stroll if you are reading this indoors.

The application allows you to either poll the location by tapping the “Update Location” button or ask the browser to continuously update your location using autoUpdate. The maximum age parameter is hardwired to zero so any requests should attempt to fetch new location data. The time out is set to 20 seconds. You will find that if you keep punching the “Update Location” button when it isn’t returning new locations you will just get a series of time out alerts 20 seconds later.

Finally the “Show Map” button will launch the maps app on iOS devices and put a place marker on where HTML5 thinks you are. You can then compare it with where the maps app thinks you are. They are often different immediately or, after a few seconds, the maps app will move your current location to a far more accurate spot as the GPS kicks in. On Android the behaviour of the “Show Map” button is less obvious as it doesn’t always appear to open the maps app and sometimes just opens the Google Maps application. You will have to open the maps app manually.

I find it amazing that there are so many posts out there raving about location info in the browser yet when I try and use it I find it actually sucks. Maybe OK for finding your nearest Starbucks but not a lot else. If you have read The Inmates Are Running the Asylum by Alan Cooper then you will be familiar with the notion of “Dancing Bear Software” – it is amazing the bear can dance, what a shame it doesn’t dance very well!

I do hope I am doing something really really dumb here and am totally wrong. Please look at the code in my app page and tell me so.

This post deals with the semantics of extraction of data from the Rhododendron monographs. Another post will deal with the technicalities of the actual extraction.

The image above shows a species description entry. It was chosen as being a small and simple example for illustrative purposes. I have marked up the bits I am interested in extracting. Red indicates important fields, blue unimportant and yellow something in between – but why those bits and those priorities? Monographs contain a great deal of other stuff such as keys and descriptions of higher taxa and discussions. We could argue for hours about what should be extracted and never come to a conclusion unless we have some guiding principles on what we are trying to do. I have therefore developed five guiding principles for the project that are probably pretty general and may be applicable to other such projects:

1. We are extracting data NOT trying to create a digital facsimile of the document. We already have the document in its physical form, as page images, as OCR’d text and as a PDF combining the two. We can read the document whenever we need to and it is highly unlikely we will want to edit it like a word processor file. We don’t therefore need to capture anything that is to do with document layout such as fonts, line breaks, paragraphs and section orders.
2. We are NOT trying to extract ALL the data in one pass. The process is not destructive and we can always return at a later date. We will probably be better at interpreting the document in the future and we may have a different perspective on how to interpret it. We should only pull out what we need today not what we think we might need at some point in the future – but see point 5.
3. Use-case driven. We should have some use-case (or story) about how the data will be used. This is the basis on which we can make all the smaller decisions.
4. Opportunistic. If it really is free then we will have it! Points 2 & 3 should not prevent us from capturing data that comes for free as a by-product of the process. We should be careful we are not kidding ourselves though and drop anything that is taking time but contribute to the use-case.
5. Provenance should only be to a useful point in the document. It is easy to get carried away with capturing provenance metadata about where in the document a piece of information comes from and what has happened to it since then. In the example above we could capture the fact that R. johnstoneanum occurs in India is defined by the first word on the fifth paragraph of the species account rather than having been extracted from a map or the type locality but before we know it we are effectively recreating the document (point 1). Instead we merely tag data with the species account (document section) the information comes from and the page on which the account occurs. If someone wants to track down where a fact came from they can be taken to the relevant section of the document and read it for themselves.

If these are general principles what’s the use-case for this project, what do we actually want to do today?

1. Create species page data for Encyclopedia of Life.
2. Data should be of biological interest – about a species of organism not about the processes of taxonomy and nomenclature. It is taken as read that the nomenclature has been sorted out by the experts and we are just dealing with the products of that process.
3. We are only interested in ‘species’ – or basic units of diversity – and so not in capturing higher levels of taxonomy such as groups, subsections, sections and subgenera.
4. We are interested in tagging the species with facts that can be extracted e.g. occurs in India; Habit Shrub; Habit Tree; etc – these are largely opportunistic.

The result is to extract the following fields of data for each species account

• altitude-around = When the description contains a single altitude value in meters, often with a circa
• altitude-max = The max altitude in meters
• altitude-min = The minimum altitude in meters
• description = The bulk of the descriptive text – possibly for further data extraction later
• distribution = A fairly uniform string describing the country and province distribution
• habitat = The habitat string – this is very variable
• name = The actual name expanded to include Rhododendron rather than R. but without the author string
• name-author = The author string
• name-cite = The protolog (original place of publication of the name)
• name-type = A description of where the type is located
• note = Most species have one or two paragraphs of notes
• volume-part-page = A pointer to the page the species account starts on
• icon-ref = A reference to a image of the specimen
• synonyms = All the synonymy as a single block of text

I’ll talk about the mechanism of doing the extraction in another post.

William Curtis (1746-1799)

This is a sideline to my working on the Edinburgh Rhododendron monographs.

The monographs often quote references to illustrations (icons) of species. This is useful as we know that these are illustrations that have been determined by the author of the account and are therefore “correctly” determined. What a shame we only have an abbreviated text string that can really only be understood by a human. An example might be “Rhododendron & Camellia Yearbook 25: f.58 (1970)”. Because these are in the botanical monographic style it is near impossible even to turn them into an OpenURL that a resolver could make sense of – so we have a bit of a challenge.

For the just-under-four-hundred species accounts I have extracted from the first two monographs I have 445 icon strings. Of these 144 contain ‘Bot. Mag.’ – for Curtis’ Botanical Magazine and so they look like a good set to try and parse and link up. The Biodiversity Heritage Libary have digitized that proportion of Bot. Mag. prior to 1920 that is out of copyright thanks to Missouri Botanic Gardens. I just need to join it all up. In fact I could download the relevant images and embed them in my data because they are out of copyright.

So a happy afternoon was spent learning about the BHL API and writing XSLT and regular expressions to parse the strings I had. The result was a match up of just 59 illustrations. About the same number I could have done manually in an afternoon! The rest of my Bot. Mag. references are post 1920 and so locked up in copyright.

But a happy by-product of the process was the fact that I downloaded and parsed all the metadata for Bot. Mag. in BHL and extracted the item IDs (books) and page IDs for what I believe are all the illustrations – a total of 8,215. So if you are faced with the same issue as me you don’t have to go to the bother of doing it. Here is a CSV file of the full list.

All Curtis Illustrations In BHL (CSV)

I have included the URLs to the resources in BHL although these are just trivial concatenations of the page IDs or item IDs and an http prefix.

Rhododendron neriiflorum

Unfortunately the names of the taxa are not included as I was solving the problem of getting from a citation to an image – I already had the name. It would be tempting to try and calculate the names for each of the illustrations but I can’t justify doing this right now so it is an exercise left to the reader.  The problem is the illustration in Curtis may come before or after the text on the species although on the plus side there is only one species per page. What I would try doing is:

• Calling for the OCR of the BHL pages ids immediately before and after the page id of the illustration
• See if the first few lines contain the page number (plate or tab. number) of the illustration.
• If they do then we have found the page for the plate so use Taxon Finder to extract the names of that page (there may be a BHL API call for this)

I am still left with 85 Bot. Mag. references I can’t link to anything because Bot. Mag. is safely locked away behind copyright at Kew and Blackwell Publishing. Wouldn’t it be nice if they created a web page for every illustration that contained at least a low resolution version.

I hope the attached file is of some use to someone and also that no one points out I could have done this more quickly and easily by some other route. If you use it please post a comment – thanks.

• Cullen, J. (1980) Revision of Rhododendron. I. subgenus Rhododendron sections Rhododendron and Pogonanthum. Notes from the Royal Botanic Garden Edinburgh. 39:1-207.
  
• Chamberlain, D.F. (1982) A revision of Rhododendron. II. Subgenus Hymenanthes. Notes from the Royal Botanic Garden Edinburgh. 39:209-486.

Between them these publications cover 544 species – more or less half the genus.

The entire run of the Notes has now be digitized to page images for BHL-Europe and so I have access to good quality pictures of the text. We have an in-house OCR service that I can drop these images into to create text or other outputs. I started by dropping all 200+ images from the first publication into the OCR and creating 200+ text files but this didn’t make sense because many of the species accounts ran across multiple pages. What I needed was the contiguous text for the whole publication. I could have concatenated the text files but I figured the OCR software would do a better job if it was working through one big document as it would learn from previous pages – OK maybe this is fantasy but it is worth a try. By using Preview (the Mac’s default PDF and image viewer) I created a single PDF containing all the images and put that through the OCR processor. The result was not only a single text file but also a PDF of the whole publication including OCR’d text. Job done! Can I stop now?

This process showed how easy it is to create digital versions of publications. The  PDFs produced are not very friendly being almost 100mb in size for each of the two publications but they can be read on line and indexed so do fulfill the basic requirements of making ‘legacy’ publications available. Because of their size I do not attach the PDFs here.

Two points jump to mind:

1. The accuracy of the OCR is masked because the text is hidden behind the page images. Although the document is searchable we can’t be sure that, if a search term is not found, it is because it isn’t there or because the OCR failed for that word in that location. This digitization process is likely to engender a false sense of security.
2. The PDF’s of the publications do not enable re-mixing or querying of the data beyond simple text searching. Question like “What species occur in Yunnan, China?” can only be answered by working through the text manually – something that might be quicker with the printed version.

Making text available to read on line is useful in that it facilitates distribution and discovery of that text but that is all it does.

The next step is to try and turn what is basically a descriptive narrative into more useful information that can be used to answer the simple questions people are likely to ask of about biodiversity. At the least it has to be massaged into a set of web pages, one for each species, for use in EOL. There are two aspects to this process:

• Syntax – this is really the easy bit although time consuming. The text of the monograph has a particular syntax – ordering of characters into words and sentences. We need to mark up the document with another syntax that will allow a machine to extract chunks of information. This isn’t too difficult to do at a course level because the monographs are highly structured but it becomes harder the more finely granular the syntax becomes. It inevitably involves a lot of manual work and I’ll cover it in another post.
• Semantics – this involves tougher decisions but isn’t that time consuming. We need to decide what chunks of information in the document we want to extract and what chunks we can practically extract and reach some kind of compromise. Different chunks of text can be seen in the document. Some of these chunks have no biological meaning at all e.g. a page or a paragraph. Others have useful biological meaning e.g. a distribution string like “NE Burma, China (Yunnan, Sichuan, W Guizhou)” in the context of a species description. The decisions made about what to extract will effect the syntax used and how long it will take to impose that syntax on the raw text of the document. Making these decisions will be the subject of another blog post.

The bulk of the work done on Rhododendron occurred just before the digital age kicked in and so the material is not integrated in a way that can be re-used and re-purposed. An example of this is what could be called The Edinburgh Rhododendron Monograph which covers 1,027 recognized species. This is actually spread over seven publications that came out over the course of 26 years in two journals and a book and is not available in a single form anywhere. The publications are:

• Cullen (1980) Subgenus: Rhododendron Sections: Rhododendron & Pogonanthum 231 species
• Argent (2006) Subgenus: Rhododendron Section: Vireya 313 species
• Chamberlain (1982) Subgenus: Hymenanthes Section: Ponticum 302 species
• Chamberlain & Rae (1990) Subgenus: Tsutsusi 117 species
• Kron (1993) Subgenus: Pentanthera Section: Pentanthera 23 species
• Judd & Kron (1995) Subgenus: Pentanthera Sections: Rhodora, Viscidula & Sciadorhodion 7 species
• Philipson & Philipson (1986) Subgenera: Azaleastrum, Therorhodion, Mumeazalea & Candidastrum 34 species

Last year I was fortunate to be awarded a Encylopedia of Life – Rubenstein Fellowship to create a species page in the encyclopedia for each of the species covered by the Edinburgh monograph – the text of all seven publications now being available electronically in various forms. The award funds me for a total of 100 days to process the OCR’d or PDF text into the EOL transfer format and to link it in to as much additional data as possible. I hope to blog my experiences good and bad.

References

• Argent, G. (2006). Rhododendrons of subgenus Vireya. Royal Horticultural Society, London.
• Chamberlain, D.F. (1982). A revision of Rhododendron. II. Subgenus Hymenanthes. Notes from the Royal Botanic Garden Edinburgh. 39:209-486.
• Chamberlain, D.F. & Rae, S.J. (1990). A revision of Rhododendron. IV. Subgenus Tustsusi. Edinburgh Journal of Botany. 47(2) 89-200.
• Cullen, J. (1980) Revision of Rhododendron. I. subgenus Rhododendron sections Rhododendron and Pogonanthum. Notes from the Royal Botanic Garden Edinburgh. 39:1-207.
• Judd, W.S. & Kron, W.S. (1995). A revision of Rhododendron sections Sciadorhodion, Rhodora and Viscidula. Edinburgh Journal of Botany. 52:1-54
• Kron, K.A. (1993). A revision of Rhododendron section Pentanthera. Edinburgh Journal of Botany. 50:249-364.
• Philipson, W.R. & Philipson, M.N. (1986). A revision of Rhododendron. III subgenera Azaleastrum, Mumeazalea, Candidastrum and Therorhodion. Notes from the Royal Botanic Garden Edinburgh. 44:1-23.

I am still looking for a flaw in how I have calculated these numbers and would welcome suggestions.

As I worked through the list I began to match them up against data sources in the GBIF Data Portal but this task became tricky as there were data sources in GBIF that had the names of museums but were clearly the results of observational studies and not catalogues of specimens. I decided to break off and do an analysis of what was in the GBIF Data Portal by way of specimens residing in Europe. This post is the results of that analysis.

GBIF Metadata Web Services

GBIF provides web services to access metadata on the data it harvests.  I extracted details of data sets hosted in Europe from these services. GBIF use three terms:

• Data Resource (a.k.a. Data Set) – the actual data set containing occurrence/specimen records. Metadata available through this web service.
• Data Provider – the organisation supplying the data. Each data resource belongs to a single data provider. Metadata available through this web service.
• Data Network - a collection of data resources often arranged on a geographic or political basis. A data resource can belong to multiple networks.

I was interested in the actual data resources but they don’t have a notion of where they are located so I started by querying the data provider service to get a list of data providers (and their resources) for each of the countries in Europe. I then called the resource service to get the metadata for each resource thus building a list of all the resources in Europe. Here is a list of the all countries and the number of resources they have.

First big surprise is that Germany has 10x the number of resources of all the other countries put together! It turns out that this is largely due to a single data provider (Pangaea.de) who publish a very large number of small data sets, many form individuals or small projects. These should shake out later in the analysis. The distribution of sources by countries isn’t too surprising. Larger, richer countries have more. But why none in Italy and only one in Sweden?

Teasing Out the Specimens

The data harvested was for all data sets but many of these are observational data and not museum and herbarium catalogues. Fortunately their is a field in the metadata that gives the default basis of record. It may be that some are mixed but in practice this is unlikely. Unfortunately this field isn’t always filled in.

It looks as though we only have 101 data sets for collections but some of those 2,821 must be collections. Can we guess which ones on some other characteristics of the data set.

• Firstly we are only interested in larger data sets – the bigger collections first strategy – so we could forget everything with less than 1,000 occurrence records. This should also clear out many of the personal data sets in Pangaea.de.
• Then we might presume that observation data sets will have more observations per taxon than specimen data sets. Instead of collecting representatives of species they are collecting distribution or ecological data.
• Then we might presume that specimen data is old and probably won’t yet be geocoded where as for observation data location is everything and they will frequently be geocoded.

Here is a query ignoring collections with < 1,000 occurrence records and showing the occurrences/taxon and percentage geocodings.

It looks to me that we could make the presumption that if an unknown data set has an average of fewer than 50 occurrences per taxon and is less than 50% geocoded then it is probably a specimen dataset. In SQL we consider a data set to be a specimen collection  WHERE basisOfRecord = ‘specimen’ OR (basisOfRecord = ‘unknown’ AND occurrenceCount > 1000 and occurrencesPerTaxon < 50 and percentGeocoded < 50).

Doing this gives us a list of  276 data sets and a total number of specimens of 7,665,815.

In my previous blog post I did not attempt to estimated the number of specimens in Europe but I did total the number mentioned in the Biodiversity Collections Index as 334 million. Working on the basis that many collections are missing from that list but that the missing collections are unlikely to be the BIG collections it seems reasonable to assume the total number of specimens in Europe will not top 500 million. If you doubt this then think where we might find another 160 collections containing a million specimens each. Arturo Ariño wrote a paper in 2010 (Biodiversity Informatics, 7 2010, pp. 81-92) where he estimated total units (specimens/lots/countable things) to be 1.2 to 2.1 billion and that GBIF had mobilized only 3% of these. My personal feeling is that these are over estimates but not by orders of magnitude.

If there are 334 million specimens in Europe then 7.7 million records in GBIF is 2.3% on the other hand if we underestimate collections and there are 0.5 billion specimens in Europe then GBIF only has 1.5% of them. Which ever you choose it is not a lot.

If we were to take just six big collections – for example:

• Muséum National d’Histoire Naturelle – Entomology
• Royal Belgian Institute of Natural Sciences
• Natural History Museum London, Department of Entomology
• Royal Museum for Central Africa
• Naturhistorisches Museum Wien
• Natural History Museum of Denmark

And then we were to digitize just 5% of their holdings. We would more than double the number of European specimens in GBIF.

What am I missing?

One thing worth sharing is a report I did for Synthesys on improving the quality of metadata on Eurorpean biodiversity collections. It includes analysis of the data in the Biodiversity Collections Index and other sources and comes to some conclusions about how we could increase our knowledge of specimens held within Europe.

In summary – the majority of specimens are in a few large collections. If we improved the coverage of a few dozen major collections then we could cover the majority of specimens held. This is important because the ecomomies of scale kick in with larger collections. One techie guy can support the digitisation of a collection containing many millions of specimens almost as easily as a collection with only a couple of hundred thousand. This is not saying smaller collections aren’t important it is merely a numbers game.

You can read a PDF of the full report but please remember this isn’t a scientific paper it is a quick look at the data to think about what to do next. NA3 Task 2.3 – Metadata on European Collections – Report and Forward Plan – PDF