Scholarly metadata – the meta-information surrounding articles – can be super useful. Although metadata does not contain the full content of articles, it contains a lot of useful information, including title, authors, abstract, URL to the article, etc.

One of the largest sources of metadata is provided via the Open Archives Initiative Protocol for Metadata Harvesting or OAI-PMH. Many publishers, provide their metadata through their own endpoint, and implement the standard OAI-PMH methods: GetRecord, Identify, ListIdentifiers, ListMetadataFormats, ListRecords, and ListSets. Many providers use OAI-PMH, including DataCite, Dryad, and PubMed.

Some data-/article-providers provide their metadata via their own APIs. For example, Nature Publishing Group provides their own metadata API here in non OAI-PMH format; you can get PLoS metadata through their search API, and the BHL (see below) provides their own custom metadata service.

In addition, CrossRef provides a number of metadata search services: metadata search and openurl.

What about the other publishers? (please tell me if I’m wrong about these three)

• Springer has a metadata API, but it is terrible, soooo…
• Elsevier, are you kidding? Well, they do have some sort of API service, but its a pain in the ass.
• Wiley, no better than Elsevier.

Note that metadata can live in other places:

• Another package being developed by David Springate, rpubmed can get PubMed metadata.
• Our wrapper to the Mendeley API, RMendeley, gets article metadata via Mendeley’s database.
• Our wrapper to the Biodiversity Heritage Library API here gets their metadata.

No, you can’t get metadata via Google Scholar – the don’t allow scraping, and don’t have expose their data via an API.

I have discussed this package in a previous blog post, but have since worked on the code a bit, and thought it deserved a new post.

You can see a tutorial for this package here, and contribute to the code here.

Install rmetadata

# install_github('rmetadata', 'ropensci') # uncomment to install
library(rmetadata)

Count OAI-PMH identifiers for a data provider.

# For DataCite.
count_identifiers("datacite")

  provider   count
1 datacite 1216193

Lookup article info via CrossRef with DOI and get a citation.

As Bibtex

print(crossref_citation("10.3998/3336451.0009.101"), style = "Bibtex")

@Article{,
  title = {In Google We Trust?},
  author = {Geoffrey Bilder},
  journal = {The Journal of Electronic Publishing},
  year = {2006},
  month = {01},
  volume = {9},
  doi = {10.3998/3336451.0009.101},
}

As regular text

print(crossref_citation("10.3998/3336451.0009.101"), style = "text")

Bilder G (2006). "In Google We Trust?" _The Journal of Electronic
Publishing_, *9*. <URL:
http://dx.doi.org/10.3998/3336451.0009.101>.

Search the CrossRef Metatdata for DOIs using free form references.

Search with title, author, year, and journal

crossref_search_free(query = "Piwowar Sharing Detailed Research Data Is Associated with Increased Citation Rate PLOS one 2007")

                                                                                             text
1 Piwowar Sharing Detailed Research Data Is Associated with Increased Citation Rate PLOS one 2007
  match                   doi score
1  TRUE 10.1038/npre.2007.361 4.905

Get a DOI and get the citation using \code{crossref_search}

# Get a DOI for a paper
doi <- crossref_search_free(query = "Piwowar sharing data PLOS one")$doi

# Get the metadata
crossref_search(doi = doi)[, 1:3]

                           doi score normalizedScore
1 10.1371/journal.pone.0000308 18.19             100

Get a random set of DOI's through CrossRef.

# Default search gets 20 random DOIs
crossref_r()

 [1] "10.4028/www.scientific.net/MSF.126-128.467"
 [2] "10.2139/ssrn.548523"                       
 [3] "10.1016/S0012-821X(02)00562-9"             
 [4] "10.1093/rsq/13.2-3.167"                    
 [5] "10.5772/55055"                             
 [6] "10.1515/BC.1999.050"                       
 [7] "10.1016/S0020-7292(98)90160-6"             
 [8] "10.1111/j.1439-0418.1985.tb02788.x"        
 [9] "10.1089/aid.2012.0115"                     
[10] "10.1016/0002-9378(95)90155-8"              
[11] "10.1001/jama.1949.02900490055028"          
[12] "10.1051/jphyscol:1989172"                  
[13] "10.1016/s0301-2115(03)00298-7"             
[14] "10.1007/BF02735292"                        
[15] "10.1016/0003-4916(65)90026-6"              
[16] "10.4156/jdcta.vol5.issue5.12"              
[17] "10.1007/s10904-009-9316-2"                 
[18] "10.1023/A:1021690001832"                   
[19] "10.1007/s12262-012-0724-0"                 
[20] "10.1007/bf02192860"

# Specify you want journal articles only
crossref_r(type = "journal_article")

 [1] "10.1016/j.jacc.2011.09.055"                                 
 [2] "10.1002/dev.420170603"                                      
 [3] "10.4315/0362-028X.JFP-10-403"                               
 [4] "10.1016/S0925-4927(98)00016-X"                              
 [5] "10.1111/j.1933-1592.2002.tb00141.x"                         
 [6] "10.1541/ieejfms.127.629"                                    
 [7] "10.5539/enrr.v3n1p62"                                       
 [8] "10.1016/S0960-9776(96)90038-7"                              
 [9] "10.1016/0925-9635(94)05240-9"                               
[10] "10.1016/s0929-693x(97)86846-7"                              
[11] "10.1002/(SICI)1096-9071(199601)48:1<53::AID-JMV9>3.0.CO;2-K"
[12] "10.1016/s0267-7261(01)00016-1"                              
[13] "10.1111/j.1748-0361.2003.tb00575.x"                         
[14] "10.1097/00005721-197701000-00011"                           
[15] "10.1007/s00894-009-0593-z"                                  
[16] "10.1071/AR9830063"                                          
[17] "10.1186/gb-2009-10-4-r39"                                   
[18] "10.2165/00128415-201113540-00038"                           
[19] "10.1007/BF00522986"                                         
[20] "10.1080/19407963.2011.539385"

Search the CrossRef Metatdata API.

# Search for two different query terms
crossref_search(query = c("renear", "palmer"), rows = 4)[, 1:3]

                            doi score normalizedScore
1       10.1126/science.1157784 3.253             100
2  10.1002/meet.2009.1450460141 2.169              66
3 10.4242/BalisageVol3.Renear01 2.102              64
4 10.4242/BalisageVol5.Renear01 2.102              64

# Get results for a certain year
crossref_search(query = c("renear", "palmer"), year = 2010)[, 1:3]

                                  doi  score normalizedScore
1            10.1002/meet.14504701218 1.0509             100
2            10.1002/meet.14504701240 1.0509             100
3           10.5270/OceanObs09.cwp.68 1.0442              99
4               10.1353/mpq.2010.0003 0.6890              65
5                  10.1353/mpq.0.0041 0.6890              65
6                  10.1353/mpq.0.0044 0.6890              65
7                  10.1353/mpq.0.0057 0.6890              65
8                    10.1386/fm.1.1.2 0.6890              65
9                    10.1386/fm.1.2.2 0.6890              65
10                   10.1386/fm.1.3.2 0.6890              65
11       10.1097/ALN.0b013e3181f09404 0.6090              57
12      10.1016/j.urology.2010.02.033 0.6090              57
13              10.1353/ect.2010.0025 0.6090              57
14               10.1117/2.4201001.04 0.6090              57
15 10.1111/j.1835-9310.1977.tb01159.x 0.6090              57
16    10.4067/S0717-69962010000100001 0.6090              57
17    10.4067/S0717-69962010000200001 0.6090              57
18           10.2105/AJPH.2009.191098 0.6029              57
19              10.1353/mpq.2010.0004 0.5167              49
20                 10.1353/mpq.0.0048 0.5167              49

Get a short DOI from shortdoi.org.

# Geta a short DOI, just the short DOI returned
short_doi(doi = "10.1371/journal.pone.0042793")

[1] "10/f2bfz9"

# Geta a short DOI, all data returned
short_doi(doi = "10.1371/journal.pone.0042793", justshort = FALSE)

$DOI
[1] "10.1371/journal.pone.0042793"

$ShortDOI
[1] "10/f2bfz9"

$IsNew
[1] FALSE

Get a record from a OAI-PMH data provider.

# Single provider, one identifier
md_getrecord(provider = "pensoft", identifier = "10.3897/zookeys.1.10")

                                                                                                title
1 A new candidate for a Gondwanaland distribution in the Zodariidae (Araneae): Australutica in Africa
      creator date             type
1 Jocqué,Rudy 2008 Research Article

# Single provider, multiple identifiers
md_getrecord(provider = "pensoft", identifier = c("10.3897/zookeys.1.10", "10.3897/zookeys.4.57"))

                                                                                                   title
1    A new candidate for a Gondwanaland distribution in the Zodariidae (Araneae): Australutica in Africa
2 Studies of Tiger Beetles. CLXXVIII. A new Lophyra (Lophyra) from Somaliland (Coleoptera, Cicindelidae)
        creator date             type
1   Jocqué,Rudy 2008 Research Article
2 Cassola,Fabio 2008 Research Article

List available metadata formats from various providers.

# List metadata formats for a provider
md_listmetadataformats(provider = "dryad")

  metadataPrefix
1         oai_dc
2            rdf
3            ore
4           mets
                                                       schema
1              http://www.openarchives.org/OAI/2.0/oai_dc.xsd
2                 http://www.openarchives.org/OAI/2.0/rdf.xsd
3 http://tweety.lanl.gov/public/schemas/2008-06/atom-tron.sch
4                  http://www.loc.gov/standards/mets/mets.xsd
                            metadataNamespace
1 http://www.openarchives.org/OAI/2.0/oai_dc/
2    http://www.openarchives.org/OAI/2.0/rdf/
3                 http://www.w3.org/2005/Atom
4                    http://www.loc.gov/METS/

# List metadata formats for a specific identifier for a provider
md_listmetadataformats(provider = "pensoft", identifier = "10.3897/zookeys.1.10")

            identifier metadataPrefix
1 10.3897/zookeys.1.10         oai_dc
2 10.3897/zookeys.1.10           mods
                                             schema
1    http://www.openarchives.org/OAI/2.0/oai_dc.xsd
2 http://www.loc.gov/standards/mods/v3/mods-3-1.xsd
                            metadataNamespace
1 http://www.openarchives.org/OAI/2.0/oai_dc/
2                  http://www.loc.gov/mods/v3

Some plotting - mean number of authors per paper

Okay, so this isn't a super useful visualization, but you can surely think of something better.

library(ggplot2)
library(ggthemes)
library(reshape)


temp <- md_listrecords(provider = "pensoft", from = "2011-10-01", until = "2012-01-01")
temp2 <- ldply(temp)[, -1]
auths <- sapply(temp2$creator, function(x) length(strsplit(as.character(x), 
    ";")[[1]]))
toplot <- data.frame(authors = auths, articletype = temp2$type)
toplot_ <- ddply(toplot, .(articletype), summarise, authors = mean(authors))
toplot_$articletype <- reorder(toplot_$articletype, toplot_$authors)

ggplot(toplot_, aes(articletype, authors)) + theme_tufte(base_size = 16) + geom_bar(stat = "identity") + 
    coord_flip()

Get the .Rmd file used to create this post at my github account - or .md file.

Written in Markdown, with help from knitr, and knitcitations.