[This article was first published on All Things R, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.
Solr is the most popular, fast and reliable open source enterprise search platform from the Apache Luene project. Among many other features, we love its powerful full-text search, hit highlighting, faceted search, and near real-time indexing. Solr powers the search and navigation features of many of the world’s largest internet sites. Solr, written in Java, uses the Lucene Java search library for full-text indexing and search, and has REST-like HTTP/XML and JSON APIs that make it easy to use from virtually any programming language including R.
We invested significant amount of time integrating our R-based data-management platform with Solr using HTTP/JSON based REST interface. This integration allowed us to index millions of data-sets in solr in real-time as these data-sets get processed by R. It took us few days to stabilize and optimize this approach and we are very proud to share this approach and source code with you. The full source code can be found and downloaded from datadolph.in’s git repository.
The script has R functions for:
- querying Solr and returning matching docs
- posting a document to solr (taking a list and converting it to JSON before posting it)
- deleting all indexes, deleting indexes for a certain document type and for a certain category within document type
# query a field for the text and return docs
querySolr <- function(queryText, queryfield=”all”) {
response <- fromJSON(getURL(paste(getQueryURL(), queryfield, “:”, queryText, sep=””)))
if(!response$responseHeader$status) #if 0
return(response$response$docs)
}
# delete all indexes from solr server
deleteAllIndexes <-function() {
response <- postForm(getUpdateURL(),
.opts = list(postfields = ‘{“delete”: {“query”:”*:*”}}’,
httpheader = c(‘Content-Type’ = ‘application/json’,
Accept = ‘application/json’)
ssl.verifypeer=FALSE
)
) #end of PostForm
return(fromJSON(response)$responseHeader[1])
}
# delete all indexes for a document type from solr server
# in this example : type = sports
deleteSportsIndexes <-function() {
response <- postForm(getUpdateURL(),
.opts = list(postfields = ‘{“delete”: {“query”:”type:sports”}}’,
httpheader = c(‘Content-Type’ = ‘application/json’,
Accept = ‘application/json’),
ssl.verifypeer=FALSE
)
) #end of PostForm
return(fromJSON(response)$responseHeader[1])
}
# delete indexes for all baskeball category in sports type from solr server
# in this example : type = sports and category: basketball
deleteSportsIndexesForCat <-function(category) {
response <- postForm(getUpdateURL(),
.opts = list(postfields =
paste(‘{“delete”: {“query”:”type:sports AND category:’, category, ‘”}}’, sep=””),
httpheader = c(‘Content-Type’ = ‘application/json’,
Accept = ‘application/json’),
ssl.verifypeer=FALSE
)
) #end of PostForm
return(fromJSON(response)$responseHeader[1])
}
#deletePadIndexesForCat(“baskeball”)
#Post a new document to Solr
postDoc <- function(doc) {
solr_update_url <- getUpdateURL()
jsonst <- toJSON(list(doc))
response <- postForm(solr_update_url,
.opts = list(postfields = jsonst,
httpheader = c(‘Content-Type’ = ‘application/json’,
Accept = ‘application/json’),
ssl.verifypeer=FALSE
)) #end of PostForm
return(fromJSON(response)$responseHeader[1])
########## Commit – only if it doesn’t work the other way ###############
#return(fromJSON(getURL(getCommitURL())))
}
Happy Coding!
To leave a comment for the author, please follow the link and comment on their blog: All Things R.
R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.