[This article was first published on novyden, and kindly contributed to R-bloggers]. (You can report issue about the content on this page here)
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.
Calculating correlations is often starting point before more advanced analytical steps take place. Big data (long data) always presents computational challenges of both scale and distributed nature. In turn they may get aggravated by the presence of large number of features (wide data). But challenges do not stop here as complex relationships induce analysis of correlations across subsets and groups.Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.
Such mix of long and wide becomes more common in the age of internet-of-things, sensor and machine data with non-human data sources dominating analytical use cases.
Thus, when computing correlations on big data the following capabilities matter:
- scale on large distributed data sets (long data)
- scale on wide distributed data sets (wide data / large number of features)
- flexibility on wide data sets (ability to permutate features such as Cartesian combinations, one-to-many, etc.)
- correlations on subsets and groups.
Correlations in R comes standard with stats function cor but it doesn’t meet most of the capabilities above. As always Teradata Aster big data analytical platform offers both scalability and functionality far exceeding capabilities above. And thanks to Aster R (TeradataAsterR) package it is available without leaving R environment.
With Aster and R integration there are multiple ways of correlating on datasets. Before sending you to the link for detailed discussion I summarized approaches discussed there by the capabilities:
| Method / Solution features | Variable (columns) Permutations | Calculating for Groups | SQL-MR | In-database R |
|---|---|---|---|---|
| Aster R ta.cor | N | N | Y | N |
| Aster R in-database ta.tapply | N | Y | N | Y |
| toaster computeCorrelations | Y | Y | Y | N |
To leave a comment for the author, please follow the link and comment on their blog: novyden.
R-bloggers.com offers daily e-mail updates about R news and tutorials about learning R and many other topics. Click here if you're looking to post or find an R/data-science job.
Want to share your content on R-bloggers? click here if you have a blog, or here if you don't.