I took part in the second Data Science London Hackathon last weekend (also my second hackathon) and it was a very different experience compared to the first hackathon. Once again Carlos and his team really looked after us. The data was released 24 hours before the competition started and even though ... [Read more...]

Preface. This is the first of what I hope will be many posts analysing experimental data, that will eventually end up in my empirical software engineering with R book (this experiment was chosen because it happens to be the one I am currently working on; having just switched to using ... [Read more...]

I notice from time to time that people download code from the drop box. The drop box lags  CRAN.  I’ve upload the newest versions to the drop box. In the future I will post new releases to the drop box before I submit to CRAN. [Read more...]

This post provides background information that can be referenced by future posts. For the last 18 months I have been working in fits and starts on a book that has the working title “Empirical Software Engineering with R”. The idea is to provide broad coverage of software engineering issues from an ... [Read more...]

by Yanchang Zhao, RDataMining.com There is a nice example on sentiment analysis with R at . In the example, the Wikileaks cable corpus is analyzed to track US sentiments of other countries and their presidents over time. The example describes … Continue reading → [Read more...]

Introduction I've recently been interested in how to communicate information using color. I don't know much about the field of Color Theory, but it's an interesting topic to me. The selection of color palettes, in particular, has been a topic I've been faced with lately. I downloaded 18 different sequential color ... [Read more...]

This past week for work I had to create some plots to show the max, min, and median of a measure across the levels of a qualitative variable, and show the max and min of the same variable within a … Continue reading → [Read more...]

I recently finished a long stretch of work on a particular project that required me to draw upon four R packages.  Each time I got back to my work on the project, I’d have to load the packages manually, as … Continue reading → [Read more...]

by Yanchang Zhao, RDataMining.com There are some nice slides and R code examples on Data Mining and Exploration at http://www.inf.ed.ac.uk/teaching/courses/dme/, which are listed below. PDF Slides: - Overview of Data Mining http://www.inf.ed.ac.uk/teaching/courses/dme/2012/slides/... [Read more...]

Browsing commandlinefu.com today, I came across this little one-liner to display which commands I use most often. Here’s what I got: Yep, seems legit. I navigate and look at files a whole bunch (ls, cd, cat), and I do a butt tonne of editing (vim). I sudo like ... [Read more...]

Usually, I like to write about the solutions to problems I’ve had, but today I only have a problem to write about. This is the second research job I’ve had outside of academia, and in both cases I’ve met with … Continue reading → [Read more...]

The Tenth Australasian Data Mining Conference (AusDM 2012) Sydney, Australia 5-7 December 2012 http://ausdm12.togaware.com/ Data mining, the art and science of intelligent analysis of (usually large) data sets for meaningful (and previously unknown) insights, is now being actively applied in … Continue reading → [Read more...]

Ever since I started using ggplot2 more often at work in order to do graphs, I’ve realized something about the use of colour in bar graphs vs. dot plots: When I’m looking at a graph displayed on the brilliant Viewsonic … Continue reading → [Read more...]

by Yanchang Zhao, RDataMining.com This post presents an example of social network analysis with R using package igraph. The data to analyze is Twitter text data of @RDataMining used in the example of Text Mining, and it can be downloaded … Continue reading → [Read more...]

The main reason why I have usually chosen to use excel to make my plots at work is because I had difficulty feeding the summary stats in R into a plotting function.  One thing I learned this week is how … Continue reading → [Read more...]

I can’t believe I didn’t learn how to do it earlier, but I never knew how to accurately copy tables from excel that had text with spaces in them, and paste into a data frame in R without generating confusion … Continue reading → [Read more...]

by Yanchang Zhao, RDataMining.com My book in draft titled “R and Data Mining: Examples and Case Studies” is now available on CRAN at http://cran.r-project.org/other-docs.html. It is scheduled to be published by Elsevier in late 2012. Its latest version can be … Continue reading → [Read more...]

The more I’ve worked with my really large data set, the more cumbersome the work has become to my work computer.  Keep in mind I’ve got a quad core with 8 gigs of RAM.  With growing irritation at how slow … Continue reading → [Read more...]

The recent Hack/Reduce hackathon in Montreal was a tonne of fun. Our team tackled a data set of consisting of Bixi (Montreal’s bicycle share system) station states at one minute temporal resolution. We used Hadoop and mapreduce to pull out some features of user behaviours. One of the ... [Read more...]

by Yanchang Zhao, RDataMining.com With a Mac, parallel computing can be achieved with package multicore. Unfortunately, it does not work under Windows. A simple way for parallel computing under Windows (and also Mac) is using package snowfall, which can work … Continue reading → [Read more...]