Recently and only recently, I have been exposed to large data structures, objects like data frames that are as big as 100MB in size (if you don't know, you can find out the size of an object with object.size(one_object) command). When you come from another background to [Read more...]

In my last blog post, I talked about Random Sampling for Plain Text in R. Now when you are performing some supervised learning on your data, if you follow the cross-validation principle which divides the data into train, holdout and test, you want to make sure each subset is different. [Read more...]

If you are mining with a text source, performing some language modelling, you need to sample from a text corpus. I was wondering how to have a function to randomly selects chunks of a text file efficiently and is doing the job fast. I wanted to keep the function as [Read more...]