[Read more...]

The idea of GLMs is that given some covariates,  has a distribution in the exponential family (Gaussian, Poisson, Gamma, etc). But that does not mean that  has a similar distribution… so there is no reason to test for a Gamma model for  before running a Gamma regression, for instance. But ... [Read more...]

Consider here the case where, in some parametric inference problem, parameter  is a point in the Simplex, For instance, consider some regression, on compositional data, __ library(compositions) __ data(DiagnosticProb) __ Y=DiagnosticProb[,"type"]-1 __ X=DiagnosticProb[,c("A","B","C")] __ model = glm(Y~ilr(X),family=binomial) __ b = ilrInv(coef(model)[... [Read more...]

[Read more...]

I think that over the past months, I have been saying non-correct things about classification with categorical covariates. Because I never took time to look at it carefuly. Consider some simulated dataset, with a logistic regression, __ n=1e3 __ set.seed(1) __ X1=runif(n) __ q=quantile(X1,(0:26)/26) __ q[1]=0 __ X2=cut(X1,... [Read more...]

[Read more...]

[Read more...]

Consider some simulated data __ set.seed(1) __ x=exp(rnorm(100)) Assume that those data are observed i.id. random variables with distribution, with . The natural idea is to consider the maximum likelihood estimator For instance, consider some maximum likelihood estimator, __ library(MASS) __ (F=fitdistr(x,"gamma")) shape rate 1.4214497 0.8619969 (0.1822570) (0.1320717) __ F$estimate[1]+c(... [Read more...]

Variable importance graphs are great tool to see, in a model, which variables are interesting. Since we usually use it with random forests, it looks like it is works well with (very) large datasets. The problem with large datasets is that a lot of features are ‘correlated’, and in that ... [Read more...]

This morning, in our mathematical statistical class, we’ve seen the use of the chi-square test. The first one was related to some goodness of fit of a multinomial distribution. Assume that . In order to test  against , use the statistic Under , . For instance, we have the number of weddings, in ... [Read more...]

This morning, in our mathematical statistics course, we’ve been discussing the ‘proportion test‘, i.e. given a sample of Bernoulli trials, with , we want to test against  A natural test (which can be related to the maximum likelihood ratio test) is  based on the statistic The test function is ... [Read more...]

In the mathematical statistics course today, we started talking about tests, and decision rules. To illustrate all the concepts introduced today, we considered the case where we have a sample  with . And we want to test   against  In the course, we’ve seen that we could use a test based ... [Read more...]

This afternoon, Jean-Luc asked me some help about an old post I did publish, minuit, l’heure du crime; and some graphs published a few days after, where I used a different visualisation, in another post. The idea is that the hour can be seen as circular, in the sense ... [Read more...]

[Read more...]

[Read more...]

Last week, in our mathematical statistics course, we’ve seen the law of large numbers (that was proven in the probability course), claiming that given a collection  of i.i.d. random variables, with To visualize that convergence, we can use __ m=100 __ mean_samples=function(n=10){ + X=matrix(rnorm(n*... [Read more...]

[Read more...]

[Read more...]

Almost 6 years ago, I posted a brief comment on a sentence I found surprising, by that time, discovered in a report claiming that the expected shortfall […] at the 99 % level corresponds quite closely to the […] value-at-risk at a 99.6% level which was inspired by a remark in Swiss Experience report, expected shortfall […] ... [Read more...]

In November, with Romuald Elie and Jérémie Jakubowicz, we will organize a session during the 100% Actuaires day, in Paris, based on a “pricing game“. We provide two datasets, (motor insurance, third party claims), with 2  years of experience, and 100,000 policies. Each ‘team’ has to submit premium proposal for 36,000 potential ... [Read more...]