Overview

In this post we first review the concept of semi-parametric copula and the accompanying estimation procedure of pseudo-likelihood estimation (PLE). We then generalize the estimation problem to the setting where the copula signal is hidden in a semi- or non-parametric regression model. Under this setting we have to base the PLE on the residuals. The particular challenge of the diverging score function is handled via the technique of the weighted residual empirical processes.

The semi-parametric copula model

Copula has been a popular method to model multivariate dependence structure since its introduction in Sklar (1959). Consider a random vector \(\mathbf{E}=(E_1,\dots,E_p)^\top\in\mathbb{R}^p\) with joint distribution function \(H\); we assume throughout that \(E_k\), \(k\in\{1,\dots,p\}\) has absolutely continuous marginal distribution function \(F_k\). Then the copula \(C\) associated with \(\mathbf{E}\) is the joint distribution function of the marginally transformed random vector \((F_1(E_1),\dots,F_p(E_p))^\top\). It is clear from this definition that \(C\) itself is always a distribution supported on the unit hypercube \([0,1]^p\), and \(C\) always has uniform marginals supported on \([0,1]\) whatever the marginals of \(\mathbf{E}\) may be. (The explicit form of \(C\) follows from the Sklar’s theorem, for instance Corollary 2.10.10 in Nelsen (2006): \(C(\mathbf{u}) = H(F_1^{\leftarrow}(u_1),\dots,F_p^{\leftarrow}(u_p))\) for \(F_k^{\leftarrow}\) the inverse of \(F_k\) and \(\mathbf{u} = (u_1,\dots,u_p)^\top\in[0,1]^p\).) Furthermore, by the invariance property, if \(g_1,\dots,g_p\) are univariate strictly increasing functions, then \(\mathbf{E}\) and its marginally transformed version \((g_1(E_1),\dots,g_p(E_p))^\top\) will admit the copula.

Thus, copula is a margin-free measure of multivariate dependence. Applied in the opposite direction, one could also start from a copula and couple the copula with arbitrary marginals to create multivariate distributions in a flexible manner. For instance, beyond the usual applications in finance and economy, copulas could be used in the latter manner to model the dependence among the repeated observations in longitudinal data (Sun, Frees, and Rosenberg (2008)).

In this post we will focus on the semi-parametric copula model that serves as a middle ground between a totally non-parametric approach to copula modelling (via the so called empirical copula, see for instance Fermanian, Radulović, and Wegkamp (2004) and Berghaus, Bücher, and Volgushev (2017)) and a totally parametric modelling of the random vector \(\mathbf{E}\). In the semi-parametric copula model, we consider a collection of possible distributions of \(\mathbf{E}\) where the copulas \(C=C(\mathbf{\cdot};\mathbf{\theta})\) are constrained to be parametrized by an Euclidean copula parameter \(\mathbf{\theta}=(\theta_1,\dots,\theta_d)^\top\), but where the marginals \(F_1,\dots,F_p\) of \(\mathbf{E}\) could range over all \(p\)-tuples of absolutely continuous univariate distribution functions.

The pseudo-likelihood method

In the semi-parametric copula model, the primary interest is often the true value \(\mathbf{\theta}^*\) of the copula parameter that determines the multivariate dependence. An obvious challenge in estimating \(\mathbf{\theta}^*\) in the copula setting is how to handle the unknown marginals \(F_1,\dots,F_p\). The canonical solution is the pseudo-likelihood estimation (PLE) introduced in Oakes (1994) and Genest, Ghoudi‬, and Rivest (1995) that we now describe.

Let \(g_1(\mathbf{\cdot};\mathbf{\theta}),\dots,g_d(\mathbf{\cdot};\mathbf{\theta})\) be a collection of appropriate score functions such that the population estimating equation \(\mathbb{E} g_m(F_1(E_1),\dots,F_p(E_p);\mathbf{\theta})=0\) holds only when \(\mathbf{\theta}=\mathbf{\theta}^*\), for all \(m\in\{1,\dots,d\}\). In principle one can always choose the score functions to be the ones in the maximum likelihood estimation, namely \(g_m(\cdot;\mathbf{\theta})=\frac{\partial}{\partial \theta_m} \log c(\mathbf{\cdot};\mathbf{\theta})\) where \(c(\mathbf{\cdot};\mathbf{\theta})\) is the density of the copula \(C(\mathbf{\cdot};\mathbf{\theta})\). Thus, if \(F_1,\dots,F_p\) were known, to estimate \(\mathbf{\theta}^*\) empirically based on a sample \(\mathbf{E}_i=(E_{i,1},\dots,E_{i,p})^\top\), \(i\in\{1,\dots,n\}\) of \(\mathbf{E}\), one could simply “find the zero” of the empirical version of the estimating equation, that is to estimate \(\mathbf{\theta}^*\) by \(\mathbf{\hat{\theta}}^{\text{parametric}}\) that solves, for all \(m\in\{1,\dots,d\}\), \[\begin{align*} \frac{1}{n} \sum_{i=1}^n g_m(F_1(E_{i,1}),\dots,F_p(E_{i,p});\hat{\theta}^{\text{parametric}}) =0 . \end{align*}\] (The superscript “parametric” points to the fact that when \(F_1,\dots,F_p\) are known, we are basically solving a parametric problem.) However, in semi-parametric copula modelling we commonly avoid setting \(F_1,\dots,F_p\) to some particular form. The PLE method solves this problem by replacing the unknown \(F_k\), \(k\in\{1,\dots,p\}\) by its empirical counterpart, namely the empirical distribution function \(F_{n,k}(t) = \frac{1}{n+1} \sum_{i=1}^n \mathbf{1}\{E_{i,k} \le t \}\). The oracle PLE estimator \(\mathbf{\hat{\theta}}^{\text{oracle}}\) of \(\mathbf{\theta}^*\) is then the one that solves the following, revised estimating equation: for all \(m\in\{1,\dots,d\}\), \[\begin{align} $\frac{1}{n} \sum_{i=1}^n g_m(F_{n,1}(E_{i,1}),\dots,F_{n,p}(E_{i,p});\mathbf{\hat{\theta}}^{\text{oracle}}) = \int g_m(\mathbf{u};\mathbf{\hat{\theta}}^{\text{oracle}})\mathrm{d}C_n(\mathbf{u}) = 0. \label{eq:EE_oracle} \end{align}\] All integrals in this post are over \([0,1]^p\). To simplify our expression above and later, we have introduced the empirical copula \(C_n\) that is a multivariate distribution function on \([0,1]^p\) with a mass of \(1/n\) at each of \((F_{n,1}(E_{i,1}),\dots,F_{n,p}(E_{i,p}))^\top\), \(i\in\{1,\dots,n\}\) (precisely, \(C_n(\mathbf{u}) = \frac{1}{n} \sum_{i=1}^n \mathbf{1}\left\{ F_{n,1}(E_{i,1})\le u_1, \dots, F_{n,p}(E_{i,p})\le u_p \right\}\)). The qualifier “oracle” in \(\mathbf{\hat{\theta}}^{\text{oracle}}\) is used to distinguish the current case when we can still directly observe the copula sample \(\mathbf{E}_i\), \(i\in\{1,\dots,n\}\) (albeit without knowing \(F_1,\dots,F_p\)), from the case when even that sample will be subject to perturbation which we now turn to.

Residual-based pseudo-likelihood for semi-parametric copula adjusted for regression

From now on we suppose that the copula signal \(\mathbf{E}=\mathbf{E}_{\mathbf{\theta}^*}\) is “hidden” in a multivariate response semi- or non-parametric regression model, a setting considered in our recent work (Zhao, Gijbels, and Van Keilegom (2022)): for a covariate \(\mathbf{X}\in\mathbb{R}^q\) (independent of \(\mathbf{E}\)) and a response \(\mathbf{Y}=(Y_1,\dots,Y_p)^\top\in\mathbb{R}^p\), \[\begin{align*} Y_1 &= m_1(\mathbf{X}) + E_1 , \\ &\quad \vdots \\ Y_p &= m_p(\mathbf{X}) + E_p . \end{align*}\] In its raw form, the model above is a purely non-parametric regression model; by specifying particular forms of the regression function \(m_k\), the above will accommodate a wide range of popular non- and semi-parametric regression variants such as the partly linear regression model and the additive model. (It’s not much more difficult to consider a more flexible, heteroscedastic model \(Y_k = m_k(\mathbf{X}) + \sigma_k(\mathbf{X})E_k\), though we refrain from doing so in this post.) Gijbels, Omelka, and Veraverbeke (2015) considered a similar model and studied the resulting empirical copula process.

Under this regression model, we can observe an i.i.d. sample \((\mathbf{Y}_1,\mathbf{X}_1),\dots,(\mathbf{Y}_n,\mathbf{X}_n)\) of \((\mathbf{Y},\mathbf{X})\), but crucially not the copula sample \(\mathbf{E}_1, \dots, \mathbf{E}_n\). To eventually arrive at our estimator for \(\mathbf{\theta}^*\) in this setting, we will first form our empirical copula \(\hat{C}_n\) based on the residuals of the regression as follows. Let \(\hat{m}_k\) be some estimator for \(m_k\). Let’s estimate the \(k\)th component of \(E_i=(E_{i,1},\dots,E_{i,p})^\top\) by the residual \[\begin{align*} \hat{E}_{i,k} = Y_{i,k} – \hat{m}_k(X_i) . \end{align*}\] Then, we form the residual-based empirical distribution and copula: \[\begin{align*} \hat{F}_{n,k}(t) = \dfrac{1}{n+1} \sum_{i=1}^n \mathbf{1}\{ \hat{E}_{i,k} \le t \},\quad \hat{C}_n(\mathbf{u}) = \frac{1}{n} \sum_{i=1}^n \prod_{k=1}^p \mathbf{1}\{\hat{F}_{n,k}(\hat{E}_{i,k})\le u_k \}. \end{align*}\] Finally, to estimate \(\mathbf{\theta}^*\), we settle for the estimator \(\hat{\theta}^{\text{residual}}\) that solves \[\begin{align} \frac{1}{n} \sum_{i=1}^n g_m( \hat{F}_{n,1}(\hat{E}_{i,1}),\dots,\hat{F}_{n,p}(\hat{E}_{i,p}) ;\hat{\theta}^{\text{residual}}) = \int g_m(\mathbf{u};\hat{\theta}^{\text{residual}})\mathrm{d}\hat{C}_n(\mathbf{u}) = 0 . \label{eq:EE_residual} \end{align}\]

Comparing above equations, we would expect that when the residual-based empirical copula \(\hat{C}_n\) is asymptotically indistinguishable from the oracle empirical copula \(C_n\), the residual-based copula parameter estimator \(\hat{\theta}^{\text{residual}}\) should be asymptotically indistinguishable from \(\hat{\theta}^{\text{oracle}}\) as well. To formally reach this conclusion, standard estimating equation theory requires (among other conditions that we will ignore in this post) that the estimating equations at the truth should become indistinguishable, namely \[\begin{align} \int g_m(\mathbf{u};\mathbf{\theta}^*)\mathrm{d}\hat{C}_n(\mathbf{u}) – \int g_m(\mathbf{u};\mathbf{\theta}^*)\mathrm{d}C_n(\mathbf{u}) = o_p(n^{-1/2}) . \label{eq:EE_diff} \end{align}\]

One typical, although ultimately restrictive, approach to establish above equation is to invoke integration by parts (Neumeyer, Omelka, and Hudecová (2019), Chen, Huang, and Yi (2021)): ideally, this would yield \[\begin{align*} \int g_m(\mathbf{u};\mathbf{\theta}^*)\mathrm{d}\hat{C}_n(\mathbf{u}) \sim &\int \hat{C}_n(\mathbf{u})\mathrm{d}g_m(\mathbf{u};\mathbf{\theta}^*) \\ \stackrel{\text{up to}~o_p(n^{-1/2})}{\approx} & C_n(\mathbf{u})\mathrm{d}g_m(\mathbf{u};\mathbf{\theta}^*) \sim \int g_m(\mathbf{u};\mathbf{\theta}^*)\mathrm{d}C_n(\mathbf{u}) . \end{align*}\] In the above “\(\sim\)” is only meant to give a drastically simplified and hence not-quite-correct representation of integration by parts (we refer the readers to Appendix A in Radulović, Wegkamp, and Zhao (2017) for a precise multivariate integration by parts formula particularly useful for copulas), but it already conveys the underlying idea: the aim is to convert \(\hat{C}_n\) and \(C_n\) in the integrals from measures to integrands so that proving the closeness between the two integrals is clearly reduced to proving the closeness between \(\hat{C}_n\) and \(C_n\). However, the integration by parts trick, although popular, often requires bounded \(g_m\) to properly define the measure \(\mathrm{d}g_m\), but this is often not satisfied for even the most common copulas. For instance, in Gaussian copula, the score functions are quadratic forms of the \(\Phi^{\leftarrow}(\mathbf{u}_k)\)’s where \(\Phi^{\leftarrow}\) is the standard normal quantile function (see Eq. (2.2) in Segers, Akker, and Werker (2014)), so are clearly divergent as \(u_k\) approaches 0 or 1.

In Zhao, Gijbels, and Van Keilegom (2022) we instead adopted a more direct approach. Let \(g_m^{[k]}(\mathbf{u};\mathbf{\theta}^*)=\frac{\partial}{\partial u_k} g_m(\mathbf{u};\mathbf{\theta}^*)\). Then Taylor-expanding above equation we see that we will need (among other ingredients) an \(o_p(n^{-1/2})\) rate for the terms on the right-hand side of \[\begin{align} \int g_m(\mathbf{u};\mathbf{\theta}^*)\mathrm{d}({\hat{C}_n-C_n})(\mathbf{u}) \approx \sum_{k=1}^p \left[ \frac{1}{n} \sum_{i=1}^n g_m^{[k]}(F_k(E_{i,k});\mathbf{\theta}^*) \left\{ \hat{F}_{n,k}(\hat{E}_{i,k})-F_{n,k}(E_{i,k}) \right\} \right] . \label{eq:EE_diff_2} \end{align}\] It is not enough that the terms \(\hat{F}_{n,k}(\hat{E}_{i,k})-F_{n,k}(E_{i,k})\) on the right-hand side are \(o_p(n^{-1/2})\) (in fact they are not), due to the divergence of \(g_m^{[k]}\). We need to take a more careful look at \(\hat{F}_{n,k}(\hat{E}_{i,k})-F_{n,k}(E_{i,k})\), whose analysis belongs to residual empirical processes. To demonstrate the benefits of considering the weighted version of such processes, we first review some basic literature on the weighted (non-residual) empirical processes in the simplified setting of the real line.

Weighted empirical processes on the real line

In this section we consider estimating a distribution function \(F=F_U\) of a random variable \(U\). We rely on the empirical distribution function \(F_n\) constructed from the i.i.d. observations \(U_1, \dots, U_n\) of \(U\): \(F_n(t) = \frac{1}{n+1} \sum_{i=1}^n \mathbf{1}\left\{U_i\le t\right\}\). The resulting classical empirical process on the real line \(\sqrt{n} (F_n -F)(t)\), \(t\in \mathbb{R}\) must be one of the most extensively studied objects in all of probability; for illustration we will just quote a form of the associated law of the iterated logarithm (LIL): \[\begin{align*} \limsup_{n\rightarrow\infty} \sup_{t} \left|\dfrac{1}{\sqrt{\log\log(n)}} \sqrt{n} (F_n -F)(t) \right| = \dfrac{1}{\sqrt{2}} . \end{align*}\] Clearly, the LIL treats all points \(t\in \mathbb{R}\) equally. However, in reality the \(F(t)\) at some \(t\) is easier to estimate than others. This is essentially because the variability \(\text{var}\left\{F_n(t)\right\}=F(t)\left\{1-F(t)\right\}/n\) approaches \(0\) when \(F(t)\) approaches \(0\) or \(1\), for any \(n\). We can clearly observe this feature in the small simulation study represented by the following figure, where the “band” enclosing the deviations (based on 100 Monte-Carlo simulations) gets narrower toward the boundaries of the support of \(U\).

Above: Plot of the deviation \(F_n-F\), for sample size \(n=50\), based on 100 Monte-Carlo simulations. For simplicity we assumed \(U\sim{Unif}(0,1)\), so \(F(t)=t\) for \(t\in[0,1]\). The deviation from each simulation is represented by a single red dashed line. The 10% and 90% quantiles of the deviations at each \(t\) value are indicated by the two blue lines. Clearly, the “band” enclosing the deviations gets narrower toward the boundaries of the support of \(U\).]

This feature can also be characterized theoretically. For instance, we can find the LIL for the weighted process \(\sqrt{n}(F_n -F)/\sqrt{F}\), that is the classical process but now scaled by an additional standard deviation factor \(1/\sqrt{F}\), from Csáki (1977): for some \(0, \[\begin{align*} \limsup_{n\rightarrow\infty} \sup_{F(t)\in(\frac{1}{n},\frac{1}{2}]} \left|\dfrac{\log\log\log(n)}{\log\log(n)} \sqrt{n}\dfrac{(F_n -F)(t)}{\sqrt{F(t)}}\right| = C. \end{align*}\] Compared to the LIL for the unweighted process earlier, we can see that the weighted process is just slightly more difficult to bound, but now \(\sqrt{n}(F_n -F)(t)\) clearly enjoys a tighter bound toward the boundaries of the support of \(U\) due to the vanishing \(\sqrt{F(t)}\) there.

Such results on the weighted empirical processes can be generalized to settings beyond the real line, for instance to sets in \(\mathbb{R}^p\) (Alexander (1987)) and sets of functions (Giné, Koltchinskii, and Wellner (2003)).

Results for residual-based estimators

For us, the idea of the weighted empirical processes will be applied to the residual empirical processes, which will further culminate in our eventual result on the residual-based estimator \(\hat{\theta}^{\text{residual}}\) for the copula parameter. We will first consider the weighted residual empirical processes.