Derivation

Let \(\xi_k,\) \(k=1,2,\ldots,n,\) be the result of \(k\)-th experiment, i.e. \(\xi_k=1\) if \(k\)-th experiment was successfull, and \(\xi_k=0\) otherwise. Then \[ X=\xi_1+\xi_2+\ldots+\xi_n, \] By assumption, \(\xi_1,\ldots,\xi_n\) are independent and each has a Bernoulli distribution with parameter \(p.\) Event \(\{X=k\}\) means that exactly \(k\) variables of \(\xi_1\ldots,\xi_n\) equal to \(1\) and others are equal to \(0.\) There are \({n\choose k}\) possibilities to choose variables that are equal to \(1.\) Each of them is \(1\) with probability \(p,\) other \(n-k\) variables are \(0\) each with probability \(1-p.\)

Moment generating function: \[ M(t)=(1-p+pe^t)^n \]

Proof

\[ M(t)=Ee^{t(\xi_1+\ldots+\xi_n)}= \] using independence \[ =Ee^{t\xi_1}Ee^{t\xi_2}\ldots Ee^{t\xi_n}=(pe^t+1-p)^n \]

Expectation: \(EX=np\)

Variance: \(V(X)=np(1-p)\)

Derivation

Expectation is the first derivative \(M'(0).\) We have \[ M'(t)=n(pe^t+1-p)^{n-1}pe^t, \ EX=M'(0)=np. \] Second moment is the second derivative \(M''(0).\) We have \[ M''(t)=n(n-1)(pe^t+1-p)^{n-1}p^2e^{2t}+n(pe^t+1-p)^{n-1}pe^t, \] \[ EX^2=M''(0)=n(n-1)p^2+np. \] Variance is \[ V(X)=EX^2-(EX)^2=n(n-1)p^2+np-n^2p^2=np(1-p) \]

Derivation

The event \(X=k\) means that first \(k\) experiments were successfull, and \((k+1)-\)st was not: \[ P(X=k)=P(\xi_1=1,\ldots,\xi_{k}=1,\xi_{k+1}=0)= \] by independence \[ =P(\xi_1=1)\ldots P(\xi_{k}=1)P(\xi_{k+1}=0)=p^{k}(1-p). \]

Moment generating function: \[ M(t)=\frac{(1-p)e^t}{1-pe^t}, \ t< \ln\frac{1}{p} \]

Proof

\[ M(t)=Ee^{tX}= \] using probability mass function \[ =\sum^\infty_{k=0} e^{tk}p^{k}(1-p)=(1-p)\sum^\infty_{k=0} (pe^t)^{k}= \] the sum of a geometric progression with a multiple \(pe^t<1\) (condition on \(t!\)) \[ =\frac{1-p}{1-pe^t} \]

Expectation: \(EX=\frac{p}{1-p}\)

Variance: \(V(X)=\frac{p}{(1-p)^2}\)

Derivation

Expectation is the first derivative \(M'(0).\) We have \[ M'(t)=\frac{(1-p)pe^t}{(1-pe^t)^2} \] \[ EX=\frac{p}{1-p} \] Second moment is the second derivative \(M''(0).\) We have \[ M''(t)=\frac{(1-p)pe^t}{(1-pe^t)^2}+\frac{2(1-p)p^2e^{2t}}{(1-pe^t)^3} \] \[ EX^2=\frac{p}{1-p}+\frac{2p^2}{(1-p)^2} \] Variance is \[ V(X)=\frac{p}{1-p}+\frac{p^2}{(1-p)^2}=\frac{p}{(1-p)^2} \]

Derivation

We find expectation as a sum over all possible values of \(X:\) \[ EX=\sum^n_{k=0}k \frac{{K\choose k}{N-K\choose n-k}}{{N\choose n}}= \] the summand with \(k=0\) is zero \[ =\frac{1}{{N\choose n}}\sum^n_{k=1}k {K\choose k}{N-K\choose n-k}= \] use the identity \(k{K\choose k}=K{K-1 \choose k-1}\) \[ =\frac{K}{{N\choose n}}\sum^n_{k=1} {K-1 \choose k-1 }{(N-1)-(K-1)\choose (n-1)-(k-1)}= \] change summation variable to \(k-1\) \[ =\frac{K}{{N\choose n}}\sum^{n-1}_{k=0} {K-1 \choose k }{(N-1)-(K-1)\choose (n-1)-k}= \] represent the sum as a sum of all probabilities for hypergeometric distribution with parameters \(N-1,K-1,n-1\) (which is 1) \[ =\frac{K{N-1\choose n-1}}{{N\choose n}}\sum^{n-1}_{k=0} \frac{{K-1 \choose k }{(N-1)-(K-1)\choose (n-1)-k}}{{N-1\choose n-1}}=\frac{K{N-1\choose n-1}}{{N\choose n}}=n\frac{K}{N} \]

Similarly we find \(EX(X-1):\) \[ EX(X-1)=\sum^n_{k=0} k(k-1)\frac{{K\choose k}{N-K\choose n-k}}{{N\choose n}}= \] \[ =\frac{1}{{N\choose n}}\sum^n_{k=2}k(k-1) {K\choose k}{N-K\choose n-k}= \] use the identity \(k(k-1){K\choose k}=K(K-1){K-2 \choose k-2}\) \[ =\frac{K(K-1)}{{N\choose n}}\sum^{n-2}_{k=0} {K-2 \choose k }{(N-2)-(K-2)\choose (n-2)-k}= \] \[ =\frac{K(K-1){N-2\choose n-2}}{{N\choose n}}=n(n-1)\frac{K(K-1)}{N(N-1)} \]

Variance is \[ V(X)=EX^2-(EX)^2=EX(X-1)+EX-(EX)^2=\] \[ =n(n-1)\frac{K(K-1)}{N(N-1)}+n\frac{K}{N}-n^2\frac{K^2}{N^2}= \] \[ =\frac{nKN^2-n^2KN-nK^2N+n^2K^2}{N^2(N-1)}=\frac{nK(N-k)(N-n)}{N^2(N-1)} \]

Derivation

Let \(\xi_l,\) \(l=1,2,\ldots,n,\) be the result of \(l\)-th experiment, \(\xi_l\in\{1,2,\ldots,k\}.\) The event \[ \{X_1=m_1,\ldots,X_k=m_k\} \] means that exactly \(m_1\) variables \(\xi_l=1,\) exactly \(m_2\) variables \(\xi_l=2,\) \(\ldots,\) exactly \(m_k\) variables \(\xi_l=k.\) When variables \(\xi_1,\ldots,\xi_l\) are already grouped according to their values, the probability becomes \(p^{m_1}_1\ldots p^{m_k}_k.\) The number of partitions of \(n\) elements into \(k\) groups by \(m_1,m_2,\ldots,m_k\) elements is \(\frac{n!}{m_1!\ldots m_k!}\).

Moment generating function: \[ M(t_1,\ldots,t_k)=Ee^{t_1X_1+\ldots+t_kX_k}=(p_1e^{t_1}+\ldots+p_k e^{t_k})^n \]

Proof

\[ M(t_1,\ldots,t_k)=Ee^{t_1X_1+\ldots+t_kX_k}= \] using probability mass function \[ =\sum_{m_1+\ldots+m_k=n}\frac{n!}{m_1!\ldots m_k!}p^{m_1}_1\ldots p^{m_k}_k e^{t_1m_1+\ldots +t_km_k}= \] \[ =\sum_{m_1+\ldots+m_k=n}\frac{n!}{m_1!\ldots m_k!}(p_1e^{t_1})^{m_1}\ldots (p_ke^{t_k})^{m_k}= \] by multinomial formula \[ =(p_1 e^{t_1}+\ldots+p_k e^{t_k})^n \]

Expectation: \(EX_j=np_j,\) \(1\leq j\leq k.\)

Variance: \(V(X_j)=np_j(1-p_j),\) \(1\leq j\leq k\)

Covariance: \(cov(X_i,X_j)=-np_ip_j,\) \(1\leq i<j\leq k.\)

Derivation

The moment generating function of a single variable \(X_j\) is obtained from \(M(t_1,\ldots,t_k)\) by letting \(t_i=0,\) \(i\ne j.\) That is \[ Ee^{t_jX_j}=M(0,\ldots,0,t_j,0,\ldots,0)=(p_1+\ldots+p_{j-1}+p_j e^{t_j}+p_{j+1}+\ldots+p_k)^n= \] using that \(p_1+\ldots+p_k=1\) \[ =(1-p_j+p_je^{t_j})^n \] This is exactly the moment generating function of a binomial distribution with parameters \(n,p_j:\) \[ X_j\sim Binomial(n,p_j) \] In particular \[ EX_j=np_j, \ V(X_j)=np_j(1-p_j). \] To compute expectation of the product \(EX_iX_j,\) \(i\ne j,\) we take second mixed derivative of \(M\) at point zero: \[ \frac{\partial M}{\partial t_i}=np_ie^{t_i}(p_1 e^{t_1}+\ldots+p_k e^{t_k})^n \] \[ \frac{\partial^2 M}{\partial t_i\partial t_j}=n(n-1)p_ip_je^{t_i+t_j}(p_1 e^{t_1}+\ldots+p_k e^{t_k})^{n-2} \] Put \(t_1=\ldots=t_k=0\) and get \[ E X_iX_j=n(n-1)p_ip_j \] So, the covariance \[ cov(X_i,X_j)=EX_iX_j-EX_iEX_j= \] \[ =n(n-1)p_ip_j-n^2p_ip_j=-np_ip_j. \]

Derivation

The event \(X=k\) means that when the \(r-\)th failure occurd there were exactly \(k\) successes. In particular it means that the process stopeed at \((k+r)\)-th experiment: \[ P(X=k)=P(\tau=k+r, S_{k+r}=k)= \] the last experiment is a failure \[ =P(\xi_{k+r}=0, S_{k+r-1}=k)= \] by independence \[ =(1-p)P(S_{k+r-1}=k)= \] the sum has a binomial distribution \[ =(1-p){k+r-1\choose k}p^{k}(1-p)^(r-1)={k+r-1\choose k}p^{k}(1-p)^r. \]

Moment generating function: \[ M(t)=\frac{(1-p)^r}{(1-pe^t)^r}, \ t< \ln\frac{1}{p} \]

Proof

\[ M(t)=Ee^{tX}= \] using probability mass function \[ =\sum^\infty_{k=0} e^{tk}{k+r-1\choose k}p^{k}(1-p)^r=(1-p)^r\sum^\infty_{k=0} {k+r-1\choose k}(pe^t)^{k}= \] the sum is a Taylor expansion of \(\frac{1}{(1-x)^r}\) around \(0,\) evaluated at \(x=pe^t\) (condition on \(t\) ensures convergence) \[ =\frac{(1-p)^r}{(1-pe^t)^r} \]

Expectation: \(EX=\frac{pr}{1-p}\)

Variance: \(V(X)=\frac{pr}{(1-p)^2}\)

Derivation

Expectation is the first derivative \(M'(0).\) We have \[ M'(t)=r\frac{p(1-p)^re^t}{(1-pe^t)^{r+1}} \] \[ EX=M'(0)=\frac{pr}{1-p} \] Second moment is the second derivative \(M''(0).\) We have \[ M''(t)=r\frac{p(1-p)^re^t}{(1-pe^t)^{r+1}}+r(r+1)\frac{p^2(1-p)^re^{2t}}{(1-pe^t)^{r+2}} \] \[ EX^2=\frac{pr}{1-p}+\frac{r(r+1)p^2}{(1-p)^2} \] Variance is \[ V(X)=\frac{pr}{1-p}+\frac{rp^2}{(1-p)^2}=\frac{pr}{(1-p)^2} \]

Derivation

The probability of \(k\) successes for binomial distribution with parameters \((n,p)\) is equal to \[ {n\choose k}p^k(1-p)^{n-k}. \] Let us find its limit when \(n\to \infty.\) \[ \lim_{n\to\infty}{n\choose k}p^k(1-p)^k=\frac{1}{k!}\lim_{n\to\infty}\frac{n!}{(n-k)!}p^k(1-p)^{n-k}= \] by Stirling formula, \(n!\sim \sqrt{2\pi n}\frac{n^n}{e^n}\) \[ =\frac{1}{k!}\lim_{n\to\infty}\frac{\sqrt{2\pi n}n^ne^{n-k}}{\sqrt{2\pi(n-k)}(n-k)^{n-k}e^n}p^k(1-p)^{n-k}= \] \[ =\frac{1}{e^kk!}\lim_{n\to\infty}\bigg(\frac{n}{n-k}\bigg)^{n-k}(np)^k(1-p)^{n-k}= \] \[ =\frac{\lambda^k}{e^kk!}\lim_{n\to\infty}\bigg(1-\frac{k}{n}\bigg)^{-(n-k)}\bigg(1-\frac{np}{n}\bigg)^{n-k}= \] use that \((1+x/n)^n\to e^x\) \[ =\frac{e^ke^{-\lambda}\lambda^k}{e^kk!}=e^{-\lambda} \frac{\lambda^k}{k!} \]

Moment generating function: \[ M(t)=e^{\lambda(e^t-1)} \]

Proof

\[ M(t)=Ee^{tX}=\sum^\infty_{k=0} e^{tk}e^{-\lambda} \frac{\lambda^k}{k!}= \] \[ =e^{-\lambda}\sum^\infty_{k=0}\frac{(\lambda e^t)^k}{k!}=e^{-\lambda}e^{\lambda e^t}=e^{\lambda(e^t-1)} \]

Expectation: \(EX=\lambda\)

Variance: \(V(X)=\lambda\)

Derivation

\[ M'(t)=\lambda e^t e^{\lambda(e^t-1)}, \ EX=M'(0)=\lambda \] \[ M''(t)=\lambda e^t e^{\lambda(e^t-1)}+\lambda^2 e^{2t} e^{\lambda(e^t-1)} \] \[ EX^2=M''(0)=\lambda+\lambda^2 \] Variance is \[ V(X)=\lambda \]