Probability theory, part II

Continuous random variable

A continuous random number is not limited to discrete values, but any continuous number within one or several ranges is possible.

Examples: weight, height, speed, intensity, …

A continuous random variable can be described by its probability density function, pdf.

Probability density function

\[ \int_{-\infty}^{\infty} f(x) dx = 1 \]

The area under the curve from \(a\) to \(b\) is the probability that the random variable \(X\) takes a value between \(a\) and \(b\).

\(P(a \leq X \leq b) = \int_a^b f(x) dx\)

Cumulative distribution function

The cumulative distribution function, cdf, \(F(x)\), is defined as:

\[F(x) = P(X \leq x) = \int_{-\infty}^x f(x) dx\]

Expected value and variance

Two important parameters of a distribution are the expected value, \(E(X) = \mu\), that describe the distributions location and the variance, \(\sigma^2\), that describe the spread.

The expected value, or population mean, is defined as;

\[E[X] = \mu = \int_{-\infty}^\infty x f(x) dx\]

The variance is defined as the expected value of the squared distance from the population mean;

\[\sigma^2 = E[(X-\mu)^2] = \int_{-\infty}^\infty (x-\mu)^2 f(x) dx\]

The square root of the variance is called the standard deviation, \(\sigma\).

Expected value and variance

As for discrete random variables, if the population is countable, \(E(X)\) and \(\sigma^2\) can also be computed by summing over all \(N\) objects in the population.

\[E[X] = \mu = \frac{1}{N}\sum_{i=1}^N x_i\]

\[\sigma^2 = \frac{1}{N} \sum_{i=1}^N (x_i-\mu)^2\]

Normal distribution

Normal distribution

As \(f(x)\) is well defined, values for the cumulative distribution function \(F(x) = \int_{- \infty}^x f(x) dx\) can be computed.

Standard normal distribution

Using transformation rules we can define

\[Z = \frac{X-\mu}{\sigma}, \, Z \sim N(0,1)\]

Values for the cumulative standard normal distribution, \(F(z)\), are tabulated and easy to compute in R using the function pnorm.

Standard normal distribution

Sum of two normal random variables

Sum of many random variables

This can be extended to the case with \(n\) independent and identically distributed normal random variables \(X_i \in N(\mu, \sigma)\).

\[\sum_{i=1}^n X_i \in N(n\mu, \sqrt{n}\sigma)\]

Central limit theorem, example

A left skewed distribution has a heavier left tail than right tail. An example might be age at death of natural causes.

Figure 1: A left skewed distribution. Can for example show the distribution of age of a mouse who died of natural causes.

Randomly sample 3, 5, 10, 15, 20, 30 observations and compute their mean value, \(m\). Repeat many times to get the distribution of mean values.

Central limit theorem, example

Figure 2: Distribution of sample means, where the means are computed based on random samples of sizes 3, 5, 10, 15, 20 and 30, respectively.

Note, mean is just the sum divided by the number of samples \(n\).

\(\chi^2\)-distribution

F-distribution

t-distribution

The ratio of a normally distributed variable and the square root of a \(\chi^2\)-distributed variable is t-distributed.

The ratio between sample mean and sample variance is t-distributed.

Random sampling

In many (most) experiments it is not feasible to examine the entire population. Instead we study a random sample.

A random sample is a random subset of individuals from a population.

There are different techniques for performing random sampling, two common techniques are simple random sampling and stratified random sampling.

Simple random sampling

A simple random sample is a random subset of individuals from a population, where every individual has the same probability of being choosen.

Simple random sampling using an urn model;

Let every individual in the population be represented by a ball. The value on each ball is the measurement we are interested in, for example height, shoe size, hair color, healthy/sick, type of cancer/no cancer, blood glucose value, etc.

Draw \(n\) balls from the urn, without replacement, to get a random sample of size \(n\).

Stratified random sampling

In stratified random sampling the population is first divided into subpopulations based on important attributes, e.g. sex (male/female), age (young/middle aged/old) or BMI (underweight/normal weight/overweight/obese). Simple random sampling is then performed within each subpopulation.

–>

–>

Sample properties

Summary statistics can be computed for a sample, such as the sum, proportion, mean and variance.

Sample proportion

The proportion of a population with a particular property is \(\pi\).

The number of individuals with the property in a simple random sample of size \(n\) is a random variable \(X\).

The proportion of individuals in a sample with the property is also a random variable;

\[P = \frac{X}{n}\] with expected value \[E[P] = \frac{E[X]}{n} = \frac{n\pi}{n} = \pi\]

Sample mean

For a particular sample of size \(n\); \(x_1, \dots, x_n\), the sample mean is

\[m = \bar x = \frac{1}{n}\displaystyle\sum_{i=1}^n x_i.\]

Note that mean of \(n\) independent identically distributed random variables, \(X_i\), is itself a random variable;

\[\bar X = \frac{1}{n}\sum_{i=1}^n X_i.\]

If \(X_i \sim N(\mu, \sigma)\), then \(\bar X \sim N\left(\mu, \frac{\sigma}{\sqrt{n}}\right)\).

When we only have a sample, the sample mean \(m\) is our best estimate of the population mean. It is possible to show that the sample mean is an unbiased estimate of the population mean, i.e. the average (over many size \(n\) samples) of the sample mean is \(\mu\), i.e. \(E[\bar X] = \mu\).

Sample variance

The sample variance is computed as;

\[s^2 = \frac{1}{n-1} \sum_{i=1}^n (x_i-m)^2.\]

The sample variance, \(S^2\), is also a random variable and it is possible to show that the sample variance is an unbiased estimate of the population variance.

Sampling distribution

A sampling distribution is a probability distribution of a sample property. A sampling distribution is obtained by sampling many times from the studied population.

Sample estimates of mean and variance are unbiased, but not perfect.

The standard deviation of the sampling distribution is called the standard error.

Standard error of the mean

For the sample mean, \(\bar X\), the variance is

\[E[(\bar X - \mu)^2] = \mathrm{var}(\bar X) = \frac{\sigma^2}{n}\] The standard error of the mean is thus;

\[SEM = \frac{\sigma}{\sqrt{n}}\] Replacing \(\sigma\) with the sample standard deviation, \(s\), we get an estimate of the standard deviation of the mean;

\[SEM \approx \frac{s}{\sqrt{n}}\]