Statistics - Fanyu Zhao

The notes is not only a review for the preparetion of quants, but also hopelfully a learning notes for a junior PhD student and also my babe.

Key Points

1. Prelimiaries

Firstly and most importantly, I need to declare what is statistics and why shall we learn statistics.The following only based on my own understanding. My understanding is pretty pretty limited (I got only a master degree, so I am definitely not an expert in Statistics) and subjective, and please provide your suggestions and even your blames to me. Glad to know your ideas.

1.1 What is Statistics?

For my understanding, Statistics is a tool, characterised by mathematics, to explain the world. Such a bull shit am I talking.

Be serious. I may say that statistics is a process to estimate the population by samples.

To do study about the population is always costly, and pretty much unpredictable. For example, to do test in the individual level, we have to collect data from all the people. The population census could only be done in a national level and conducted by the gov. Even that, the census is unable to be performed in a year by year basis, and there are measurement errors always. Thus, a more cost-effective way would be to estimate the population through the data from a small set of people who are randomly selected.

Another example could be the weather forecast, which is similar as doing a time series analysis or panel data analysis. Tthe forecast may most likely to be biased, because things changes unpredictabily and irregularly. So we may say that is even impossible to and the full data to estimate the population (factors related to weather in this case). Thus, a simpler way might be that we collect different factors and historical data about such as termperature, because we may assume the temperature changes are consistent over a short period of time.

However, there are gaps between population and sample. How could we connect those gaps? The answer is Statistics. Statistics provide some mathematical proven methods to make the sample have a better capture about the population, based on assumptions.

Let's begin our study.

2. Probability

2.1 Conditional Probability

2.2 Mutually Exclusive and Independent Events

If two events are independent, then

3. Random Variables - r.v.

3.1 Definition

Random Variables: $X,Y,Z$$X, Y, Z$

Observations: $x,y,z$$x,y,z$

3.2 Probability Mass/Density Funciton - p.d.f. (For Discrete r.v. or Continuous r.v.)

3.2.1 Definition

p.d.f captures the probability that a r.v. $X$$X$ has a given value of $x$$x$.

3.2.2 Properties of p.d.f.

1. $f(x)\ge 0$$f(x)\geq 0$, since probability is always positive.
2. ${\int }_{-\mathrm{\infty }}^{+\mathrm{\infty }}f(x)dx=1$$\int_{-\infty}^{+\infty} f(x)\ dx=1$
3. $P(a<X<b)={\int }_a^{b}f(x)dx$$P(a<X<b)=\int_a^b f(x) \ dx$

Repalce the intergal with summation for discrete r.v.

P.S. For continuous r.v. $X$$X$, $P(X=x)=0$$P(X=x)=0$. That means for a continuous r.v., any points on the p.d.f have a zero probability.

For example, the probability of selecting a number “3” among 1 to 10 is zero.

3.3 Cumulative Distribution Function - c.d.f

3.4 Expectation

3.5 Variance and Standard Deviation

3.6 Moments

The first moment, $E(X)=\mu$$E(X)=\mu$.

The $n^{th}$$n^{th}$ moment, $E(X^n)={\int }_x{x}^{n}f(x)dx$$E(X^n)=\int_x x^n\ f(x)\ dx$.

The second central moment about mean. $E(x-E(X))={\sigma }^2$$E(x-E(X))=\sigma^2$, Variance.

The third central moment, $E(X-E(X){)}^3$$E(X-E(X))^3$. $Skewness=\frac{E(X-E(X){)}^3}{{\sigma }^3}$$Skewness = \frac{E(X-E(X))^3}{\sigma^3}$. Standard normal dist has a Skewness of 0. (Right or Left Tails)

The Fourth central moment, $E(X-E(X){)}^4$$E(X-E(X))^4$. $Kurtosis=\frac{E(X-E(X){)}^4}{{\sigma }^4}$$Kurtosis = \frac{E(X-E(X))^4}{\sigma^4}$. Standard normal dist has a Kurtosis of 3. (Fat or This, Tall or Short).

3.7 Covariance

4. Distribution

The meaning of distributions, and the properties (mean & var).

4.1 Bernoulli DIst

4.2 Binomial Dist

4.3 Possion Dist

4.4 Normal Dist & Standard Normal

For a standard normal dist,

4.4.1 Standardisation

4.4.2 Propertities of Normal Dist

One / Two /Three standard deviation regions.

5. Central Limit Theorem - CLM

i.i.d. - idependent identical distributed

Suppose $X_1,X_2,...,X_n$$X_1,X_2,...,X_n$ are $n$$n$ idependent r.v., each have same distribution, and as the number $n$$n$ increases, the distribtuion of

would behave like normal distribution.

Key facts:

1. The distribtuion of $X$$X$ is not stated. We do not have to restrict the distribution of r.v.s, as long as they are in same dist.
2. If $X$$X$ is a r.v. with mean $\mu$$\mu$ and standard deviation $\sigma$$\sigma$ from a random dist, the CLT tells that the distribution of sample mean, $\overline{X}$$\bar{X}$ is normal dist.

Therefore, we would get the distribution of $\overline{X}$$\bar{X}$,

By standardising it,

Also, for $S_n=X_1+X_2+...+X_n$$S_n=X_1+X_2+...+X_n$.

The more obervations there are, the more similar the distribution to normal would be. Also, the less standard deviation means the estimate has less variations and is more accurate.

Why is CLT important?

It is important because it provide a way to use repeated obersevations to estimate the whole population, which is impossible to be observed.

6. A Few Notations

Recall, our aim of using statistics is to find the true population. We may assume the true population follows a distribtuion, and that distribution has some parameters. What we are doing right now is to use the sample data (feasiblly collectable) to presume the population parameters.

• Estimator: a function, using sample or available data, to estimate the population. i.e. $\overline{x}$$\bar{x}$ and $S^2$$S^2$.
• Estimate: the value/figure we truely calculated. By inputing data into estimator, the output is the estimate.

Population (Population Parameters that we want to get but can never get)

Population Mean: $\mu =\frac{\sum x_i}{N}$$\mu=\frac{\sum x_i}{N}$.

Population Variance: ${\sigma }^2=\frac{\sum (x_i-\mu {)}^2}{N}$$\sigma^2=\frac{\sum (x_i-\mu)^2}{N}$.

Sample Estimator

Sample Mean: $\overline{x}=\frac{\sum x_i}{N}$$\bar{x}=\frac{\sum x_i}{N}$.

Sample Variance: ${\hat{\sigma }}^2=\frac{\sum (x_i-\overline{x}{)}^2}{N}$$\hat{\sigma}^2=\frac{\sum (x_i-\bar{x})^2}{N}$

Throw data into sample estimators would get the estimates, and those estimates are then applied to presume the population parameters.

Remember that sample is only part of population ,we collect data from the sample is because they are more accessible and feasible to get. Still, we need to use our sample data to be representatitive to the population, or in another word, to have some foreseers about the whole population. Therefore, we use a different notation on sample statistics.

A important aspect is that we needs our sample to have better representativeness of the population. There are some measurements.

6.1 Unbiasness

If $E(\overline{X})=\mu ,\text{or}E(S^2)={\sigma }^2$$E(\bar{X})=\mu, \text{or} \ E(S^2)=\sigma^2$ (the expectation of our sample estimate is equal to the population), then we would say the estimator is unbiased.

The unbiased estimator of sample variance is $S^2=\frac{\sum (x_i-\overline{x}{)}^2}{n-1}$$S^2=\frac{\sum (x_i-\bar{x})^2}{n-1}$.

Why the denominator is "n-1"?

There would be a long discussion to talk about that. We can simply understand "-1" as the adjustment of the $\overline{x}$$\bar{x}$ in the numerator, because $\overline{x}$$\bar{x}$ is calculated to represent the population mean $\mu$$\mu$ and $\overline{x}$$\bar{x}$ is not intrinsicly available (it is costly, to save for the cost, the denominator has a deduction).

In sum, $S^2$$S^2$ is an unbiased estimator of population variance, ${\sigma }^2$$\sigma^2$. We also have a special name for the sample standard deviation, Standard Error, s.e..

6.2 Consistency

If there is a estimator such that as $n\to \mathrm{\infty }$$n\rightarrow \infty$, the estimator goes close to the population parameter, we may say that estimator is consistent.

For example, although ${\hat{\sigma }}^2=\frac{\sum (x_i-\mu {)}^2}{N}$$\hat{\sigma}^2=\frac{\sum (x_i-\mu)^2}{N}$ is biased, it is consistent if the number of observation keeps increasing.

Flawness of discussion is available in this section, awaiting to be updated.

7. Estiamtion

7.1 Maximum Likelihood Estimation - MLE

By assuming a probability distribtuion of the r.v. $X$$X$, fitting into sample observations and trying to find the parameters that can maximise the joint probability (likelihood function).

To illustrate the problem, we need to find the parameters $\lambda$$\lambda$ that can maximise the likelihood function.

The value of the parameters ${\lambda }_0$$\lambda_0$ is our MLE estimator. (Remember what estimator is? See section 6).

For example

Assume r.v. $X\sim N(\mu ,{\sigma }^2)$$X\sim N(\mu,\sigma^2)$. Let $x_1,x_2,...,x_n$$x_1,x_2,...,x_n$ be a random sample of i.i.d. observations. We use MLE to find the value of $\mu$$\mu$ and ${\sigma }^2$$\sigma^2$. So, we need to maximise the log-likelihhod function (instead of using likelihood function, we do a logarithm tranformation for easier calculation. Because the log tranformation is monotonoic, the tranformation is legal).

$F.O.C.$$F.O.C.$

We would find the MLE estimators are same as the OLS estimator in the following section.

7.2 Regression

Assume a linear model through which we can have a minimum sum mean squred.

, where

$F.O.C.$$F.O.C.$

By Fanyu Zhao