Zac Blanco     Blog     Education     Projects     About

Probability & Random Processes

These notes are for the version of the class which is taught by professor Yates in the spring of 2016.

Outline

Chapter 1 - January 21st 2016

We’re going to start first with Set Theory

Set Theory

A set is a collection of things. We usually will use capital letters to denote sets

parts of a set are called elements. When we use mathematical notation to denote an element as being part of a set we use the following symbol \in

Example: - x belongs to A

Set Unions

The union of sets and is the set of all elements that are either in A or in B or in both The union of A and B

Set Intersections

The intersection of sets is set of elements where each elements present in the set of A is also contained in B. We use the symbol to show the intersection of two sets.

Note!: Typically the is not used in problems and the following statements are equivalent:

Mutually Exclusive set what we call two sets with no common elements

We call the universal set the set which contains everything. That is if we imagine all of our elements being made up of sets then we get

A partition is a set which is mutually exclusive and collectively exhaustive part of S.

Theorem 1.1

De Morgan’s law relates all three basic operations

This can be proved by using the following logical steps

  1. Suppose x belongs the complement of the union of A and B
  2. Step 1 implies that x does not belong to the union of sets A and B
  3. Step 2 implies that x does not belong to set A nor set B.
  4. Step 3 implies that x must then belong to the complement of set A and the complement of set B.
  5. Step 4 Then must imply that x is contained within the intersection of sets and .

Applying Set Theory to Probability

An experiment consists of a procedure and observations. The is uncertainty in what will be observed; otherwise, performing the experiment would be unnecessary

an outcome of an experiment is any possible observation of that experiment

The sample space of an experiment is the finest-grain, mutually exclusive, collectively exhaustive set of all possible outcomes.

  • Sample space = universal set

an event is a set of outcomes of an experiment

Probability Axioms

A probability measure P[.] id s function that maps events in the sample space to real numbers.

  • Axiom 1 - For any event A,
  • Axiom 2 -
  • Axiom 3 For any countable collection of mutually exclusive events:

Theorem 1.3

if and for .

In other words

Theorem 1.4

The probability measure satisfies the following:

  • For any A and B (not necessarily mutually exclusive)
  • .

Theorem 1.5

The probability of an event is the sum of the probabilities of the outcomes contained in the event:

Theorem 1.6

For an experiment with sample space in which each outcome is equally likely:

Section 1.4 - Conditional Probability

Definition, Conditional Probability: The probability of the event A given the occurrence of event B is:

The vertical bar, “” stands for the word “given”.

What this equation does is “scale up” the probability of all events inside the event B. This effectively makes B the “new” sample space because all events that occur after B is given must be contained within B

Theorem 1.7

A conditional probability measure has the following properties that correspond to the axioms of probability.

  1. If with for then

Partitions and the Law of Total Probability

Theorem 1.8

For a partition and any event A in the sample space, let for , the events are mutually exclusive and:

Theorem 1.9

For any event A, and partition

Theorem 1.10

For a partition with for all i,

Theorem 1.11: Bayes’ Theorem

What this is saying is that the probability of an event B, given the event A, will be exactly equal to the probability of A divided by the probability of B, multiplied by the probability of A given B

Section 1.6 Independence

Definition: Independence - Events A and B are independent if and only if:

A comment on event independence and mutually exclusivity. The two are not synonyms. In some contexts they can have similar meanings. But in probability this is not the case.

Mutually exclusive events A and B have no outcomes in common and therefore . In most situations independent events are not mutually exclusive.

Exceptions occur only when . When we have to calculate probabilities, knowledge that events A and B are mutually exclusive is very helpful. Axiom 3 enables us to add their probabilities to obtain the probability of the union. Knowledge that events C and D are independent is also very useful. Definition 1.6 enables us to multiply their probabilities to obtain the probability of the intersection.

Definition 1.7 - Three Independent Events

Three events are mutually independent if and only if:

  • are independent
  • are independent
  • are independent

and finally as long as:

Definition 1.8 More than two Independent Events

This definition is the same as above. All collections of events must be mutually exclusive of one another,

The probability of intersections of these events must be equal to the product of their individual probabilities.

Chapter 2 - Tree Diagrams

Below is an example problem which will introduce us to tree diagrams


Tree Diagram Example Problem


From this we can see that tree diagrams help us to calculate probabilities of sequential events.

Tree diagrams display the outcomes of the subexperiments in a sequential experiment. The labels of the branches are probabilities and conditional probabilities. The probability of an outcome of the entire experiment is the product of the probabilities of branches from the root of the tree to a leaf.

Many experiments will consist of what we call subexperiments. The procedure followed for each subexperiment may depend on the results of the previous subexperiments.

In doing this we will assemble outcomes into different partitions, starting at the root of the tree. From there we branch (and label according probabilities) based on the probabilities of each outcome.

Section 2.2 - Counting Methods

In all application of probability theory, it should be important to understand the sample space of an experiment.

The methods in this section will help determine the number of outcomes in the sample space of an experiment.

Theorem 2.1

An experiment consists of two subexperiments. If one subexperiment has outcomes and the other subexperiment has outcomes, then the experiment has a total of outcomes.

Theorem 2.2

The number of -permutations of distinguishable objects is:

Something that might help to remember permutations is that a permutation depends on the order in which the elements are in whereas a combination does not depend on the order.

For example, think if we had 100 students. Imagine that we want to put any 50 of those students into a line. This means that the number of different permutations (arrangements) of that line is equal to

Theorem 2.3

The number of ways to choose k objects out of n distinguishable objects is:

Sometimes this is called a combination where the order in which items are picked does not matter, as opposed to the permutation.

Definition 2.1: choose

For any integer , we define

Theorem 2.4

Given distinguishable object, there are ways to choose with replacement an ordered sample of objects.

Theorem 2.5

for repetitions of a subexperiment with sample space , the sample space of the sequential experiment has outcomes

Theorem 2.6

The number of observation frequencies for n subexperiments with sample space with 0 appearing times and 1 appearing times is

Theorem 2.7

For repetitions of a subexperiment with sample space the number of length observation sequences with appearing times is

Independent Trials

Theorem 2.8

The probability of failures and successes in independent trials is:

Chapter 3 - Discrete Random Variables

Definition - Discrete Variables - A random variable assigns numbers to outcomes in the sample space of an experiment.

A probability model always begins with an experiments. And each of the random variables is directly related to the experiment.

There are 3 types of relationships

  1. The random variable is the observation
  2. The random variable is a function of the observation
  3. The random variable is a function of another random variable.

Throughout most of the book probability is examined in models that assign numbers to the outcomes of the sample space.

When we observe the observation we call it a random variable. Typically in our notation the name of our random variable is a capital letter. e.g. .

The set of possible values for is the range of

Since we often consider multiple variables at a time, we denote the range of a random variable by the letter with a subscript that is the name of the random variable.

Thus, is the range of the random variable , is the range of the random variable and so on…

Definition 3.1

Random Variable consists of an experiment with a probability measure defined on a sample space and a function that assigns a real number to each outcome in the sample space of the experiment.

A note on notation:

  • On occasion should we need to identify the random variable with respect to a sample outcome, then we define this as . (In other words we define a function which relates the value of X to the outcomes - X is a function of , the outcome)

  • Sometimes we will write to emphasize that there is a set of sample points for which .

We can then adopt the following shorthand notation:

Definition 3.2

is a discrete random variable if the range of is a countable set

Definition 3.3

Probability Mass Function (PMF) - The probability mass function of the discrete random variable X is

A way of putting this is that for a certain discrete random variable which is described by , then saying is the probability of the event in which that occurs, or in other words it is , where the big is the random variable and simply denotes that it is the PMF function.

Theorem 3.1

For a discrete random variable with a PMF of and range

  1. For any event , the probability that is in the set is

To put this into words we can say that to calculate the probability of an event, , then it is the sum total probability of all outcomes for which are contained within

Families of Random Variables

  • In practical applications, certain types of random variables appear over and over across many experiments.
  • In each of these families (types) the PMF of all the random variables will have the same mathematical form
  • Differ only in values of one or two parameters
  • Depending on family, each PMF will contain only 1-2 parameters
  • If we assign numbers to the parameters we can obtain a specific variable.
  • Typical nomenclature for a family consists of the family name followed by one or two parameters in parentheses.

Example: binomial(n, p) refers in general to the family of binomial random variables

Binomial(7, 0.1) refers to the binomial random variable with parameters and

Definition 3.4

X is a Bernoulli (p) random variable if the PMF of X has the form

  • The number of successes in a single trial.

Definition 3.5 Geometric (p) Random Variable

X is a geometric (p) random variable if the PMF of X has the form

  • Number of trials until (up to and including) the first success

Definition 3.6 Binomial (n, p) Random Variable

X is a binomial (n, p) random variable if the PMF of X has the form

Where and is an integer such that

  • the number of successes in n trials

Definition 3.7 Pascal (k, p) Random Variable

X is a pascal (k, p) random variable if the PMF of X has the form:

Where and k is an integer such that

  • the number of trials until the success

If then x is geometric.

  • Bernoulli (p) = number successes in 1 trial
  • Geometric (p) = number of trials until 1st success
  • Binomial (n, p) = number of successes in n trials
  • Pascal (k, p) = number of trials until k successes
Name of Random Variable Description of variable PMF Function
Bernoulli number successes in 1 trial
Geometric number of trials until 1st success
Binomial number of successes in n trials
Pascal number of trials until k successes

Definition 3.8 Discrete Uniform Random Variable

X is a discrete uniform random variable if the PMF of X has the form:

  • X is uniformly distributed between and .

Definition 3.9 Poisson () Random Variable

X is a poisson () random variable if the PMF of X has the form

  • Call the phenomena of interest an arrival.
  • Poisson model specifies an average rate.
  • arrivals per second, over a time interval of seconds.
  • The number of arrivals has a Poisson PMF with

Section 3.4 - Cumulative Distribution Function (CDF)

Like the PMF, the CDF of a random variable X expresses the probability model of an experiment as a mathematical function. The function is the probability for every number .

The cumulative distribution function (CDF) of a random variable, X, is

Theorem 3.2

  • - The probability which X is less than infinity.
  • - The probability which X is less than infinity.
  • For all
  • For and , and arbitrarily small positive number
  • for all x such that

Theorem 3.3

For all

In other words, to find the probability of an event between two values, you can simply subtract the values of the CDF at and because it is the sum total probability up to each of those points, subtracting them results in only the probability of values between and .

Definition 3.13 Expected Value

The expected value of is

This is sometimes referred to as the mean of a random variable or function

Theorem 3.4

The Bernoulli random variable has the expected value

Theorem 3.5

The geometric random variable has the expected value

Theorem 3.6

The Poisson () random variable in definition 3.9 has the expected value

Theorem 3.7

1. For the binomial random variable of definition 3.6

2. For the pascal random variable of definition 3.7

3. For the discrete uniform random variable of definition 3.7

Expected Value Table

Function Type Expected Value
Bernoulli
Geometric
Poisson
Binomial
Pascal
Discrete Uniform

Theorem 3.8

Perform Bernoulli trials. In each trial let the probability of success be where is a constant and . Let the random variable be the number of successes in trials. As then converges to the PMF of a Poisson random variable

Definition 3.11 Mode

A mode of random variable X is a number satisfying for all x

Definition 3.12 Median

A median of random variable X is a number that satisfies these properties

Functions of a Random Variable

Definition 3.14

Each sample value y of a derived random variable Y is a mathematical function of a sample value x of another random variable X. We adopt the notation to describe the relationship of the two random variables.

Definition 3.9

For a discrete random variable X, the PMF of is

Expected Value of a Derived Random Variable

Theorem 3.10

Given a random variable X with the PMF and derived random variable , the expected value of is

Theorem 3.11

For any random variable X,

Theorem 3.12

For any random variable X,

Variance and Standard Deviation

Definition 3.15 Variance

The variance of a random variable X is:

You might have heard variance before as the square of standard deviation (). Thus follows the next definition.

Definition 3.16 Standard Deviation

The standard deviation of a random variable X is:

Theorem 3.13

In the absence of observations, the minimum mean square error estimate of random variable X is:

Theorem 3.14

Theorem 3.17

For the Random variable, X

a. The nth moment is b. The nth central moment is

Theorem 3.15

Theorem 3.16

Variance Table

Function Type Variance
Bernoulli
Geometric
Poisson
Binomial
Pascal
Discrete Uniform

Full Table of Discrete Random Variables

Name of Random Variable Description of variable PMF Function Expected Value- Variance -
Bernoulli number successes in 1 trial
Geometric number of trials until 1st success
Binomial number of successes in n trials
Pascal number of trials until k successes
Poisson Probability of x arrivals in T seconds
Discrete Uniform Probability of any value between k and l

Chapter 4 - Continuous Random Variables

Example:

Imagine a circle with an infinite amount of points around the circle. If we were to pick any one point at random. What would the probability of picking that point be?

So what if we ask, what is the probability we choose the point of , or .

Well, it turns out, the answer, . This will lead us into the first section

Section 4.1 - Continuous Sample Space

In a continuous sample space, there are an infinite number of points you could choose. But the probability of picking a single point is zero.

Section 4.2 - The Cumulative Distribution Function

Definition 4.1 - Cumulative Distribution Function (CDF)

Recall this from previous sections. It applies to continuous random variables as the probability of any single event is 0, but for all values up to and including a single value is how we calculate the probability of a continuous function.

Theorem 4.1

For any random variable X

Definition 4.2 - Continuous Random Variable

is a continuous random variable if the CDF is a continuous function

Section 4.3 - The Probability Density Function

The probability density function (PDF) of a continuous random variable X is:

This function is simply defined as the derivative of the CDF function for a continuous random variable.

Theorem 4.2

For a continuous random variable X with a PDF of

Theorem 4.3

Section 4.4 - Expected Values

The expected value of a continuous random variable X is:

Theorem 4.4

The expected value of a function of a random variable X is:

Theorem 4.5

For any random variable

Section 4.5 - Families of Continuous Random Variables

Definition 4.5 Uniform Random Variable

X is a uniform (a, b) random variable if the PDF of X is

Where the condition is that the parameter

Theorem 4.6

If X is a uniform (a, b) random variable

  • The CDF of X is:
  • The expected value of X is
  • The variance of X is

Theorem 4.7

Let X be a uniform random variable, where a and b are both integers. Let . Then K is a discrete uniform random variable.

Note that is the “ceiling” of the range of X, or the greatest value in the set

Definition 4.6 - Exponential Random Variable

X is an exponential () random variable if the PDF of X is:

Theorem 4.8

If X is an exponential () random variable

  • The CDF of X is:
  • The expected value of X is
  • The variance of X is

Theorem 4.9

If X is an exponential () random variable, then is a geometric (p) random variable with

Definition 4.7 - Erlang Random Variable

X is an Erlang () random variable if the PDF of X is:

where the parameter of and the parameter is an integer.

Theorem 4.10

If X is an Erlang () random variable, then

Theorem 4.11

Let denote a Poisson () random variable. For any , the CDF of an Erlang () random variable X, satisfies

Section 4.6 - Gaussian Random Variables

Definition 4.8 - Gaussian Random Variable

is a Gaussian random variable if the PDF of is:

The parameter can be any real number and the parameter

Theorem 4.12

If X is a Gaussian random variable:

and

Theorem 4.13

If X is a Gaussian random variable, is Gaussian

Definition 4.9 - Standard Normal Random Variable

The standard normal random variable is the Gaussian (0, 1) random variable

Definition 4.10 - Standard Normal CDF

The CDF of the standard normal random variable Z is:

Theorem 4.14 - If X is a Gaussian random variable, then the CDF of X is:

And the probability that X is in the interval of (a, b] (includes b, excludes a):

Theorem 4.15

Definition 4.11 - Standard Normal Complementary CDF

The standard normal complementary CDF is:

Another way of looking at is is that

Section 4.7 - Delta Functions, Mixed Random Variables

Definition 4.12 - Unit Step Function

The unit step function is:

Definition 4.12 - Unit Impulse (Delta) Function

Let

Then the unit impulse function is

Now as approaches the delta function for each the area under the curve of the equals 1

Theorem 4.16

For any continuous function g(x)

Theorem 4.17

Chapter 5 - Multiviews and Pairs of Random Variables

Pairs of Random Variables

We should consider experiments that produce a collection of random variables .

A pair of random variables and is the same as the two dimensional vector, this way. Since the components of this vector are both random variables we call a random vector.

Definition 5.1 - Joint Cumulative Distribution Function (CDF)

The joint cumulative distribution function of random variables and is

So if you can imagine a two dimensional graph that the CDF represents the probability of being within a rectangular region which the point that is picked is x and y

Theorem 5.1

For any pair of random variables

  • If then

Theorem 5.2

Section 5.2 - Joint Probability Mass Function (PMF)

Definition 5.2 - Joint PMF

The joint PMF of discrete random variables and is

Theorem 5.3 For discrete random variables and and any set in the plane, the probability of the event is

Section 5.3 - Marginal PMF

Theorem 5.4

For Discrete random variables X and Y with joint PMF

Section 5.4 Joint Probability Density Function

Definition 5.3 - Joint PDF

The joint PDF of the continuous random variables and is a function with the property

Theorem 5.5

Theorem 5.6

A joint PDF has the following properties corresponding to the first and second axioms of probability

Theorem 5.7

The probability of the continuous random variables (X, Y) are in A is:

Section 5.5 - Marginal PDF

Theorem 5.8

If and are random variables with joint PDF

Section 5.6 - Independent Random Variables

Definition 5.4 - Independent Random Variables

Random variable and are independent if and only if the following are true:

For discrete random variables:

For continuous random variables:

Section 5.7 - Expected Value of a Function of Two Random Variables

Theorem 5.9

For random variables and the expected value of is:

For discrete random variables:

For continuous random variables:

Theorem 5.10

Theorem 5.11

For any two random variables X and Y

Theorem 5.12

The variance of the sum of two random variables is:

Definition 5.5 - Covariance

The covariance of two random variables and is:

Definition 5.6 - Correlation Coefficient

The correlation coefficient of two random variables X and Y is:

Theorem 5.13

If and , then

Theorem 5.14

Theorem 5.15

If X and Y are random variables such that then:

Definition 5.7 - Correlation

The correlation of two variables and is

Theorem 5.16

  • .

  • .

  • If , and

Definition 5.8 - Orthogonal Random Variables

Random variables X and Y are orthogonal if

Definition 5.9 - Uncorrelated Random Variables

Random variables and are uncorrelated if

Theorem 5.17

For independent random variables X and Y

Section 5.9 - Bivariate Gaussian Random Variables

Definition 5.10 - Bivariate Gaussian Random Variables

Random variables and have bivariate Gaussian PDF with the parameters , , , , and that satisfying if

Theorem 5.18

If X and Y are bivariate Gaussian random variables in Definition 5.10, X is the Gaussian and the random variable Y is the Gaussian random variable:

Theorem 5.19

Bivariate Gaussian random variables X and Y in Definition 5.10 have the correlational coefficient

Theorem 5.20

Bivariate Gaussian random variables X and Y are uncorrelated if and only if they are independent

Theorem 5.21

If X and Y are bivariate Gaussian random variables with PDF given by definition 5.10 and and are given by the linearly independent equations

Then, and are bivariate Gaussian random variables such that

  • where
  • where

Section 5.10 - Multivariate Probability Models

Definition 5.11 - Multivariate Joint CDF

The joint CDF of is:

Definition 5.12 - Multivariate Joint PMF

The joint PMF of the discrete random variables is:

Definition 5.13 - Multivariate Joint PDF

The joint PDF of the continuous random variables of is:

Theorem 5.22

if are discrete random variables that have a joint PMF

Theorem 5.23

if are continuous random variables that have a joint PDF

Theorem 5.24

The probability of an event A expressed in terms of the random variables is:

Discrete:

Continuous:

Theorem 5.25

For a joint PMF of discrete random variables some marginal PMFs are:

Theorem 5.26

For a joint PDF of discrete random variables some marginal PDFs are:

Definition 5.14 - N Independent Random Variables

Random variables are independent if for all

Discrete:

Continuous

Definition 5.15 - Independent and Identically Distributed (iid)

are independent and identically distributed if:

Discrete:

Continuous

Chapter 6 - Probability Models of Derived Random Variables

Section 6.1 - PMF of a Function of Two Discrete Random Variables

Theorem 6.1

For discrete random variables X and Y, the derived random variable has the PMF

Section 6.2 - Functions Yielding Continuous Random Variables

Theorem 6.2

If where then has a CDF and PDF of:

CDF PDF

Theorem 6.3

Given that where

  • If is uniform then is uniform
  • If is exponential then is exponential
  • If is Erlang then is Erlang
  • If is Gaussian then is Gaussian

Theorem 6.4

Given that

CDF PDF

Theorem 6.5

Let be a uniform random variable and let denote a cumulative distribution function with an inverse defined for the random variable has CDF

Section 6.4 - Continuous Function of Two Continuous Random Variables

Theorem 6.6

For continuous random variables and , the CDF of is

Theorem 6.7

For continuous random variables and , the CDF of is:

Chapter 7 - Conditional Probability Models

Section 7.1 - Conditioning a Random Variable by an Event

Definition 7.1 - Conditional CDF

Given the event with the conditional cumulative distribution function of X is

Definition 7.2 - Conditional PMF Given an Event

Given the event B with , the conditional probability mass function of X is:

Definition 7.3 - Conditional PDF Given an Event

Theorem 7.1

For a random variable X and an event with , the conditional PDF of given is:

Discrete

Continuous

Theorem 7.2

For random variable resulting from an experiment with partition

Discrete:
Continuous:

Section 7.2 - Conditional Expected Value Given an Event

Theorem 7.3

Discrete X Continuous X
for any for any
The conditional probability that X is in C: The conditional probability that X is in the set C is:

Definition 7.4 - Conditional Expected Value

Discrete:
Continuous:

Theorem 7.4

For a random variable resulting from an experiment with partitions

Theorem 7.5

The conditional expected value of given the condition, is:

Discrete:
Continuous:

Section 7.3 - Condition Two Random Variables by an Event

Definition 7.6 - For discrete random variables and , and event, with , the conditional joint PMF of and given is:

Theorem 7.6

For any event , a region of the plane with

Definition 7.7 - Conditional Joint PDF

Given an event with , the conditional joint probability density function of and is:

Theorem 7.7 - Conditional Expected Value

For random variables and and an event of nonzero probability, the conditional expected value of given is:

Discrete:
Continuous:

Section 7.4 - Condition by a Random Variable

Definition 7.8 - Conditional PMF

For any event such that , the conditional PMF of given

Theorem 7.8

For discrete random variables and with joint PMF , and and such that and

Definition 7.9 - Conditional PDF

For y such that , the conditional PDF of given is:

Theorem 7.9

For discrete random variables and with joint PMF and and such that and :

Theorem 7.10

For continuous random variables and with joint PDF and and such that and :

Theorem 7.11

If X and Y are independent:

Discrete: , and
Continuous: , and

Section 7.5 - Conditional Expected Value Given a Random Variable

Definition 7.10 - Conditional Expected Value of a Function

For any , the conditional expected value of given is:

Discrete:
Continuous:

Theorem 7.12

For independent random variables and

  • for all
  • for all

Definition 7.11 - Conditional Expected Value Function

The conditional expected value is a function of random variable Y such that if , then

Theorem 7.13 - Iterated Expectation

Theorem 7.14

Theorem 7.15

If and are the bivariate Gaussian random variables in Definition 5.10, the conditional PDF of given is:

where, given , the conditional expected value and variance of are:

Theorem 7.16

If and are the bivariate Gaussian random variables in Definition 5.10, the conditional PDF of given is:

where, given , the conditional expected value and variance of are:

Chapter 9 - Sums of Random Variables

Section 9.1 - Expected Values of Sums

Theorem 9.1

For any set of random variables , the sum has the expected value:

Theorem 9.2

The variance of is:

Theorem 9.3

When are uncorrelated:

Section 9.2 - Moment Generating Functions

Definition 9.1 - Moment Generating Function (MGF)

For a random variable , the moment generating function (MGF) of is:

Theorem 9.4

A random variable with MGF has n-th moment:

Theorem 9.5

The MGF of is

Section 9.3 - MGF of the Sum of Independent Random Variables

Theorem 9.6

For a set of independent random variables , the moment generating function of is:

When are iid, then each with MGF

Theorem 9.7

If are independent Poisson random variables, is a Poisson random variable

Theorem 9.8

The sum of independent Gaussian random variables is a Gaussian random variable

Theorem 9.9

If are iid exponential () random variables, then has the Erlang PDF:

Theorem 9.12 - Central Limit Theorem

Given a sequence of iid random variables with expected value and variance , the CDF of

Definition 9.2 - Central Limit Theorem Approximation

Let be the sum of iid random variables, each with and . The central limit theorem approximation of the CDF of is:

Chapter 10 - The Sample Mean

Definition 10.1 - Sample Mean

For iid random variables with PDF , the sample mean of is the random variable:

Theorem 10.1

The sample mean has the expected value and variance

Mean:

Variance:

A note on the sample variance: the the above theorem actually states that the sample mean converges to the expected value as the number of sample, approaches

Section 10.3 - The Laws of Large Numbers

Theorem 10.5 - Weak Law of Large Numbers (Finite Samples)

For any constant

Theorem 10.6 The Weak Law of Large Numbers (Infinite Samples)

If has finite variance, then, for any constant

Theorem 10.7

as , the relative frequency converges to for any constant