Introduction to Probability, 2nd Edition\nFollows the course, can borrow from engineering library
\nTODO:\nSet calendar for homework self grades and resubmission time\nSet midterm time on calendar\nChoose discussion time\nCountable vs uncountable
\nHumans are naturally good at thinking about probability. For example, you can easily walk down a crowded street and estimate what path allows you to avoid bumping into people. However, humans are pretty awful at turning those innate estimates into quantifiable numbers. The goal is to take the natural probability estimation into something that a robot can understand.
\nQuantify + Model
\nInference. Humans are also really good at taking in noisy observations and infer what is really happening.
\nHow do I generate some image from AI? Take a natural image and repeatedly add noise to it. You then end up with an image of pure noise. The idea behind stable diffusion and dalle-2 is to reverse this process. We generate a bunch of noise, then reduce noise until we end up with a image.
\nModern theory of probability has an axiomatic starting point. Before the 1930s, probability was a heuristic field.\nAll of probability are logical deductions from the starting point.
\nProbability space: A probability space (Ω, F, P) is a triple.\nΩ is a set, the sample space.\nF is a family of subsets of Ω, called events\nP is a probability measure
\nRules:
\nF is a σ-algebra, it contains Ω itself. F is closed under complements and countable unions.
\nP is a function that maps events to a number between 0 and 1. P: F -> [0,1]
Probability measures must obey Komogorov Axioms
\nP(A) >= 0 Probability of any event must be non-negativeP(Ω) = 1 The probability of any outcome occuring must be equal to 1P(U) = ΣP(Ai)P = Unif(0,1), then P({x}) = 0 for x ∈ [0,1]. If 1 = P([0,1]), then Σ P({x}) = 0 <- Dive into this laterExamples of the Above Rules
\nFlip a coin with bias P(Heads with probability p, Tails with probability 1-p).
\nΩ now equals all possible configurations of atoms in the universe. How do we represent this?
\nNote: The probability space is usually implicitly described, except for HW 1.\nLol
\nEx: If A ∈ F, P(A^C) = 1 - P(A). The probability of A’s complement is 1 - probability of A.
Since F is σ algebra, then A^C ∈ F.
\nEx: If A and B are events and A is a subset of B, then P(A) <= P(B)\nB = A U (B \\ A). B is equal to the disjoint union of A and B minus the elements of A\nP(B) = P(A U (B \\ A)) => P(A) + P(B\\A) Using axiom 3\nP(A) + P(B\\A) >= P(A) Using axiom 1
Ex: If A and B are events, then P(A U B) = P(A) + P(B) - P(A ∩ B). Aka inclusion exclusion principle\nProof: AUB, A∩B ∈ F\nA U B = B U (A \\ B)\nP(A U B) = P(B) + P(A\\(A∩B)) Using axiom 3\nP(B) + P(A\\(A∩B)) = P(B) + P(A) - P(A ∩B) Using Axiom 3
Ex: Ω = countable set\nF = 2 ^ Ω, each individual is an event {w} ∈ F, for all w ∈ Ω\nP(A) = Σ for all A ⊆ Ω\nThis is a discrete sample space.
\nEx: Law of total probability. If A1, A2, …, partition Ω, then (Ω = U Ai). Mutually exclusive, collectively exhaustive.\nThen, for any B ∈ F, then P(B) = Σ P(B ∩ Ai)\nB = U(Ai ∩ B) then apply axiom 3
The goal of probability is to take complicated things and use that to approximate them into multiple simpler events.
\nMathematicians: Given a probability space(model), what can I say about outcomes of experiments that derive from this model?\nStatistician: Given outcomes, how do I choose a good model(probability space)?\nEngineers: Given a real world problem, how do I choose a model that captures the essence of the model and then use the model to draw insight.
\nConditional Probability: If B ∈ F and P(B) > 0, then conditional probability P(A|B) = P(A ∩ B) / P(B)
Intuition: Probability A occurs given we know that B occurred
\nFormal Definition: P(.|B) gives a restriction of our model (Ω, F, P) to those samples in B\nIf (B, F|B, P(.|B)) is a probability space itself, where F|B = \\{A ∩ B: A ∈ F\\}
Ex. If A1, A2, … ∈ F, P(Ai) > 0, A partion Ω\nP(B) = Σ P(B ∩ Ai) = Σ P(B|Ai) * P(Ai)
Sometimes, it’s easy to express P(B|A), but we are really interested in P(A|B)\nLet A be the state of the experiment, B be the observation data
\nP(A|B) - given the observation, we want the state. This is the task of inference.
\nSuppose we forgot Bayes Rule, let’s try to recreate it from definition of conditional probability\nP(A|B) = P(B ∩ A) / P(B) = P(A) * P(B|A) / P(B)
Ex. Suppose we have the following model: 85% of students got a pass, 15% of students got a no pass. 60% students that got passes went to lecture, 40% of students did not go to lecture. 10% of students that got no passes went to lecture, 90% didn’t go to lecture.
\nWe want P(NP | N) / P(NP | Y). How much more likely are you to NP when you don’t attend lecture vs when you do attend lecture?\nP(NP ∩ N) / P(NP ∩ Y) * P(Y) / P(N)
\nEx. Suppose we roll two dice and the sum is 10. What is the probability roll 1 was = 4?
\nB = \\{Sum of rolls = 10\\}\nA = \\{first roll is 4\\}\n\nP(A|B) = P(A ∩ B) / P(B)\n = P({First roll is 4, sum of rolls is 10}) / P({sum of rolls is 10})\n = P({first: 4, second: 6}) / P({(4,6), (5,5), (6,4)})\n = (1/36) / (3/36)\n = 1/3Ex. Consider event A1 … An\nP(∩ A) = P(A1 | ∩ni = 2 Ai) P (∩ Ai)
\nEx. Given n people in a room, what is the probability that more than 2 people share a birthday?
\nAi = {person i does not share a birthday with any of the people j = 1, …, i-1}
\nP(Ai | ∩ i-1j=1 Aj) = (365 - (i-1)) / 365
\nThis is because the person needs to land in one of the days not in the 365.\nP(no shared birthdays) = P(∩ Ai)
\nUse 1-x <= e^-x.\nApproximate using taylor series
\nLet n = 23 - there are 23 kids in a class.\nP(shared birthday) >= 1 - e^((i-1)/365))
\nEvents A, B are independent if P(A ∩ B) = P(A) * P(B)\nDisjoint events are not independent\nNote:\\ in a special case where P(A) > 0: A,B: dependent <=> P(B|A) = P(B)
\nIn general, collection A1, A2, … ∈ F are independent events if P(∩i ∈ S Ai) = ∏ i ∈ S P(Ai) for all finite sets of indices S
\nIf A1, A2, … ∈ F are independent, then B1, B2, … are independent, where each Ai = Bi or BiC\nIntuitively, we assume that knowing A means we know A’s complement
\n∩i=2n Ai\n∏P(Ai) = P(∩ Ai) = P(∩) + P(A1)\n(1- P(Ai)) ∏ P(Ai) = P(A1^C ∩ ∏i=2Ai)\nThis shows A1C, A2, … , An are independent\nTherefore, B1, B2, … , Bn are independent
\nOften times we have two events that we think of as independent but aren’t. There might be a confounding variable C\nIf A,B,C are such that P(C) > 0 and P(A∩B|C) = P(A|C) P(B|C), then A,B are said to be conditionally independent given C
\nConsider two coins with bias p!=q\nPick a coin at random, and flip twice. H(i) = Event that flip i is heads\nAre the two coinflips independent? No. If p is 1 and q is 0, then we know what the next coin flip is given our first coin flip.
\nP(Hi) = p + q / 2\nP(H1 ∩ H2) = p2 + q2 / 2 != P(H1) P(H2) = (p+q)2 / 2\nC = {pick coin p} H1, H2 conditionally independent given C
\nTODO:\nSet calendar for homework self grades and resubmission time\nSet midterm time on calendar\nCountable vs uncountable
\nPassword: Teapot
\nDef: A random variable X is a function X: Ω -> R. It implies it is valid to write things like P(X <= 3). P(X <=3) is shorthand for P({ω: X(ω) <= 3})\nRandom conditions have a measurability condition
\nOften times we want to compute more complex probabilities, such as P(α < X < β).\n{ω: X(ω) < β} = Un >=1{ω: X(ω) < β -1/n} ∈ F\n{ω: X(ω) > α} = {ω: X(ω) <= α}C ∈ F\nP(X ∈ B) for pretty much any B(subset of R) that you want\nHere, the technical name for B is Borel sets
\nAnother consequence of the definition of random variables:\nIf X,Y are random variables on (Ω, F, P):
\nFor any random variable X on probability space (Ω, F, P), we can define its distribution(aka its “law”) Lx via Lx(B) := P(X ∈ B), B ∈ R\nFor example, Lx({x}) = P(X=x)
\nThe histogram of values that a random variable would take.
\nSimilar to a probability measure.
\nIn practice, we often describe our model for experimental outcomes in terms of distributions. Given this, I can always construct a probability space and random variable X that has this distribution.
\nEx: Given distribution L, we can consider probability space (R, B, L) and random variable X(ω)=x, ω ∈ Ω = R\nDistribution of X
\nImportant Class of Random Variables;\nDiscrete Random Variables: A random variable that takes countably many values\nEx: X = flip of a p-biased coin. This is a bernoulli distribution with probability p
\nX = roll a dice. This is a uniform distribution over {1,2,3,4,5,6}\nX = number of coin flips until I flip heads. Geometric Distribution\nX = number of heads in n flips. Binomial Distribution (n, p)
\nIn case of a discrete random variable X, the distribution of X can be summarized by its probability mass function.\nPx(x) := P({X=x}) = P{ω: X(ω) = x}
\nBy axioms, px(x) >= 0 and Σpx(x) = 1
\nFor a pair of discrete random variables X,Y defined on the common probability space (Ω, F, P), their joint distribution is:\nsummarized by the join probability mass function defined via\nPxy = P(X=x, Y=y) = P({ω: X(ω) = x && Y(ω) = y}) = P({ω: X(ω)} ∩ {ω: Y(ω) = y})\nWe can obtain the “marginal” distributions by Px(x) = Σ Pxy(x,y) using the law of total probability Σ P(x) = 1
\nDef: Discrete Random Variables X,Y are independent if their probability mass function factors into their marginals\nPxy(x,y) = Px(x)Py(y)
\nThis is equivalent to saying {ω: X(ω) = x}) and {ω: Y(ω) = y}) are independent events for all x,y
\nExamples of Joint Distributions:\nX = {0 if patient tests negative with probability 0.9, 1 if patient tests positive with probability 0.1}\nY = {0 if patient is negative, 1 if patient is positive}
\nWe know the false positive rate of the test is 5%, the false negative rate of test is 1%
\nQuestion: What is PXY?\nI test my patient. The test is either negative(.9) or positive(.1).
\nTakes in a random variable, spits out a distribution\nFor a discrete random variable X on (Ω, F, P), we define its expectation E(X) = ΣxP{X=x} = Σxpx(x)
\nEx: If X Ω = {0,1}n, P({ω}) = p{number of heads} (1-p){number of tails}\nie a sequence of independent p biased coin flips
\nX(ω) = {ω = 9}. <= Number of heads\nE[X] = Σ{i : ωi= 1}, p{i : ωi= 1} (1-p){i : ωi= 1}
\n= Σk=0n (n k) pk (1-p)n-k = np
\nThe most important property of expectation is that it is linear.\nExpectation is an operator, it takes a function and spits out a number, just like an integral.\nJust like how integrals are linear, expectation is also linear.
\nThe most important and only property of expectation E[X+Y] = E[X] + E[Y]
If X,Y random variables defined on a common probability space. E[aX + bY] = aE[X] + bE[Y] <= There is no need for expectation to be independent or discrete.
LemmaL E[g(z)] = Σ g(z) pz(z)
\nBy the law of the unconscious statistician\nE[aX + bY] = Σ(ax + by)Pxy(xy) = aΣxPxy(x,y) + bΣxPxy(x,y)
\nEx: n people put their hats in a bucket, and each draw 1 hat. Let X = number of people who get their own hat back.\nX = Σi=1n1A\nAi = {gets own hat back}\nE[x] = Σi=1 E[1M/sub>]= n * 1/n = 1
\nEx: Coupon collector problem\nHow many on boxes on average do I need to buy before I collect all N coupons\nXi = number of boxes to buy to get ith coupon, after getting (i-1)th\nR[].
\nFor non-negative integer values random value Xi\nE[X] = Σk=1infP{x=k}
\nProof: write P({x>= k}) = Σx>=k P{x >= k} = Σx>=k P(X=x)
\nX1 = Geom(p), X2 = Geom(p)\nM = min{X1, X2}\nP{M>= k} = P{x1}, P{x2}
\nIE[M] = Σ P{x1 >= k} P{x2>=k}\n= 1 / (1- (1-p)*(1-q)))
\nQuantitative Notion of variability of X around E[X]\nVar(X) = E[(X - E[X])2] = E[X2 - 2XE[X] + (E[X])2]= E[X2] - (E[X])2
\nIf expectation is the price of a stock, variance is the volatility.\nVariance is not the only way to measure the variability of a random variable. Another way to measure the variance of a random variable is entropy.\nEntropy = Σ Px(X) log(1/Px(X)) = E[log(1/Px(X))]
\nUnlike expectation, variance is not linear with respect to its arguments. Var(X+X) = ?\nVar(X+Y) != Var(X) + Var(Y)\nVar(X+Y)
\nBy definition of variance\n= E[X+Y - E[X] - E[Y]]2
\nOpen up the square using foil\n= Var(X) + Var(Y) + 2 E[(X-E[X])(Y-E[Y])]
\nPositive covariance means X and Y move in the same direction\nIf X and Y are independent, covariance is 0
\nIf X,Y are uncorrelated(covariance of X,Y = 0), then Var(X+Y) = Var(X) + Var(Y)
\nCorrelation coefficient: Covariance(X, Y) / sqrt(Var(X)*Var(Y)). It is always between -1 and + 1
\nCovariance(X,Y) <= sqrt(Var(X) * Var(Y))\nUsing cauchy schwartz inequality, xTy / (||X|| ||Y||) is between [-1, 1]
\nE[XY] <= E[X2]1/2 E[Y2]1/2\nE[XY] = Σ Pxy(x,y)1/2 x * Pxy(x,y)1/2 y <= (ΣxyPxy(x,y)x2)1/2 (ΣxyPxy(x,y)y2)1/2
\nLet X ~ Binomial(n, p)\nX = Σ Xi XiBernuolli(p)\nVar(X) = Σ Var(Xi) = n Var(X1)
\nPoisson means fish in french. Think number of arrivals when you see poisson\nIf I dip a net into the water and pull it out, a poisson distribution is a good way to checking how much fish I pull out.
\nX ~ Poisson(λ) λ = rate\nPλ)(k) = λk/k! e-λ, k = 0,1,2,…
\nImagine trying to take a bus in berkeley. They are supposed to be at each station every 30 minutes. However, there are times when no buses come for 45 minutes then two buses come in the 46th and 47th minute. You can model the arrival of the buses using a Poisson distribution
\nWhere does Poisson come from?\nEach interval something will arrive with probability pn = λ/n. The probability that something arrives in an interval is Xn ~ Binomial(n, pn)
\nP(Xn=k) = Binomial PMF = (n k) * (λ/n) * (1-λ/n)n-k\n= n(n-1)…(n-k+1) (1/n-λ)k λk/k! (1-λ/n)k\nas n goes to infinity, this distribution converges to a poisson distribution\nλk/k! * e-k
\nPassword: Mirror
\nP(A|C) := P(A ∩ C) / P(C)
When X,Y are discrete: their joint pmf is Pxy
\nPx|y(x|y) := Px|y(x,y) / Py(y) = P({X=x} ∩ {Y=y}) / P({Y=y})
\nSince Px|y is a pmf for each y with P<sub>y</sub>(y) > 0, we can take expectation of X with respect to it.
This is defined as conditional expectation\nE[X|Y=y] := Σx⊆X xPx|y\nOnce we have the value of Y, what is the value of x?\nWe usually just write E[X|Y] to denote Σx⊆X xPx|y evaluated at random variable Y. Thus, the conditional expectation is itself a random variable since it is a function of the random variable Y.
\nMost important property of Conditional expectation:
\nFor all functions f, we can say E[f(Y)X] = E[f(Y)*E[X|Y]]
Example: Iterated Expectation\nE[X] = E[E[X|Y]]\nWe get this using the tower property by setting f(Y) = 1
\nExample: Let N>= 0 be an integer valued random variable. Let’s flip a fair coin N times, and let X = the number of heads.\nThere are two sources of randomness, the number of flips and whether we get heads or tails.\nE[X] = E[E[X|N]]\nWe can fix the number of coin tosses, and since half of coin tosses are heads\nE[X|N] = N / 2\nSubstitute\nE[N/2]\nUse linearity of expectation\n1/2 E[N]
\nVar(X) = E[Var(X|N)] + Var(E[X|N])\nUsing the same substitution as above of E[X|N] = 1/2\nUse binomial theorem for E[Var(X|N)] = E[N/4]. Variance of a binomial = ?\nVar(X) = E[N/4] + Var(N/2)\nVar(X) = 1/4 * E[N} + 1/2 * Var(N)
\nRecall Var(X) = E[X-E[X]2]\nDefinition: Conditional Variance of X given {Y=y}\nVar(X|Y=y) = Σpx|y(x|y)(X-E[X|Y=y])2
\nJust like conditional expectation, we write Var(X|Y) to denote the random variable evaluated at Y.
\nTheorem: Law of total variance
\nVar(X) = E[Var(X|Y)] + Var(E[X|Y])
\nProof:
\nVar(X) = E[X2] - (E[X])2
\n= E[E[X2|Y]] - (E[E[X|Y]])2
\n= E[Var(X|Y)] + E[E[X|Y]2] - (E[E[X|Y]])2
\n= E[Var(X|Y)] + Var(E[X|Y])
\nNow let’s start with Var(X|Y=y)
\n= E[X2|Y=y] - (E[X|Y=y])2
\n= E[Var(X|Y) + E(X|Y)2] - (E[E[X|Y]])2
\nNote: Random Variables need not be discrete or continuous or combinations thereof\nFor a random variable X, we can always define its cumulative distribution function abbreviated CDF via
\nFx(X) := P{X<=x}
\nDefinition: A random variable X has continuous distribution if there exists a function fx such that
\nFx(x) = ∫-infxfx(u)du
\nfx is called the density of X aka a pdf(probability density function)
\nFor fx to be a density, it just needs to be non-negative and the integral of the density to be equal to 1, since total probability = 1\nfx >= 0\n∫fxdx = 1
\nGood for modeling analog signals from the real world.
\nContinuous distributions usually described by their density.\nX ~ Uniform(a,b) = {1/(b-a) a <= x <=b, 0 otherwise}\nX ~ Exp(λ) = {λe-λx if x>=0, 0 otherwise}\nX ~ N(µ,σ2) = 1/sqrt(2pi*σ)exp(-(x-µ)2 / 2σ2)
\nWe say X1, X2, …, Xn are jointly continuous if\nFX1X2…Xn(x1x2…xn) = P{X1<=x1, X2<=x2, … Xn<=xn}
\n= ∫∫ FX1X2…Xn(x1x2…xn)du1du2…dun
\nExample:\nLet a dart land uniformly at random on a 2d dartboard of radius r.\nThe joint density will be flat over the dartboard
\n{1/pi*r2, x2 + y2}\nIndependence: Random Variables X,Y are independent if Fxy(x,y) = Fx(x) * Fy(x)\nIf X,Y are continuous and independent, their joint density is the product of the marginals
\nSame as discrete case, replace sums with integrals\nE[X] = ∫x fx(x)dx
\nMore generally:\nE[g(X1…Xn)] = ∫…∫g(x1…xn) fx1 … xn(x1 … xn)dx1 … dxn
\nExample:\nVar(X) = ∫(X-E[X])2 fx(x)dx
\nExample: X ~ Unif(a,b)\nE[X] = ∫ uniform density function
\n= ∫ x * 1 / (b-a) dx
\n= 1/2 * (b2 - a2) / (b-a)
\n= 1/2 * (b+a)
\nExample: Var(X) = E[X2] - (E[X])2\n= 1/2 (b-a)2
\nLet r = sqrt(x2 + y2)\nCompute the probability that the dart is in the middle half of the dartboard.
\nP{R<= r/2} = P{x2 + y2 <= r2 / 4}
\n= E[{x2 + y2 <= r2 / 4}]
\n= 1/(pi * r2) ∫ {x2 + y2 <= r2 / 4} dx dy
\nDiscrete: Expectations come from PMF and sums\nContinuous: Expectations come from PDF and integrals
\nUniform(a,b)\nExp(N)\nN(µ, σ2)
\nSuppose we want to model a memoryless process. For example, as a switch sending packets, it doesn’t matter how long you wait for a packet to show up, the expected wait time for the next packet doesn’t change.
\nMathematically: Let X be a non-negative random variable with the memoryless property.\nMemoryless = P{X>t+s|X>s} = P{X>t} for all t,s > 0\nIf I’ve waited for s seconds, the probability P{X>t} is still unchanged
\nP({X>t+s} ∩ {x > s}) = P({x > t}) P({x > s})\nGet rid of {x > s} since if x > t + s, x must be greater than s
\nF(t + s) = F(t) F(s)\nThe only unique solution to the functional equality that is a CDF(must be non-negative, monotone increasing) of the form Fx = e-λx
\nIf x has memoryless property, F(X) = 1 - e-λt for some λ > 0
\nThe sum of independent effects. Height of NBA players, satisfaction of jobs, sum of voltage across resistor.
\nWe call N(0,1), the standard normal distribution.
\nCDF: 1/2pi ∫exp(-µ/2)2du
\nP(X<= x) = P((x-µ)/σ <= (x-µ)/σ) = Φ((x-µ)/σ)\nGaussians have many nice properties
\nIf X is gaussian, then so is aX + b.\nIf X,Y are independent gaussians, then X+Y is gaussian
\nExample: Let V ~ N(1, 52) be input voltage to chip, averaged over 1 second.\nOur chip fails if voltage dips below 0.5 volts or exceeds 2.5 volts for any one second period
\nProbabilty(chip fails in 60 second duration) <= 60 * Probability(chip fails in 1 second)\n60 * Probability(chip fails in 1 second) =\n= 60 * Probability(V < 0.5) + Probability(V > 2.5)\n= Φ((0.5 - 1)/σ) + (1 - Φ((2.5-1)/σ))
\nExample: cellphone sends a bit B ∈ {-1, 1} to tower. Tower receives Y = B + N, N ~ N(0,1). Tower makes decisions B(Y) = sign(Y)\nP(err | B = 1)
\nGiven that I receive {Y=y} what is the probability that B = 1\nP({B = 1}) = P(Y ∈ [y, y + 𝛿] | B = 1) P(1)
\n= PB|Y(1 | y) = PB(1) * fY|B(y|1) / fy(y)\nThis is bayes rule for one
\n= (1/sqrt(2pi) * exp(-(y-1)2/2)) / (1/2 * 1/sqrt(2pi) * exp(-(y+1)2)/2) + …\n= 1 / (1 + e-2y)
\nThis is b\nExample:\nLast example motivates the definition of conditional density\nIf X,Y are jointly continuous, we define conditional density of X given Y:
\nfx|y(x|y) = fxy(x,y) / fy(y)\ninterpretation: is the density of X given I know {Y=y}
\nBayes rule\nfx|y(x|y) = fx(x) / fy(y) * fy|x(y|x)
\nExample:\nFor X,Y that are jointly continuous, we have\nE[X|Y=y] = ∫ x fx|y(x|y)dx\nCE still satisfies tower property: E[g(y|x)] = E[g(y)E[X|Y]]
\nDistribution of X give by CDF Fx. Suppose we define Y = g(x) for some function g
What is the distribution of Y?
\nExample:\nX continuous random variable, Y = aX + b\nFy(y) = P{aX <= y-b}\n= P{x <= (y-b) / a if a > 0, x >= (y-b)/a if a < 0}\n= {Fx((y-b)/a) if a > 0, 1 - Fx((y-b)/a) if a < 0}
\nfy(y) = {1/a * Fx((y-b)/a) if a > 0, -1/a * Fx((y-b)/a) if a > 0}\n= 1/|a| * fx((y-b)/a)
\nY = AX + b\nfy(y) = 1/|det(A)|fx(A-1(y-b))
\nWeak law of large numbers: Everyone in the room takes a coin and flips it 500 times. Then we can saw the empirical frequency is between .49 and .51. Then everyone flips it 500 more times. Then the empirical frequency is between .495 and .505. However, someone might have been between .49 and .51 in the first 500 but not .495 ad .505 in the first 1000\nStrong law of large numbers: The empirical frequency will go to its true probability with probability 1.
\nCLT is a statement about convergence in a distribution
\nIf I observe (x1, …, xn), then I describe it via bitstring (1, xxx…x) -> log(At/2)\nIf I observe a sequence not in the typical set, I describe it brute force via bitstring
\nWhat is the performance(average number of bits needed per symbol observed) for this scheme?\n1/n * E[number of bits in representation] ≤ 1/n (2 + n(H(x)+E))P(x1…xn) + 1/n (2 + nlog(x)) * P(x1…xn) ≤ H(x) + E/2 + 4/n + log|x| * probability of sequence not being in typical set
\nQuestion: How are we going to show we can compress down to the entropy?\nAnswer: Use concentration. For a sequence(X1, x2, …, xn), let the probability of observing it be p(x1…xn) = Πi=1n Px(xi)
\nTheorem: Asymptotic Equipartition Property - If Xs are iid with respect to Px, then -1/n log p(x1…xn) converges to H(x) in probability
\nTypical Set: For E > 0, for each n ≤ 1 define typical set\nA := { (x1…xn) : p(x1…xn) >= 2-n(H(x) + E)}
\nWhy is this called a typical set? It is a set of typical sequences.
\nQuestion: Suppose I have N objects. What is the max number of bits I need to represent each object?\nAnswer: max of log(N) bits
\nMarkov models: Map events that have memory\nPoisson Processes: Random arrivals: customers coming into a store, beta particles detected by geiger counter, vehicles passing a tollbooth\nCounting Processs: Starts at 0, right continuous, integer valued\nPoison Process: a counting process with Exp(λ) interarrival times
\nThm: If Nt is a Poisson process with rate λ, then Nt ~ Poisson(λt)\nP{Nt = n} = e-λt * (λt)^n / n!
\nProof:\nP{Nt = n} = P{Tn <= t < Tn+1}\n= E[1{Tn} <= t 1t < Tn+1]
\nIncrements of Poisson process are stationary: Nt+s - Ns = Nt + tau - Ntau
\nPoisson processes have independent increments:\nif t0 < t1 < … < tk
\n=> increments Nt1 - Nt0, Nt2 - Nt1, … , Nt1 - Nt0 are independent
Thm: If Nt is a counting process with independent stationary increments and Nt ~ Poisson(λt), then Nt ~ Poisson Process(λ)
\nConditional distribution of Arrivals\nThm: Conditioned on {Nt = n}, (T1, T2, …, Tn) = (Unif(1), … Unif(n))
\nExample: Let cars pass through a tollbooth according to Poisson Process(λ).\nQuestion: What is the probability that no cars pass in 2 minutes\nP{N2 = 0} = (λ2)^0 * e^(-2λ) / 0! = e^(-2λ)
\nQuestion: If 10 vehicles pass in 2 minutes, what is the distribution of vehicles that passed in the first 30 seconds?\nBinomial(10, 1/4)
\nMerging: If N1,t is poisson process with rate λ and N2,t is an independent poisson process with rate u, then N1,t + N2,t ~ PP(λ+u)
\nSplitting: Independently mask arrival with 1.\nExample: Let vehicles pass through a tollbooth with PP(λ). Let 1/3 of the vehicles be cards and 2/3s be trucks
\nIf 10 vehicles pass in the first 2 minutes, what is the distribution of cars that pass in the first 30 seconds?
\nG ~ G(n,p): G is a graph with n verticies where each edge appears indepedently with probability p.
\nMonotone Graph Properties have sharp thresholds: P {G ~ G(n,p) has property P} = {0 if p < threshold n , 1 if p > threshold n}\nThere is some phase transition phenomena.
\nEx: p = {graph G has >= 1 edge} Monotone property.\nClaim: t(n) = 1/n^2\nProof: Let x = # of edges in G, p = c / n^2\nWhat is the probability G has zero edges?\nP{G has zero edges} = P{X=0} = (1-p)^(n C 2) = e^(-c/2)\nThis evaluates to basically 1 if c << 1 and 0 if c >> 1
\nGoal: For property p={graph G is connected}, show that t(n) = (log(n)/n)\nIf lambda > 1, then P{G~G(n,p) is connected} -> 1 as n->infinity\nIf lambda < 1, then P{G~G(n,p) is connected} -> 0 as n->infinity
\nLemma: If X is a random variable, then P(X=0) <= Var(x)/(E[X])^2\nProof:\nVar(x) = E[X-E[X]]^2\n= P(X=0) * E[X-(E[X])^2 | X=0] + P(X!=0) * E[(X-E[X])^2 | X=0]\n= P(X=0) * E[X]^2 + P(X!=0) * E[(X-E[X])^2 | X=0]
\nCase lambda < 1:\nWill show: P{G has isolated vertex} -> 1\n{G has isolated vertex} C {G is disconnected}\nX = Σ Ii = # of isolated verticies, where Ii = {0 if i not isolated, 1 if i isolated}\nIi ~ Bern(q) where qi := (1-p)^(n-1)\nVar(X) = Σ Var(Ii) + Σ Covar(Ii, Ij)\n= nq(1-q) + n(n-1)Cov(I1, I_2)
\nWhere Cov(I1, I2) = E[I1I2] - (E[I])^2\nP{G has no isolated vertices} = P{X=0} <= (nq(1-q) + n(n-1) * pnq^2/(1-p))/(n^2q^2)\n<= (1-q)/nq + p/(1-p)\n(1-q)/nq goes to zero since nq -> infinity\np/(1-p) goes to zero as p -> 0
\nP{G disconnected} = P{U_k=1^{n/2} {There exists a set of k vertices separated from the rest of G}}\nApply union bound again\nP{(n C k) P {vertices 1, …, }}
\nGiven data, how do I choose the model that generated the data\nX is the state of nature, often parameter or hypothesis -> P_{Y|X} -> Y produces some observation(data)\nX may or may not be the random variable\nIf it is, then the distribution of X is called a prior(Bayesian Inference)
\nGoal: Infer X from Y\nX -> Model -> Y
\nReasonable Approaches:
\nFor any test X:Y, there are two fundamental error rates, false positives and false negatives
\nFalse Positive: P{X(Y) = 1 | X = 0}
\nFalse Negative: P{X(Y) = 0 | X = 1}
\nQuestion: Given a constraint on type 1 error rate, how do we find the test that minimizes type II error rate?\nSolution:\nSolve X = argmin P{X(Y) = 0| X=1}\nAnswer: Randomized threshold tests are optimal: Neyman-Pearson Theorem\nGiven Beta, the optimal decision rule is X(Y) = 1 if L(Y) > lambda, 0 if L(Y) < lambda, Bernoulli(gamma) if L(Y) = lambda\nWhere lambda and gamma are chosen such that P{X(Y) = 1 | X=0} = beta
\nB = P(L(Y) >= lambda | X=0)\n= P(Y/sigma >= 1/(2sigma) + philog(lambda|X=0))\n= 1 - phi(1/(2sigma) + sigmalog(lambda))
\nQuestion: Where does randomization enter the picture\nAnswer: Deterministic threshold tests define and lie on error curve, but don’t necessarily sweep the whole thing\nEx: If Y is discrete, say binary-valued, then L(Y) takes at most 2 values\nTakes two tests and randomizes between them
\nThreshold tests are optimal
\nEstimation under mean squared error\nGoal: Estimate X based on Y under some loss function\nX -> Model -> Y
","fields":{"slug":"/posts/academics/probability","tagSlugs":["/tag/list/","/tag/uc-berkeley/"]},"frontmatter":{"date":"2023-01-10T23:46:37.121Z","description":"Real applications of probability","tags":["List","UC Berkeley"],"title":"Probability in Electrical Engineering and Computer Science"}}},"pageContext":{"slug":"/posts/academics/probability"}},"staticQueryHashes":["251939775","401334301","825871152"]}