Albert Su

Showerthoughts

Fri, 21 Nov 2025 23:46:37 GMT

Useful stuff

Get people to contribute to an idea and they will be more likely to accept it. Studies show that it doesn’t take that much involvement in the creative process to get people to take possession of a proposed plan.
Easy way to get people to like you Them: Tell me about yourself? You: I did blah blah blah…, what about you? Them: Blah Blah Blah You: Wow, that’s so hard!
Why not faster? Why can’t we build this startup/finish this homework/learn this skill faster?
If you really want to connect with someone, take them for a long, scenic walk. Not being face to face takes some of the pressure off, and the scenery puts you in the right mood to open up.
If you haven’t used your voice in a while (sleeping, lonely, etc) and suddenly need to take a phone call, hum for a few seconds prior. Your vocal cords won’t let you down.
If you want to offer a deep and memorable compliment, ask someone how they did something. It gives them the opportunity to tell their story, and shows your genuine interest.
When complementing people, always compliment them on something they have earned, not something they have.
When you’re leading a conversation, it’s very good to invite other people who might be more quiet by asking them what they think or joking with them.
Fully commit to friendly gestures, such as high fives, back pats, shoulder touches. Fully committing matters as it shows more confidence but also helps with being more trustworthy.

Not Useful stuff

Several times in life I’ve cold called a company to confirm my interview time. I didn’t have one prior to my call, but in their confusion and inability to even find my resume I’ve managed to secure an interview about four out of five times. Twice I’ve gotten the job.
When smoking meats in a barbecue, instead of using wood smoke, use a juul for extra flavor.
For group interviews, have the candidate play a level of overcooked to see how well they work as a team and respond to stress
Things that are complete are completed, things that are incomplete are incomplete
Denero’s first lecture(Intro to Computer Science) was very impressive because he did some playing around with numbers while teaching us how to use + - * and /. He kept making small examples ie 4+3 is 7 to show what the operators where, something that we naturally know how to use. However, he later created another expression 1+2*3+4/5… which ended up evaluating to 2019

Longer stuff

Time Management:
- My advice might be different than what you are expecting but try to think of your problem from a different angle. Let me explain: you’re currently seeing your problem as how to optimize your internship search but you’ve implicitly assumed that you’re going to dedicate your time primarily to school and secondarily to finding an internship.
- I would suggest freeing up 100% of your time as a thought experiment and then only take actions that contribute to your goal. If your goal is grad school, you might need to keep your GPA up a bit higher, and that means you can’t do as many interesting projects and look for internships. However, if your goal is a job after graduation then think of your resume as your primary goal, and if your classes align with that, spend more time engaging with those classes. If they don’t, then miss an assignment or just do the main parts and sacrifice the points.
- For me, once I saw school as secondary and my projects and portfolio as primary, I found I was more interested in my coursework because I saw it as something that would help me solve my real problems, and I absolutely blew away the projects I was assigned because I didn’t just care to fulfill the requirements, but also to present the projects as impressive and relevant to industry on my website and github.
- Furthermore, I found that this method keeps me much more interested, which is a more correlated with success and retention, and so I became much more productive, and was able to still keep up my other extra curricular commitments. YMMV.
On meaning: One always finds one’s burden again.
- Camus uses Sisyphus as a metaphor to describe our absurd existence. Like Sisyphus, we are in a constant cycle of striving for purpose and discovering that it is all devoid of meaning. In Sisyphus’ case, the gods condemned him to this task with the idea that all his efforts will be futile, the punishment being that he puts forward a monumental effort for nothing. However, it is not up to the gods to determine what gives Sisyphus purpose. Sisyphus can find meaning in this task if he so chooses, and by finding meaning in this task, he revolts against the gods who want him to suffer. Likewise, we are faced with the Absurd; a universe seemingly without objective meaning. By responding to this with our own subjective meaning, we revolt against the Absurd.
- “The struggle towards the heights is enough to fill a man’s heart. One must imagine Sisyphus happy.”
- I don’t know if I necessarily believe that Sisyphus has to be HAPPY to defy the gods and find purpose, but I think the idea Camus is trying to convey is that nothing/nobody gets to decide what your purpose is. And even though the universe is without meaning, that doesn’t mean that we have to live meaningless lives.
- The way I interpret the quote in the picture is: at the beginning of every day, we are all too aware of the burdens we will face, and the struggle we are about to undergo. No matter what, we will always strive for something bigger than ourselves. As we accomplish one thing, another goal replaces it, and we are constantly struggling to be complete. By recognizing how difficult life is, and how we are always incomplete in some sense, we can live in rebellion to the negative feelings that keep us from moving forward. We voluntarily take on the burden of life, knowing that burden will never fully be lifted. Why? Camus sees that struggle as purpose. And whatever reason you subjectively have
- “I leave Sisyphus at the foot of the mountain. One always finds one’s burden again.”

Personal Project Ideas

Mon, 10 Nov 2025 23:46:37 GMT

Completion Time: 10 years

The social dilemma: No financial incentive for companies to keep gathering data. Make a tax on data or a way to track how much data companies gather

Completion Time: 1 year

ShopLYFT - Like LYFT but for groceries except instead of paying for the groceries, you just walk out of the store with them(all orders must be $900 or less)
List of source code for ethereum projects that allows you to query for specific exploits, ie keccak(msg.data) not being the same in every transaction
Cal clone club - clone the startups/large companies and perform the same service without making any profits
Blockchain that keeps track of pollution, forces people to pay on their agreements, using IOT devices that track pollution.
Use crypto economics like proof of work/stake to make a review system that cannot be bought like yelp, amazon reviews, rotten tomatoes
U2f security key for in person use ie yubikey for houses
Delivery drivers in doordash need to wait to drop off package. Use public key private key cryptography to let them open a box to drop package in
Dao owning football/sports team

Completion Time: 1 month

You should build a smart contract that allows illegal immigrants to pay the people that smuggle them across the border only when they actually cross the border
Create a discord server where you can ask questions and get paid in crypto using discord commands for answering technical questions
Quiz pop up questions as google chrome extension for youtube lecture videos
Box next to soccer field or basketball court that tracks the number of people there to predict when to go to pickup at optimal time
Optimal size for bug bounty research
Short Quizzes at the end of articles to test knowledge/for fun
Take series 66 exam for fun lol
Create match highlights(soccer, basketball) based on number of comments in live chat

Completion Time: 1 week

Write up an article on people’s ama, for example this one: https://www.teamblind.com/post/Started-in-VC-right-out-of-undergrad—AMA-4pN5cL2r
Use student loan pricing data from places like SoFi to create school rankings
Scaling Gracepoint, article on how they did it
Pretend to be a student to ask recruiters to rank most prestigious company, most prestigious school as a part of a survey
Article on zkproofs https://www.youtube.com/watch?v=fOGdb1CTu5c

Reading List

Mon, 10 Nov 2025 23:46:37 GMT

To Read

The green book - gaddafi Network State New Lombard Street Lombard Street End of Finance Global History - Nature of Money https://www.seriouseats.com/the-food-lab-answers-questions-about-food-science Alchemy of Finance Outsiders: Eight Unconventional CEOs Principles - Ray Dalio More Money than God

Read

Never Split the difference All you need for college Admissions - https://nikhilsardana.github.io/guide.pdf

Kinesis Labs

Mon, 20 Jan 2025 00:00:00 GMT

This one really sucks. It was a series of unfortunate mistakes that led to a multinational company repeatedly issuing DMCA takedown notices on our website.

Timeline of Events: 1. We create Kinesis Labs, a stableswap on Evmos. 2. I personally didn’t know it at the time, but Kinesis is based off a character from Maplestory - named Kinesis. Maplestory is owned by Nexon Corporation 3. When we published the site, we bought the domain kinesislabs.co(this is fine, there are tons of products out there named Kinesis). The problem is that we used a picture off google images of the Maplestory Kinesis. 4. This image was only a temporary placeholder while we commissioned art(for 3k USD -_-) from our artist. What they drew is pretty distinctive from the Maplestory Kinesis. 5. At the same time, we received two messages from someone in Nexon Corportaion in telegram and twitter asking us if we knowingly infriged copyright laws. Obviously, we ignore these messages, as we get tons of similar spam every day. a. Side Note: I feel like even if we did respond to these messages, it wouldn’t have done much good since it seems like the reporter already filed a complaint to Nexon and was trying to get a confession out of us. He wasn’t trying to help us make sure our website was compliant. b. Side Side Note: We looked at this guy’s twitter profile. From what we’ve gathered, this is lieterally your average defi user who happened to follow us because they were interested in our project. They then decided to report our website to their employer/ 6. We ignore the message again, oops, and we receive notice of a DMCA on Netlify. If we don’t respond, they will take down our site in 7 days. 7. The notice was extremely incomplete of course, the pictures they linked were no longer on our site, so we draft a reply. We also asked our friends for advice, most said we were fine, and one of our friends actually put us in contact with a law firm. We didn’t immediately engage with the firm because it was expensive and we did not raise any money, so all legal fees would come from our own pockets. 8. We draft a reply, under my name, telling them the links they sent were invalid and no longer on our side. For some legal reason, drafting a reply opens you personally up to legal liability. Of course, this does nothing - apparently we filed the response in an incorrect format to AWS. 9. We migrate to Vercel in the meantime. We then get another DMCA on vercel - even though they linked the webpage from Netlify. 10. We draft up a response to vercel, the problem right now isn’t just the legal issues but that fact that these hosting websites take own our site immediately upon receiving a DMCA notice, not after the dispute has been disputed.

Things to do when...

Sun, 10 Nov 2024 23:46:37 GMT

What are my strengths? - No forethought and no when to quit, but has drive to put something out into the world What do I need to improve on? - People skills could be improved - Very profit driven What do I want in a job? - I want to work like a dog. I want to work at a place where people are encouraged to sleep under their desks and pull all-nighters. I’m not sure why I’m like this, but I’ve noticed that I can’t sit too still and do nothing, I’m always building side projects(Kinesis, Stardust, Klade) or working for Risk Harbor/Yoz/0xFoobar during school and my internships - I feel like something is missing if I don’t have a project I can throw myself in. Maybe things will change, but I would have to wait and see and I think it would be more fun this way in the first place. (want to prove something to myself) - Somewhere with high upside and is has a long term vision - I want to work with smart, brutally honest, and quick learners. I want the people I work with the call me out when I make mistakes. - In person About Me - Basic stats + story of how you grew up ○ In my early life up to the end of middle school, I honestly don’t remember doing much besides going to school, going to a Chinese afterschool, and playing a bunch of different sports(cross-country, track, basketball, soccer). I also went to church and did a bunch of Boy Scouts related stuff which is honestly pretty underrated - I don’t think there are any other organizations that let a 14 year old plan a 12 day trip to the middle of the New Mexico Desert whilst having to worry about transportation, airfare, gear, and keeping everyone alive lol. I also played this one video game called Realm of the Mad God, where you go around and kill enemies with other players. This lowkey taught me a decent bit about economics - supply and demand, liquidity, stores of value, how to scam people, etc, ○ During high school, I was lucky enough that my friends in middle school knew a bit more of how one is “supposed to” play high school, so I did clubs like DECA and FBLA at their recommendations. I actually made a few good friends in FBLA and DECA, mostly because I went super try hard. Lynbrook was really good at this event called Parliamentary Procedure, which was essentially a set of rules that maximize the efficiency of organizational meetings, most commonly used by stuff like the English Parliament and the US Congress. Unfortunately, this meant memorizing this really old book called Roberts Rules of Order which was 1100+ pages. We also memorized the exact wording for each of our roleplays which was doubly painful since there were probably a few hundred prompts and a few hundred different responses to these prompts. Luckily, I’m still pretty good friends with one of the people I did Parliamentary Procedure with so I guess it was worth it in the end. It also showed me that you can control the outcome of seemingly random “businessy” competitions where you are judged on more subjective indicators. Before, I thought judging for business case competitions was super random and arbitrary when in fact you could break down what the judges wanted to see, even when the judges themselves didn’t know exactly what they wanted to see. (We also won 1st in the section, 2nd in the state, and 3rd at Nationals lol using tricks like these) Unfortunately, I didn’t figure out DECA until senior year, but you can just read the DECA tips section if you want to win at DECA. I honestly don’t have many key people in my life that I can recall influencing me directly, rather, it’s probably been people influencing me indirectly via osmosis. I’m guessing that my academic success was at least partially influenced by the people around me, my parents had some influence but I think it was mostly my peers that drove me to do the things that one needs to do to become “successful”. For example, I only learned about SAT prep from my peers, not my parents. I needed to find my own free SAT classes online and practice tests which I would probably not have done without my friends pressuring me by comparing my SAT scores to theirs. When my friends started taking AP Classes, I felt like I needed to keep up and ended up taking 10 APs over 2 years, self-studying most of them. I’m super privileged - grew up in the Bay Area at a good to elite high school without having to worry about anything but school and my interests - yet my parents never pushed me to do better in school(only had a 3.8 uw gpa) or to do cookie cutter extracurriculars(they were fine with me quitting piano and band) - Past professional life ○ I think doing computer science would have been much less fun without my peers. We managed to gamify the worst part of computer science, the job and internship hunt. Additionally, classes are hard but being able to bond through shared trauma is another huge benefit of the Berkeley computer science curriculum. ○ I think one interesting thing is how I ended up working in crypto. As a freshman, I tried to do basically everything. I tried Berkeley clubs(got rejected from all of them), worked in a chemistry research lab to see if things other than computer science were worth doing, did research through the haas neuroeconomics labs to see what applied computer science was like, and took as many classes as I could. However, nothing really interested me, since the cool things one could do in each of these programs were limited by my experience. Thus, I needed to find a field where one could make major contributions without having any experience and a nascent field was best for this. ○ I quit my haas neuroeconomic lab with one of my friends that also worked there and started building stuff for crypto. I worked on that project for a bit over a year and eventually went and joined a crypto startup that I found on discord after the project died. Afterward, I ghosted that startup to work on my own project again. I just find it super interesting that all of these events couldn’t really have been predicted from one timestep back. In high school, there was no way I could have predicted that I would be working in a research lab at a business school. While working at the research lab, I had no idea my next step would be to work on something in crypto. And while working on our project, I had no idea I would leave that to become an employee of another crypto startup. ○ In the meantime, I also worked one summer at the NSA. Originally, I wanted to work there since I wanted to know a few random things like what crypto systems does the government know how to break and what parts of the US government are controlled by who but at the end of the day I just thought it was really cool and that it was an opportunity I would never have again. I don’t think I learned too much technically, it was more high level architecture, what is possible and not possible, and a lot of systems reliability engineering principles(read the google SRE book if you want to know what I did during this internship). Another thing I took away was the ability to move to a new city and establish routines, find friends, and generally survive. I met some really cool people at my apartment, such as a Harvard lawyer who works at the white house(ex. SEC) and a basketball trainer who used to train Jalen Smith(Indiana Pacers) ○ The summer afterwards, I did an internship at Coinbase which looking back, I shouldn’t have done. It was mostly grunt solidity, typescript, and Go work that wasn’t very fun and everyone there was super depressed since 18% of the company was laid off on the second day of work and coinbase wasn’t doing anything cool that I thought it would. However, this gave me time to go to Seattle for a month and meet some pretty cool people like the Eigenlayer team and Thomas. ○ Finally, I did a lot of stuff in crypto: see below for more info. - Any life goals? ○ Still figuring this out - Do you do anything for fun? ○ Play basketball/soccer, sports where there is some level of teamwork involved and that forces you to immerse yourself in that game. ○ Some other random stuff I’ve tried recently that I’ve enjoyed: § Writing - both technical content as well as reflections like this. Weird since I hated writing in middle and high school § Climbing - both outdoor and indoor are really fun, but I need someone with experience to take me outdoors and indoors is too expensive/too much of a commitment to buy a pass § Skiing/Snowboarding - incredibly fun and expensive sport, however, I kinda get bored skiing/snowboarding since after a while even the double blacks don’t provide the type of challenge and fear I crave. Maybe teaching others is a good alternative that might make things more fun? § Chess - mostly as a crutch for short term entertainment instead of playing more traditional video games § Going to Hackathons - I feel like suffering together for long periods of time is the quickest way to make friends, and as such, hackathons are a perfect place to do so. Also, usually prizes are pretty decent for the amount of work you put in and I think I’ve figured out a way to win the game of hackathons. § Hiking/Outdoors stuff - Vanilla hiking is kinda boring, unless the scenery is crazy(unlikely since I’ve seen most types of views). One thing I realize I enjoy while hiking is peer pressuring my friends to go swimming in the nearest lake/river on the hike, it makes things seem more visceral and real. § Finding Free Food - Hillel, VC dinners, club events, finding free food not only gives you free food, but also an excuse to do whatever the free food providers are doing, which is usually something new. § Traveling - But am I just traveling since I’m running away from my problems or is there something I’m chasing after? Traveling is lowkey pretty fun though, and I feel like no matter where I go I know someone there or I can make friends pretty quickly if I’m by myself which makes traveling so much more fun. Also, ever since I stopped grinding at life, I feel like traveling is 10x more fun since there isn’t really work stuff I’m always thinking about - When were you most happy? ○ Doing dumb stuff with friends, such as going to a random barber in Mexico and telling them to cut it however they thought looked good. ○ Also, it sounds weird but sometimes I’ll really enjoy coding. I vividly remember being really happy while working on the Penn blockchain hackathon project - Who do you respect? If no one, are there parts of anyone you respect? ○ I respect anyone who has ever built anything. Just the willingness to build something and bringing it into the world is something I wish more people would do ○ Also, I really respect anyone that went through difficulties in their childhood to get to where they are today

Crypto things I’ve done in the past From most recent to least recent

Stardust Built a p2p lending platform for ETH San Francisco. Honestly, this idea has been floating around in me and a friend’s head - here’s a brief description:

Our initial focus is semi-locked tokens - strong liquidity for the underlying, despite these derivative tokens difficult to list within a money market. This represents assets like LSDs (rpETH, stETH, etc), timelocked LPs (Chicken Bonds) and, veTokens. We know what happens when things go right – Aave listing stETH and the corresponding looped leveraged resulted in billions in TVL for both Aave and Lido. We also knows what goes wrong with these derivative tokens. Manipulable oracles, in addition to low underlying dex liquidity have meant that it can be difficult to use these derivatives in a money market context. This is why we’re taking a different approach to lending for these assets, a physically settled P2P lending market for these assets. We built it out and got interest from DAOs that wanted to get cashflows from their holdings, like Alchemix, Redacted Cartel, and M11 Credit as well as made it to the final round of the AllianceDAO and Geometry accelerators. However, I then got cold feet from doing this startup for two reasons: firstly, I wasn’t sure if this is the problem I wanted to work on for the next 3-5 years and secondly, I wasn’t sure if I was good enough to found a startup. Because of this, I decided to stop working on Stardust and stepped back from crypto altogether. Website: https://stard.st/

Kinesis Labs I wanted to get some experience running a DeFi project, so ran this saddle finance fork on Evmos and added some features like dynamic fees! Got to ~600k tvl and learned a bunch about how DeFi really works behind the scenes. Made enough mon ey from this and Risk Harbor to give myself some time to sit and think about what to do next which is really helpful.

Website: https://kinesislabs.co/

Evmos Intern I wanted to help out some of my friends(ex-blockchain at berkeley) with Evmos, and created the Evmos Intern account to grow DeFi on Evmos. I also wanted to see if growing a project via constructive and positive tweets would be different from throwing fud and technical explanations on twitter. Here are some sample tweets: https://twitter.com/evmosintern/status/1548737661950521344 https://twitter.com/evmosintern/status/1548737976561045504

ParadigmEng420 I wanted to grow an anon twitter account, just to see if I could. I got the perfect opportunity when the Nomad bridge got hacked. I was pretty early to the hack, as only ~40 minutes had gone by since the initial hack transactions. I then took a few hours to look into how the hack worked and published a tweet thread explaining the technicals, frontrunning samczsun’s inevitable thread. I went from 0-1k followers from that thread. https://twitter.com/ParadigmEng420/status/1554249610574450688

The second big thread was the aptos fud thread. Since aptos mainnet had just been announced and most of crypto twitter hated aptos, it was the perfect time to publish a hit piece to gain traction off that sentiment. I threw together some lies and half truths about the aptos tps and the token distribution and went from 1k-5k followers off that tweet. https://twitter.com/ParadigmEng420/status/1582121408548651009

Wormhole explorer Wormhole has a rate limit, where only a certain amount of value can be sent cross-chain in any given day. This was a CalHacks project that tracked past on-chain transactions on Wormhole, but also off-chain transactions in their gossip network to make sure any transactions sent wouldn’t be rejected by their guardians. I worked on this with 3 other members of blockchain at berkeley.

Jump Crypto liked the project enough such that they ended up paying blockchain at berkeley to work on two continuations of this project for 15k each which was pretty cool.

Muteswap I worked on this project with 0xfoobar and his team, mostly along the lines of research and business development. At the end of the day, the legal risk of this project was simply unacceptable to us since it is basically a glorified tornado cash + uniswap and tornado cash had just gotten sanctioned so we agreed to shut it down. https://twitter.com/muteswap

Risk Harbor This was my first serious foray into crypto. I mostly did contracts and integrations for them, growing from 0 to 200 million TVL, writing wrappers that would programmatically detect whether a given DeFi protocol had been hacked and automatically pay out insurance if there had been a hack.

Klade This was my first crypto project, we built a way for bitcoin miners to hedge increases in hash rate. Unfortunately, it went nowhere, mostly because a) the mechanism was not capital efficient and b) we didn’t understand business development(or anything for that matter).

Before Crypto First coding experience was writing games with tynker I didn’t get comfortable enough with building hackathon level projects till the 11th grade, where I started doing a lot of “NLP” projects for hackathons, which basically meant using the Amazon Alexa SDK(back when amazon docs were super ugly) and feeding it a series of hardcoded Q&A logic or using basic word vectorization tools like GloVe and word2vec.

Non Crypto Worked on Coinbase NFTs Worked at the NSA Did research using NLP(GloVe) at the haas neuroeconomics lab

Shorter Crypto Experiences Governance at Blockchain at Berkeley: Delegate on Aave, Compound, Dydx, Optimism, Uniswap Ex: Maker, FEI, Soon: Euler

Basically, we vote yes on everything until something big comes up, then we wait for the big boys like jump and a16z to call us and tell us how to vote.

Quickfire Favorite Project: Favorite Github Repo: Most bullish on: Least bullish on:

Ideas on Tech

Mon, 20 Nov 2023 23:46:37 GMT

Short form

If you ever need to send someone instructions on how to do something or are showing them something they will need to remember how to do. Instead of worrying notes, windows +r to open run command and type in psr.exe. This opens a utility you can then record all your steps. It creates a nice step by step document detailing everything you click on and captures screenshots.
Join Facebook Group to get names of people that go to a certain university Use chrome extension that allows you to get people’s email given organization, first and last name Make a mass email list in hours.
Can you hash people’s faces into something unique and robust?

CS61c material: https://jyunicorn.github.io/teaching.html
List of Remote companies hiring: https://github.com/yanirs/established-remote
NSF REUs: https://www.nsf.gov/crssprgm/reu/list_result.jsp?unitid=5049
Top University Admissions officer: https://www.removeddit.com/r/ApplyingToCollege/comments/9f8zl3/_/
16b notes: https://github.com/rahularya50/EE16B-Notes, https://github.com/saahilchadha1/EE16B-Notes/tree/master/pdfs

DeFi

Defi Accelerator: https://www.defialliance.co/
Defi safety: https://docs.defisafety.com/misc-and-in-work/pancakeswap-process-quality-review-v2#author-details
Defi score: https://defiscore.io/overview
Intro to bitcoin: https://michaelnielsen.org/ddi/how-the-bitcoin-protocol-actually-works/

Internships

Leetcode prep: https://github.com/jwasham/coding-interview-university
Resume Rules: https://www.nicksingh.com/posts/36-resume-rules-for-software-engineers
School of SRE: https://linkedin.github.io/school-of-sre/
Blind top 75: https://www.teamblind.com/post/New-Year-Gift---Curated-List-of-Top-75-LeetCode-Questions-to-Save-Your-Time-OaM1orEU
System Design Framework: https://www.teamblind.com/post/Willing-to-help-for-system-design-xQQ5u63y

Learning Tech Stuff

Web dev tutorial: https://www.theodinproject.com/
SQL tutorial: https://mystery.knightlab.com/
No code websites: https://www.nocode.tech/

Misc

Places to shill at: https://www.notion.so/logology/Where-to-post-your-project-4b6b9c20d7d7462f871e7e71b6449f62
Polygraph stuff: https://antipolygraph.org/documents/dodpi-interrogation.pdf
Notebook fan control: https://github.com/hirschmann/nbfc/wiki/FAQ
Bootstrap themes: https://www.creative-tim.com/
Hawk and serpent(Investing read): https://artemiscm.docsend.com/view/taygkbn
Networking with people: https://bryanwang.substack.com/p/a-guide-to-authentic-networking

Long form

Toxicity in Computer Science:
- A lot of people get into software because the computer understands them. The computer doesn’t make faces that you don’t understand or behave in ways that are unpredictable. So you spend more time with the computer than people. The software doesn’t make fun of you or talk about you behind your back.
- Then as you grow older and enter college you meet hundreds of other people just like you. And now all of the social deficiencies that you had in high school are celebrated as eccentricities. So you are pushed even further away from people and into the world of software. And in this world the only thing that matters is your grade on the assignment. Nobody grades you how well you are liked personally by your peers.
- Then you enter the workforce and you are a rockstar. You’re making more money out of the gate than most of your high school friends who majored in communication or business. And the more you learn, the more money you make. You buy a new car, you get a better apartment. And you must be smart because you look back at all of the people who made fun of you in high school and they are working at an insurance company for $16 an hour and you think : “I always knew I was smarter than them.”
- So when someone asks a question on Reddit or Stackoverflow you jump on them because why can’t they see it? It’s obvious. Maybe they shouldn’t be in this profession because they can’t compete with how very smart you are.

Cool projects from eth hackathons

https://ethglobal.com/showcase/safenode-shtqy https://ethglobal.com/showcase/2inch-wxtu0 https://ethglobal.com/showcase/freedombridge-fcab4 https://ethglobal.com/showcase/cross-margin-engine-for-derivatives-zgy04 https://ethglobal.com/showcase/sdao-sorition-based-dao-gjcbr https://ethglobal.com/showcase/leverage-yield-yf3a4 https://ethglobal.com/showcase/sheeth-the-wallet-condom-pvq8n https://ethglobal.com/showcase/nested-amm-uwe1a https://ethglobal.com/showcase/decentra-date-j6u9t

Dynamic Fees Postmortem

Fri, 20 Jan 2023 00:00:00 GMT

Kinesis Dynamic Fees Postmortem

Contact Information

Telegram(Preferred): @kinesislabs

Email: team@kinesislabs.co

Discord: Dr.Kinesis#0157

Key Points

All user funds are safe
The issue here is incorrect arthimetic leading to safemath reverting when attemping to swap tokens, meaning swaps are disabled. However, users can still unstake and withdraw from the pools.
The plan is to set rewardsPerSecond to 0 on the three current Simple Rewarders, send the rest of the staking rewards to the new SimpleRewarders, and ask users to migrate their funds to the new SimpleRewarders

The Issue

We initially got a report from multiple users in discord that they were having trouble swapping tokens in the new DAI/USDC/USDT pools. We initially chalked these up to RPC issues, which are not abnormal with the current Evmos infrastructure. However, when we got the following report from GV of Swiss Staking, we knew something was probably wrong as a power user like GV would know to change the RPCs and to try the transaction again after waiting a few minutes.

We then tested the exact swap ourselves, axlDAI for ibc G-DAI and got the following error.

The error is execution reverted: SafeMath: division by zero. This was a very bad sign and we immediately knew something went very badly wrong with the contracts - this error could no longer be from Evmos infrastructure or our frontend.

After triaging this a bit further, we found the exact line in the frontend that made this call to the swap() function and thus we knew the issue was in the swap() function of the contracts.

The swap function is very simple, just two lines:

    function swap(
        uint8 tokenIndexFrom,
        uint8 tokenIndexTo,
        uint256 dx,
        uint256 minDy,
        uint256 deadline
    )
        external
        payable
        virtual
        nonReentrant
        whenNotPaused
        deadlineCheck(deadline)
        returns (uint256)
    \{
        updateSwapFee(minDy);
        return swapStorage.swap(tokenIndexFrom, tokenIndexTo, dx, minDy);
    \}

There were only three functions that could have held this error, the deadlineCheck() modifier, the updateSwapFee() function, or the swapStorage.swap() function.

The deadlineCheck() does not have a SafeMath div call, meaning the error could not have been there. Additionally, the SafeMath divs in swapStorage.swap() both use constants, meaning it could not be those functions either.

This narrows down the scope of the issue to the updateSwapFee() function. For full disclosure, this is a newly added function, which dynamically changes the swap fee based on the volume swapped in the last hour. There is one SafeMath div in the function.

uint256 newSwapFee =
(feeFactor.mul(hourlyVolume))
.div(averageVolume).add(baseFee);

Since the error was execution reverted: SafeMath: division by zero, we know that averageVolume is zero, making the entire swap() function revert.

Average Volume is set in Line 527 and we can quickly see the error.

averageVolume = averageVolume
.add(averageVolume.sub(recentVolume[recentVolumePointer].volume))
.mul((10 minutes / (block.timestamp.sub(creationTimestamp))));

10 minutes after the contract is created, block.timestamp.sub(creationTimestamp) will be greater than 10 minutes. In solidity, there are no decimals, so 1/2 will round to zero. Thus, (10 minutes/block.timestamp.sub(creationTimestamp)) will round to zero after the first 10 minutes, meaning that Average Volume will always be set to zero, and thus the swap() function will always revert.

Triaging the Issue

The first thing we tested was withdrawing liquidity as we want to be 100% sure that user funds were not at risk. To this effect, we tested Deposit, Withdraw, Stake All, and Unstake All to ensure there are no issues with any of these functionality. Luckily, there were no issues here and USER FUNDS ARE SAFE.

Thus, there are a few issues to keep in mind. We want Kinesis to be a useful application on Evmos instead of a place to stake stablecoins to receive Evmos. Thus, we would need to either upgrade the contract or to deploy new contracts. Unfortunately, we do not have upgrade functionality to violate the core principles of decentralization and immutability.

We then decided the best way forward would be to redeploy the contracts and send the rest of the rewards(87,343.52 wEVMOS) to the new simple rewarders.

Next Steps

Redeploying Contracts

Contracts have already been redeployed to these addresses:

USDCPool: 0x35bF604084FBE407996c394D3558E58c90281000
USDCPoolLPToken: 0xfD2fd675176a8Ed1CF643886ee557929FDEcBBfD
USDTPool: 0x89E9703309DA4aC51C739D7d674F91489830310E
USDTPoolLPToken: 0x78549EF94dB08E8bf2e528F0aE97F186Fc51185E
DAIPool: 0x155377C4f5489026cD8340fF350ae6aa082FBE69
DAIPoolLPToken: 0x9607FFD92DC6846F913129d8351a848240BEC4E9
MiniChefV2: 0x82a4A94eC86f03bf1a5979B41e2Ae8478C93f9C3
SimpleRewarder_usdc: 0x5Ff26712e2Ec975d7F25a31Ba7c9FbA395655076
SimpleRewarder_usdt: 0xDc85D4f941B8adc7cF223C9AFd5305c2684D7090
SimpleRewarder_dai: 0x0BCC8b2E95e0e59a23fc6D8DD653636047cf3C3f

The frontend has also been redeployed here with the new pools and the old pools. https://frontend-git-a-featpostmortem-kinesislabs.vercel.app/#/

Sending Rewards

Afterwards, we need sent the rest of the wEVMOS liquidity mining rewards to the new Simple Rewarders. This process would be the exact same as what LPX already has done in the past.

LPX, can you initiate the transactions for this when we have agreed to a path forward?

Public Announcement to Migrate Funds

We will need to make a public announcement for users to migrate funds from the old pools which will be depreceated to the new pools. There are two options here:

We can admit we messed up and explain the full scope of the issue
We can hide this as a routine upgrade to activate dynamic fees.

We are leaning towards the former reason, however, if others in the Evmos ecosystem wish to use the latter reasoning to keep confidence in the ecosystem, that is understandable as well.

Set Rewards Per Second to Zero

Finally, we would set rewards per second on the current SimpleRewarder Contracts. This is to ensure everyone can unstake their LP tokens from Minichef such that they can withdraw their LP tokens, as if there are not enough LP tokens on Minichef, users will no longer be able to unstake their LP tokens.

LPX, can you initiate these three transactions? You would need to call
-

function setRewardPerSecond(uint256 _rewardPerSecond) public onlyOwner \{
        rewardPerSecond = _rewardPerSecond;
        emit LogRewardPerSecond(_rewardPerSecond);
    \}

in SimpleRewarder.sol with _rewardPerSecond set to 0.

TODOs on Kinesis’ End

Update Minichef to the new minichef contract address on the frontend when rewards are live on the new contracts

Artificial Growth - Giving out 100k in 60 days to grow faster

Fri, 20 Jan 2023 00:00:00 GMT

The easiest way to grow your defi protocol is to start liquidity mining. Normally, projects will distribute their token https://commonwealth.im/evmos/discussion/4752-bootstrap-liquidity-on-evmos-dapps

https://commonwealth.im/evmos/discussion/6844-temperature-check-start-liquidity-mining-campaign-for-gravity-celer-and-axelar-assets

https://commonwealth.im/evmos/discussion/7017-passed-proposal-for-axelar-network-celer-network-and-gravity-bridge-liquidity-mining

Appendix

Temperature check - Start Liquidity Mining Campaign for Gravity, Celer, and Axelar assets

Hi Evmos Community!

We are planning to make a proposal next week: Liquidity mining campaigns for stablecoins from Celer Network, Gravity Bridge, and Axelar Network.

The goal of this proposal is to bring stablecoin liquidity back to Evmos. In the past, Kinesis Labs has reached over 255k USD worth of stablecoins in TVL without any incentives, with over 11,000 transactions and 400 unique liquidity providers. All of this was organic liquidity from the Evmos community, there were no funds or deals done to bootstrap this liquidity.

However, with the recent nomad hack, both Kinesis and Evmos need a way to bootstrap liquidity to jump start the Evmos DeFi ecosystem. We propose the following plan to bring stablecoin liquidity back to Evmos:

33k Evmos tokens to be distributed to the Axelar Network Pool over 6 months as soon as they finish their deployment to Evmos mainnet. Please note that the assets in this pool are to be decided as we are still unsure what assets will get adoption on Evmos, but the pool will likely consist of blue chip stablecoins.

33k Evmos tokens to be distributed to the Celer Network Pool over 6 months as soon as the proposal passes. Please note that the assets in this pool are to be decided as we are still unsure what assets will get adoption on Evmos, but the pool will likely consist of blue chip stablecoins.

33k Evmos tokens to be distributed to the Gravity Bridge Pool over 6 months as soon as this proposal passes. Please note that the assets in this pool are to be decided as we are still unsure what assets will get adoption on Evmos, but the pool will likely consist of blue chip stablecoins.

We are requesting a total of 99k Evmos for this proposal to be linearly distributed over 6 months.

All of these funds will be distributed to liquidity providers on Kinesis through the minichef contract, no funds will ever reach an EOA or multisig - these funds are not for the Kinesis Team nor the Axelar, Celer, or Gravity teams. Additionally, the only fee on Kinesis is the 0.04% swap fee which goes entirely to LPs, and Kinesis does not currently have a token or any other value extraction mechanism. We are building this project as members of the Evmos ecosystem as a public good.

The multisig that controls these funds will be a 4 out of 6 consisting of: Nic Z, Co-Founder of Tharsis (Evmos Core Team) Akash Khosla, Co-Founder of Tharsis (Evmos Core Team) Liam DiGregorio, Head of Business Development at Tharsis (Evmos Core Team) Mo Dong, Co-Founder of Celer Network Deborah Simpier, Co-Founder and CEO of Althea (Gravity Bridge) Georgios Vlachos, Co-Founder of Axelar

The reasoning behind these members is that these would be the key stakeholders for this proposal as they are members of Tharsis, the team building Evmos as well as one representative from each bridge that will be launching. There will be

Additionally, Celer will match this with a minimum of 1m in Celer stablecoin liquidity, which will be deposited into the Celer stablecoin pools.

We’ve been working closely with the Axelar Network, Celer Network, Gravity Bridge teams on this initiative. If there’s any questions, the Kinesis + Axelar + Celer + Gravity bridge teams will be happy to answer down below.

Background on Axelar Network Summary: Axelar network is built from the ground up using Cosmos SDK to interoperate between Cosmos chains, EVM chains, Bitcoin, and other PoS and PoW chains. Satellite is the cross-chain asset transfer app built on top of the network and has amassed over $37m TVL and 100k cross-chain transactions since launching in February. It has also recently become the official canonical bridge for Osmosis and plans to further expand its reach into the Cosmos and EVM ecosystem.

Satellite currently supports Avalanche, Polygon, Terra, Osmosis, Moonbeam, Fantom, Ethereum and Cosmos Hub with plans to support all major EVM chains over the next couple months.

Satellite connects all Cosmos chains via IBC and its GCP protocol with multiple ecosystems. It’s built on top of Axelar’s “decentralized overlay network” that connects different blockchain stacks. Axelar serves as the translation layer that allows Cosmos assets to flow freely to all Axelar-interconnected networks and back. It is permissionless, meaning that anyone can join, submit transactions, relay, and validate.

Axelar has been audited by Ackeeblockchain, Cure53, NCC, Oak Security, Commonprefix labs and independent researchers and is also dedicating at least 5% of AXL supply to an insurance fund/program that will help recuperate any stolen funds to ensure the safety of its users.

Security:

Axelar believes that bridging solutions should be at least as secure as the chains being bridged. Axelar’s solution is having a Cosmos chain, one with a similar validator set to other chains. Additionally, they are heavily invested in securing their smart contracts and in applying cutting-edge cryptography.

The Axelar network isolates functionalities in modules at the Cosmos SDK level, e.g., one module for bitcoin, another for EVM chains, another for IBC transfers, etc.

Each module handles the respective transactions from source and destination chains and are isolated from each other and messages can only be passed through a router module. If there is an issue with one or all chains, a special transfer command that can be supplied by the governance multisig can pause the router module.

There is also minimal reliance on smart contracts and are only used for key-rotations and mint/burn operations.

The mint/burn contracts can also be freezed in case of a compromise and have a rate-limit functionality that allows to minimize how much funds can be stolen in case of an attack. This is not enabled on the mainnet at the moment, but can be considered along with other “access control” policies to minimize attacks.

There will also be an insurance fund/program of at least 5% of the AXL supply to compensate users in case of stolen funds. Axelar has also been audited by Ackeeblockchain, Cure53, NCC, Oak Security, Commonprefix labs and independent researchers and gone through a dozen audits.

Costs:

Axelar currently has a set fee on each transfer that covers gas costs on source + destination chain. Axelar foundation subsidizes any increase in the destination gas cost if need be and is also considering subsiding fees further from its treasury.

Transactions are also batched, reducing the gas costs for the users significantly. Axelar also has active implementation of threshold signatures that can be used to replace individual validator signatures and further save gas costs. While not currently deployed in EVM contracts, this is just one of the many code-level gas optimizations that Axelar will ship in the future.

Users only pay the fee in the asset they transfer on the source chain, and all other fees (relaying / intermediary / destination chains) are covered by services around the network.

App: https://satellite.axelar.network/

Background on Celer Network Summary: Celer Network is a multi-chain operating system built with Cosmos SDK. Celer released the world’s first EVM-supported inter-chain message SDK. and has a track record of no hacks for any product since Celer originated in 2018.

Celer cBridge is a token bridge application built on top of Celer that supports 45 different tokens across 21 different blockchains and layer-2 rollups. cBridge has $400M TVL and has processed more than$ 4B of cross-chain volume for 105K unique users with a daily volume of $50M-$ 60M across 21 different blockchains.

cBridge comes with two security models that every user can choose between, the Cosmos-consensus security model and an optimistic-rollup-style delay buffer security model. The consensus model is security that of a validator-based bridge, while the optimistic rollup style also has a delay buffer to allow an independent watchtower service to double check the message.

Users have single-click UX with fast confirmation of the bridge between source and destination chains and with 2,500-3,500 daily active users comfortable with using Celer Network.

Security:

Cosmos-consensus Security Model: Celer SGN is running as a Cosmos chain with the same level of liveness assumption which is very high like other Cosmos chains. Celer provides an L1-blockchain level security like Cosmos or Polygon as a Proof-of-Stake chain built on Tendermint with CELR as the staking asset. If a validator acts maliciously, their CELR will be slashed.

Optimistic-rollup-style delay buffer Security Model: Inter-chain dApps can maintain full security with an optimistic-style delay buffer. Instead of instantly processing a message through the consensus model, one can inject a mandatory delay buffer and run an independent watchtower service to double-validate the message on the source chain. If the watchtower service detects any inconsistency, it can prevent the message from being processed before the delay expires.

For transactions larger than a certain amount, two-transactions are needed to make the bridge happen: a “commit” transaction that will trigger the time buffer and then after the time buffer, a “confirm” transaction. The watchtower system also monitors the bridge for small transactions as well to detect and prevent any unexpected smart contract bugs. There is also rate-limiting for token bridging as the last-resort to worst-case scenario.

In addition, Celer cBridge smart contract has been heavily audited and also has a $2M ImmuneFi bug bounty. Celer Network also has a track record since 2018 of no hacking event happening for any product they have built.

Costs: cBridge charges the gas fee on the destination chain to cover the operation costs of validators. The fee is not related to the size of the transaction, unlike many other bridging solutions.

Celer also has no plans to change the fee structure and remains committed to supporting inter-chain apps where IBC is not available. Unlike other bridges that plan on changing to percentage-based fees later, Celer will only charge a flat fee based on the size of the message passed to provide revenue for validators.

There are also no platform fees charged by the Celer Network for token asset bridging.

App: https://cbridge.celer.network/#/transfer Background on Gravity Bridge

Summary: The Gravity Bridge blockchain is the trustless, neutral bridge between Ethereum and the Cosmos ecosystem. The principal focus of the Gravity community is on providing the most effective and secure bridge possible, instead of a DeFi application on the local chain.

Gravity Bridge protocol focuses on creating a simple bridge design that focuses on providing better security by eliminating design complexity and using non-upgradeable contracts which have been audited by Certick, Least Authority and Cod4rena.

The Gravity Bridge protocol provides strong censorship resistance guarantees and the bridge will remain live as long as the chain remains live, i.e. with more than 2/3 of validators active.

Currently supports Ethereum and has plans to support additional EVM chains before the end of 2022.

While there is not currently a set-aside for an insurance fund, one could easily be set aside from the community pool which contains 50% of $GRAV token distribution which could also contain a collection of assets.

Security:

Gravity Bridge is trustless, decentralized and highly efficient and offers the closest approximation of IBC security possible. It is highly decentralized with all funds directly controlled by validators with slashing to deter validators from acting in bad faith.

The Gravity Bridge design means that the only way to attack the bridge is to go through the validators themselves. This results in a trustless bridge that costs less to operate than a custodial one. It is not possible for a validator to avoid signing a specific transaction or for a relayer to avoid relaying it. Anyone can submit evidence of a signature over a non-protocol message to slash a validator which makes it extremely difficult for validators to steal funds.

The Gravity Bridge, like all bridges, has two components. A Solidity contract on Ethereum and a CosmosSDK module on the Gravity Bridge blockchain.

The solidity contract that holds funds on Ethereum is merely 580 lines of code, which makes it compact and easy to review. It has been audited by three independent teams (Certik, Least Authority, and Code4rena), and it is not upgradable. It requires no trusted parties and can be temporarily paused by governance vote in case of a possible security breach. As a final layer of protection the Gravity Bridge blockchain performs a series of integrity checks that will halt the chain on any unexpected behavior.

While there’s nothing set for an insurance fund at the moment, the $GRAV token holders can choose to use some funds from the large community pool to help recuperate stolen funds.

Costs:

Robust relaying reward systems combined with reduced gas costs creates extremely fast & reliable transfers with a decentralized, self-funded, relayer community

Gravity Bridge batches withdraw transactions. This works like a rollup on Ethereum. Executing many transactions in a single shared context is far more efficient than doing so individually. This process provides huge efficiency gains, up to 96% reduction in gas cost.

Once bridge traffic has matured, tx fees could be enabled by the community to sustain the community pool. These tx fees are payable in a wide variety of tokens, not solely in GRAV. Other mechanisms, like a portion of the inflation, could be used to fund the community pool.

App: https://spacestation.zone/

Background on Kinesis About Kinesis: Kinesis is a decentralized automated market maker (AMM) on the EVMOS blockchain, optimized for trading pegged value crypto assets with minimal slippage such as stablecoins. Through Kinesis, all traders can swap their desired stablecoins between EVM compatible blockchains and Cosmos chains.

Kinesis enables efficient, swift, and low-slippage swaps for traders and high-yield for liquidity providers. We believe in collaboration with other Cosmos projects, in maximizing the interoperability of Kinesis, and in building out the stablecoin ecosystem for Cosmos.

Kinesis has no governance token, no admin fees, and no value extraction mechanism. We are building this project as members of the Evmos ecosystem as a public good.

Crypto

Fri, 20 Jan 2023 00:00:00 GMT

Everything I’m still bullish on

Bullish on Team and Idea

Eigenlayer: You’ve probably heard of Eigenlayer by now. For me, Eigenlayer is one a project that has the potential to grow so big that it becomes an existential threat to Ethereum, since if the supply of re-staked ETH approaches the total supply of staked ETH, weird incentives around the Ethereum Consensus mechanism could happen - or worse yet, there is an exploit in one of Eigenlayers’ contracts. The reason for Eigenlayer’s near guarenteed success is simple: anything that can be innovated on in the Ethereum ecosystem will be tested with Eigenlayer first. The main risks to Eigenlayer are also close to solved. Eigenlayer needs to grow fast enough to generate an ecosystem of applications that use restaked ETH, otherwise they risk someone forking their ecosystem and giving a bit less of their tokens to investors. k

Additionally, the team is a powerhouse. Instead of being a collection of anons that met on the internet, Sreeram Kanaan brought his entire research lab from the University of Washington - Seattle, a top 10 CS research university. Having met a few of the team, here are some highlights:

Sreeram: Professor at UW, Postdoc at Stanford and UC Berkeley. He’s one of those rare people who has massive raw brainpower. I learned more about consensus mechanisms in 1 hour talking to this man than in the previous six months.
Gautam: A paradigm fellow, a child genius who only eats frozen vegetables and hard boiled eggs for entire weeks since “life is too comfortable”
CryptoBanker: Has found multiple exploits in live contracts as a whitehat
Bowen, Shoubik, and the rest of the lab: PhDs at UW, nuff said.

Hot take: One year after Eigenlayer launches, it will have the highest TVL in all of Crypto.

Projects to be built on top of Eigenlayer

Hyperscale Data Availability Layer: We can build a DA layer providing high DA rate and low cost by utilizing EigenLayer re-staking and some of the cutting edge ideas in data availability developed in the Ethereum community, including Danksharding. Notes: The Eigenlayer team is already working on this as the first product, not much alpha to take from here unless you want to challenge them on their own turf.

Decentralized Sequencers: Many rollups require decentralized sequencers for managing MEV and censorship-resistance at L2. These can be easily built on EigenLayer exploiting ETH stakers. Notes: Not sure how this would work nor what a value a decentralized sequencer would bring

Light-node bridges: It is easy to build light-node bridges to Ethereum using EigenLayer. For example, the rainbow bridge between Near and Ethereum is based on an optimistic pattern but it suffers from high-latency due to the high gas-cost of verification. It is possible for re-stakers to verify off-chain whether such inputs are correct, and if a strong cryptoeconomic quorum signs off on the bridge input, then the bridge input is considered accepted. If someone challenges, then the bridge input can be verified and the restakers can be slashed in the slow mode. Notes: This is just an optimistic bridge with no privileged positions, it also speeds up existing bridges

Fast-mode bridges for Rollups For ZK rollups, since the proof verification expense on Ethereum is still high, the rollup sequencers write to Ethereum somewhat infrequently, thus affecting composability and delaying strong confirmation guarantees. It is possible for a large amount of restaked quorum to participate in ZK proof verification offchain and certifying that proofs are correct on chain. These fast mode bridge claim if they turn out to be wrong can then trigger a slower slashing path. For optimistic rollups, eigenlayer can enable a much larger collateral pool to participate in certifying stateroots under risk of slashing. Notes: This just speeds up bridging from rollup to eth mainnet.

Oracles: There have been proposals for enshrining price feeds into Ethereum or using the UNI token quorum for providing price feeds. Such oracles can be potentially built via Eigenlayer if all we require is majority trust on Ethereum re-stakers, and it is an opt-in layer. Notes: This is just a majority trust oracle. Simple, and not sure if someone would use this over coinbase. There would need to be major advancements needed here for this to be interesting, but sreeram and team say they have something interesting here.

Opt-In Event-Driven Activation: Event-driven activations such as liquidations, collateral refilling are currently not available natively on Etheruem. While they can be built on a separate layer such as keepers, such keeper nodes which do not control blockspace cannot make strong guarantees on inclusion of event-driven actions. In EigenLayer, native ethereum re-stakers, who are also block-proposers, can provide strong guarantees on event-driven inclusion at the risk of getting slashed. Notes: This could also be a major step forward for account abstraction, allowing miners to effectively pay the gas for this transaction if they wanted to. Event driven activations stuff for liquidations, collateral refilling aren’t as interesting but we could check the crazy stuff people do for MEV to see if something would be interesting there.

Opt-In MEV Management : We note that a manifold variety of opt-in MEV management methods become feasible under EigenLayer, including Proposal-Builder Separation, MEV smoothing, threshold encryption for transaction inclusion. As a simple example, MEV smoothing can be built on top of EigenLayer by a committee of ETH restakers who then equally share their MEV. Any restakers who deviate from the prescribed MEV smoothing behavior can be slashed.

Sidechains with ultra-low latency: EigenLayer allows the creation of re-staked sidechains where ETH re-stakers can participate in new consensus protocols, some of which have very low latency and very high throughput. Of course, a precondition for opting in to such sidechains is a higher computational power.

Single-slot finality It is possible to imagine single-slot finality, where nodes sign off on the finality of block via an opt-in mechanism EigenLayer. The core idea would be that nodes who have restaked can now attest that they will not build on a chain that does not include the testified block, thus creating a potential finality pathway. Designing it so that it is truly opt-in and does not break the consensus protocol, is an important direction of research Notes: useful for IBC, but requires much more to be practical

Risk Harbor

Bullish on Idea

Morpho/Ajna/P2P lending Waymont Beanstalk Eigenlayer Primitive finance UXD protocol Nomad On chain foreign exchange or interest rate swaps Gammaswap Lien protocol Sudoswap Nounsdao Risk swap: Use bond curves to swap between underwriting capital and protection purchase. Triggers when something defaults

Bullish on Team

Crocswap

Other

Cosmos Independent Security Auditors

Everything I’m bearish on

Bearish on Team and Idea

Nexus Mutual

Bearish on Team

Canto - Just because of my work on Evmos Saddle Finance - poor code quality

Bearish on Idea

MEV Waves(the blockchain)

Other

Things you might not have know

TVL is fake

People fake TVL all the time. Whether through recursive lending/borrowing, liquidity mining rewards, or a waves play, what you see if often not what you get. However, some projects and firms take this one step further where they make deals to pledge some portion of project tokens in exchange for a certain TVL amount. Some examples of this include platypus finance, where alameda deposited hundreds of millions in stablecoins and dumped the tokens that they received from the team.

Random list of protocols that probably have an exploit or two in them

Compound forks on weird EVM alt L1s. Compound is safe protocol. Compound forks are not safe protocols. There are a wide variety of possible exploits on a compound fork:
- if you use bad oracles, you will get flash loaned and drained
- even if you have flash loan checks, the prices of the least liquid token can probably be manipulated, as there usually isn’t too much liquidity on an alt L1.
- If you have a single ERC-777 or ERC-677 token, someone will hit you with a reentrancy attack
- If you use an old version of the compound v2 codebase, you will get hit with the .transfer() hack, the same one that stole 80 million from the Fei pool in Rari

Tiers of DeFi Teams

There’s basically three tiers of DeFi Teams Tier 1: 30+ people, basically the MakerDAOs and the Uniswaps of the world. They usually have strong PMF and are heavily recognized, they ship open sourced code in addition to working on their project, and generally try to advance the industry in addition to their own projects.

Tier 2: Top End: Frax Finance, Bottom End: Saddle Finance. This is a barebones team, where there is at least one person covering each of the major facets required to run a DeFi project. They usually have one of each of the following: Solidity Engineer, Frontend Engineer, Designer(usually contracted), business lead(basically takes all the calls), community lead(sometimes combined with business lead, basically someone to sit in discord all day), and founder. Fun fact, legal action will destroy them, as they have no bandwidth or money to deal with that.

Tier 3: This is either a solo dev or an anon “team” of devs. It usually takes them a while to get going since they are basically a one man show, telegram takes a while for a response, things don’t ship very quickly, tweets usually happen at most once a day, etc. I know this because I did this in the past.

List of Novel or Interesting Defi Protocols you probably haven’t heard of

https://www.gamma.xyz/ https://end.gg/ https://www.arpeggi.io/ https://sloika.xyz/ https://aspect.co/status https://beta.catalog.works/ https://koop.xyz/ https://www.defifranc.com/ https://fantasm.finance/ Timeless finance LIT token distribution tom howard is building stablecoins in cosmos gyroscope

Lessons from a UC Berkeley 4.0

Fri, 20 Jan 2023 00:00:00 GMT

Note: I am not the UC Berkeley 4.0 :{

Midterm 1

Always over study for midterm 1, you don’t know how well your classmates will do on the tests nor the structure and format for the tests, thus, you need to be extra prepared.

One Day Before the Exam, you haven’t started studying. What do you do?

Watch every single lecture and take notes. If you have time left, “Give your brain the tools to do the problems, eventually when you are under enough pressure, your brain will do it for you.”

“If you do practice problems, you will be really good at the practice problems. But if you get a problem that you haven’t seen before, you won’t get it. Going over the lecture and notes is much more important.”

“Optimally, go through all the lectures and take notes, then go back and fill in the notes(since if you are watching lecture it should move too fast for you. Then do the discussions which should now be pretty easy(except if the discussion is full of old exam questions). Then, you are chilling, you don’t even need to take practice tests” A day or two before the exam, take 2-3 hours and split your cheat sheet into the number of lectures, then fill in your cheat sheet according to the lecture.”

Why do you succeed when other berkeley students fail?

Up to +1 std deviations, it’s just work. Up to +2 std deviations, it’s just talent. So many kids are afraid of leaving questions blank, in 170 and 162, I left one subpart blank. Don’t skip questions, but it is coding, its fine to skip. If the point total is too little for the work, come back to it later.

Do you forget stuff after classes are over?

No, I just remember stuff.

Translating skills from class to work?

“I’ve never tried leetcode”

Homeworks

“I just sit down and do it starting at 8 in the morning the day it is due. If I’m not feeling it, I just won’t do the homework” Note: This man didn’t do 3 homeworks in algorithms even though you only have two homework drops

Piazza

“I fucking hate piazza(online forum to ask questions) cause they are fucking dumbasses on there. 170 was the first time since the problems are too much to do by yourself. When people leave how they’ve done problems, then that is useful to look. ” It’s better to talk to people in real life that you know to bounce ideas off of.

Have you ever gotten worse than an A in a class?

Yeah, in AP Calc B. Because I played too many video games

Probability in Electrical Engineering and Computer Science

Tue, 10 Jan 2023 23:46:37 GMT

Introduction to Probability, 2nd Edition Follows the course, can borrow from engineering library

TODO: Set calendar for homework self grades and resubmission time Set midterm time on calendar Choose discussion time Countable vs uncountable

Probability: Math + a way of thinking about uncertainty

Humans are naturally good at thinking about probability. For example, you can easily walk down a crowded street and estimate what path allows you to avoid bumping into people. However, humans are pretty awful at turning those innate estimates into quantifiable numbers. The goal is to take the natural probability estimation into something that a robot can understand.

Quantify + Model

Inference. Humans are also really good at taking in noisy observations and infer what is really happening.

How do I generate some image from AI? Take a natural image and repeatedly add noise to it. You then end up with an image of pure noise. The idea behind stable diffusion and dalle-2 is to reverse this process. We generate a bunch of noise, then reduce noise until we end up with a image.

Modern theory of probability has an axiomatic starting point. Before the 1930s, probability was a heuristic field. All of probability are logical deductions from the starting point.

Probability space: A probability space (Ω, F, P) is a triple. Ω is a set, the sample space. F is a family of subsets of Ω, called events P is a probability measure

Rules:

F is a σ-algebra, it contains Ω itself. F is closed under complements and countable unions.
- If A ∈ F, then any of A’s complements belong to F
- countable unions: If A1, A2, … ∈ F, then the union of Ai ∈ F
- By DeMorgan’s Laws, this implies that F is also closed under countable intersections
P is a function that maps events to a number between 0 and 1. P: F -> [0,1]
- Probability measures must obey Komogorov Axioms
  1. P(A) >= 0 Probability of any event must be non-negative
  2. P(Ω) = 1 The probability of any outcome occuring must be equal to 1
  3. σ-additivity aka countable additivity. If A1, A2, … is a countable sequence of events and disjoint, then P(U) = ΣP(Ai)
  4. counterexample: P = Unif(0,1), then P({x}) = 0 for x ∈ [0,1]. If 1 = P([0,1]), then Σ P({x}) = 0 <- Dive into this later

Examples of the Above Rules

Flip a coin with bias P(Heads with probability p, Tails with probability 1-p).
- Ω = {H, T} The sample space is just heads and tables
- F = 2 ^ σ = {H, T, Ø, {H, T}} These are all subsets of σ
- P(H) = p, P(T) = 1-p, P(Ø) = 0, P({H, T}) = 1 Why are Ø and {H, T} here?
- We then want to create an experiment of n “independent” flips. Is our vocabulary rich enough for this? no
- Ω = {H, T}^n, F = 2^Ω, P(f1, f2, … , fn) = p^{number of flips fi = H} * (1-p)^{number of flips fi = T}
- There can be multiple answers for a probability space. It just needs to be rich enough to capture all the probabilities
Ω now equals all possible configurations of atoms in the universe. How do we represent this?
- Let A = {set of configurations that lead to flip heads}
- Let B = {set of configurations that lead to flip tails}
- F = {Ø, A, B, Ω = {A U B}}
- P(A) = p, P(B) = 1-p, P(Ø) = 0, P(Ω) = 1

Note: The probability space is usually implicitly described, except for HW 1. Lol

All of the rules of probability you are likely familiar with are consequences of the three axioms.

Ex: If A ∈ F, P(A^C) = 1 - P(A). The probability of A’s complement is 1 - probability of A.

Since F is σ algebra, then A^C ∈ F.
- A U A^C. A and A complement is a disjoint union equal to Ω
- 1 = P(Ω) Using axiom 2
- P(A U A^C) = P(A) + P(A^C) Using Axiom 3

Ex: If A and B are events and A is a subset of B, then P(A) <= P(B) B = A U (B \ A). B is equal to the disjoint union of A and B minus the elements of A P(B) = P(A U (B \ A)) => P(A) + P(B\A) Using axiom 3 P(A) + P(B\A) >= P(A) Using axiom 1

Ex: If A and B are events, then P(A U B) = P(A) + P(B) - P(A ∩ B). Aka inclusion exclusion principle Proof: AUB, A∩B ∈ F A U B = B U (A \ B) P(A U B) = P(B) + P(A\(A∩B)) Using axiom 3 P(B) + P(A\(A∩B)) = P(B) + P(A) - P(A ∩B) Using Axiom 3

Ex: Ω = countable set F = 2 ^ Ω, each individual is an event {w} ∈ F, for all w ∈ Ω P(A) = Σ for all A ⊆ Ω This is a discrete sample space.

Ex: Law of total probability. If A1, A2, …, partition Ω, then (Ω = U Ai). Mutually exclusive, collectively exhaustive. Then, for any B ∈ F, then P(B) = Σ P(B ∩ Ai) B = U(Ai ∩ B) then apply axiom 3

The goal of probability is to take complicated things and use that to approximate them into multiple simpler events.

Mathematicians: Given a probability space(model), what can I say about outcomes of experiments that derive from this model? Statistician: Given outcomes, how do I choose a good model(probability space)? Engineers: Given a real world problem, how do I choose a model that captures the essence of the model and then use the model to draw insight.

Lecture 2: Conditional Probability, Independence, Random Variables

Conditional Probability: If B ∈ F and P(B) > 0, then conditional probability P(A|B) = P(A ∩ B) / P(B)

Intuition: Probability A occurs given we know that B occurred

Formal Definition: P(.|B) gives a restriction of our model (Ω, F, P) to those samples in B If (B, F|B, P(.|B)) is a probability space itself, where F|B = \{A ∩ B: A ∈ F\}

Ex. If A1, A2, … ∈ F, P(Ai) > 0, A partion Ω P(B) = Σ P(B ∩ Ai) = Σ P(B|Ai) * P(Ai)

Bayes Rule

Sometimes, it’s easy to express P(B|A), but we are really interested in P(A|B) Let A be the state of the experiment, B be the observation data

P(A|B) - given the observation, we want the state. This is the task of inference.

Suppose we forgot Bayes Rule, let’s try to recreate it from definition of conditional probability P(A|B) = P(B ∩ A) / P(B) = P(A) * P(B|A) / P(B)

Ex. Suppose we have the following model: 85% of students got a pass, 15% of students got a no pass. 60% students that got passes went to lecture, 40% of students did not go to lecture. 10% of students that got no passes went to lecture, 90% didn’t go to lecture.

We want P(NP | N) / P(NP | Y). How much more likely are you to NP when you don’t attend lecture vs when you do attend lecture? P(NP ∩ N) / P(NP ∩ Y) * P(Y) / P(N)

Ex. Suppose we roll two dice and the sum is 10. What is the probability roll 1 was = 4?

B = \{Sum of rolls = 10\}
A = \{first roll is 4\}

P(A|B) = P(A ∩ B) / P(B)
       = P({First roll is 4, sum of rolls is 10}) / P({sum of rolls is 10})
       = P({first: 4, second: 6}) / P({(4,6), (5,5), (6,4)})
       = (1/36) / (3/36)
       = 1/3

Conditioning also allows us to usefully decompose intersections of events.

Ex. Consider event A1 … An P(∩ A) = P(A₁ | ∩ⁿ_{i = 2} A_i) P (∩ A_i)

Ex. Given n people in a room, what is the probability that more than 2 people share a birthday?

A_i = {person i does not share a birthday with any of the people j = 1, …, i-1}

P(A_i | ∩ ^i-1_j=1 A_j) = (365 - (i-1)) / 365

This is because the person needs to land in one of the days not in the 365. P(no shared birthdays) = P(∩ A_i)

Use 1-x <= e^-x. Approximate using taylor series

Let n = 23 - there are 23 kids in a class. P(shared birthday) >= 1 - e^((i-1)/365))

Independence

Events A, B are independent if P(A ∩ B) = P(A) * P(B) Disjoint events are not independent Note:\ in a special case where P(A) > 0: A,B: dependent <=> P(B|A) = P(B)

In general, collection A₁, A₂, … ∈ F are independent events if P(∩_{i ∈ S} A_i) = ∏ _{i ∈ S} P(A_i) for all finite sets of indices S

If A₁, A₂, … ∈ F are independent, then B₁, B₂, … are independent, where each A_i = B_i or B_i^C Intuitively, we assume that knowing A means we know A’s complement

∩_i=2ⁿ A_i ∏P(Ai) = P(∩ Ai) = P(∩) + P(A1) (1- P(Ai)) ∏ P(Ai) = P(A1^C ∩ ∏_i=2Ai) This shows A₁^C, A₂, … , A_n are independent Therefore, B₁, B₂, … , B_n are independent

Conditional Independence

Often times we have two events that we think of as independent but aren’t. There might be a confounding variable C If A,B,C are such that P(C) > 0 and P(A∩B|C) = P(A|C) P(B|C), then A,B are said to be conditionally independent given C

Consider two coins with bias p!=q Pick a coin at random, and flip twice. H(i) = Event that flip i is heads Are the two coinflips independent? No. If p is 1 and q is 0, then we know what the next coin flip is given our first coin flip.

P(H_i) = p + q / 2 P(H₁ ∩ H₂) = p² + q² / 2 != P(H₁) P(H₂) = (p+q)² / 2 C = {pick coin p} H₁, H₂ conditionally independent given C

TODO: Set calendar for homework self grades and resubmission time Set midterm time on calendar Countable vs uncountable

Random Variables

Password: Teapot

Def: A random variable X is a function X: Ω -> R. It implies it is valid to write things like P(X <= 3). P(X <=3) is shorthand for P({ω: X(ω) <= 3}) Random conditions have a measurability condition

Often times we want to compute more complex probabilities, such as P(α < X < β). {ω: X(ω) < β} = U_{n >=1}{ω: X(ω) < β -1/n} ∈ F {ω: X(ω) > α} = {ω: X(ω) <= α}^C ∈ F P(X ∈ B) for pretty much any B(subset of R) that you want Here, the technical name for B is Borel sets

Another consequence of the definition of random variables: If X,Y are random variables on (Ω, F, P):

X+Y is a random variable
X*Y is a random variable
|X|^p is a random variable

Probability Distributions

For any random variable X on probability space (Ω, F, P), we can define its distribution(aka its “law”) L_x via L_x(B) := P(X ∈ B), B ∈ R For example, L_x({x}) = P(X=x)

The histogram of values that a random variable would take.

Similar to a probability measure.

In practice, we often describe our model for experimental outcomes in terms of distributions. Given this, I can always construct a probability space and random variable X that has this distribution.

Ex: Given distribution L, we can consider probability space (R, B, L) and random variable X(ω)=x, ω ∈ Ω = R Distribution of X

Important Class of Random Variables; Discrete Random Variables: A random variable that takes countably many values Ex: X = flip of a p-biased coin. This is a bernoulli distribution with probability p

X = roll a dice. This is a uniform distribution over {1,2,3,4,5,6} X = number of coin flips until I flip heads. Geometric Distribution X = number of heads in n flips. Binomial Distribution (n, p)

In case of a discrete random variable X, the distribution of X can be summarized by its probability mass function. P_x(x) := P({X=x}) = P{ω: X(ω) = x}

By axioms, p_x(x) >= 0 and Σp_x(x) = 1

Joint Distributions

For a pair of discrete random variables X,Y defined on the common probability space (Ω, F, P), their joint distribution is: summarized by the join probability mass function defined via P_xy = P(X=x, Y=y) = P({ω: X(ω) = x && Y(ω) = y}) = P({ω: X(ω)} ∩ {ω: Y(ω) = y}) We can obtain the “marginal” distributions by P_x(x) = Σ P_xy(x,y) using the law of total probability Σ P(x) = 1

Independent Random Variables

Def: Discrete Random Variables X,Y are independent if their probability mass function factors into their marginals P_xy(x,y) = P_x(x)P_y(y)

This is equivalent to saying {ω: X(ω) = x}) and {ω: Y(ω) = y}) are independent events for all x,y

Examples of Joint Distributions: X = {0 if patient tests negative with probability 0.9, 1 if patient tests positive with probability 0.1} Y = {0 if patient is negative, 1 if patient is positive}

We know the false positive rate of the test is 5%, the false negative rate of test is 1%

Question: What is P_XY? I test my patient. The test is either negative(.9) or positive(.1).

Expectation - Used for discrete random variables

Takes in a random variable, spits out a distribution For a discrete random variable X on (Ω, F, P), we define its expectation E(X) = ΣxP{X=x} = Σxp_x(x)

Ex: If X Ω = {0,1}ⁿ, P({ω}) = p^{{number of heads}} (1-p)^{{number of tails}} ie a sequence of independent p biased coin flips

X(ω) = {ω = 9}. <= Number of heads E[X] = Σ{i : ω_i= 1}, p^{{i : ω_i= 1}} (1-p)^{{i : ω_i= 1}}

= Σ_k=0ⁿ (n k) p^k (1-p)^n-k = np

The most important property of expectation is that it is linear. Expectation is an operator, it takes a function and spits out a number, just like an integral. Just like how integrals are linear, expectation is also linear.

The most important and only property of expectation E[X+Y] = E[X] + E[Y]

Linearity of Expectation

If X,Y random variables defined on a common probability space. E[aX + bY] = aE[X] + bE[Y] <= There is no need for expectation to be independent or discrete.

LemmaL E[g(z)] = Σ g(z) p_z(z)

By the law of the unconscious statistician E[aX + bY] = Σ(ax + by)P_xy(xy) = aΣxP_xy(x,y) + bΣxP_xy(x,y)

Sum of Independent Binomials, Variance, Covariance, Correlation Coefficient, Entropy

General Strategy for Expectation

We introduce indicator random variables and express random variable of interest in terms of the indicators, then use linearity of expectation
In general, the indicator for event A is denoted by 1_A(w) = {0 if ω is not A, 1 if ω is A} a. 1_A = Bernuolli(P(A)), E[1_a] = P(A)

Ex: n people put their hats in a bucket, and each draw 1 hat. Let X = number of people who get their own hat back. X = Σ_i=1ⁿ1_A A_i = {gets own hat back} E[x] = Σ_i=1 E[1_{M/sub>]= n * 1/n = 1}

Ex: Coupon collector problem How many on boxes on average do I need to buy before I collect all N coupons X_i = number of boxes to buy to get ith coupon, after getting (i-1)^th
R[].

Tail sum formula: Another way to compute expectation without using linearity of expectation.

For non-negative integer values random value X_i E[X] = Σ_k=1^infP{x=k}

Proof: write P({x>= k}) = Σ_x>=k P{x >= k} = Σ_x>=k P(X=x)

X₁ = Geom(p), X₂ = Geom(p) M = min{X₁, X₂} P{M>= k} = P{x₁}, P{x₂}

IE[M] = Σ P{x₁ >= k} P{x₂>=k} = 1 / (1- (1-p)*(1-q)))

Variance

Quantitative Notion of variability of X around E[X] Var(X) = E[(X - E[X])²] = E[X² - 2XE[X] + (E[X])²]= E[X²] - (E[X])²

If expectation is the price of a stock, variance is the volatility. Variance is not the only way to measure the variability of a random variable. Another way to measure the variance of a random variable is entropy. Entropy = Σ P_x(X) log(1/P_x(X)) = E[log(1/P_x(X))]

Unlike expectation, variance is not linear with respect to its arguments. Var(X+X) = ? Var(X+Y) != Var(X) + Var(Y) Var(X+Y)

By definition of variance = E[X+Y - E[X] - E[Y]]²

Open up the square using foil = Var(X) + Var(Y) + 2 E[(X-E[X])(Y-E[Y])]

[X - E[X])(Y-E[Y]) = Covariance

Positive covariance means X and Y move in the same direction If X and Y are independent, covariance is 0

If X,Y are uncorrelated(covariance of X,Y = 0), then Var(X+Y) = Var(X) + Var(Y)

Correlation coefficient: Covariance(X, Y) / sqrt(Var(X)*Var(Y)). It is always between -1 and + 1

Covariance(X,Y) <= sqrt(Var(X) * Var(Y)) Using cauchy schwartz inequality, x^Ty / (||X|| ||Y||) is between [-1, 1]

E[XY] <= E[X²]^1/2 E[Y²]^1/2 E[XY] = Σ P_xy(x,y)^1/2 x * P_xy(x,y)^1/2 y <= (Σ_xyP_xy(x,y)x²)^1/2 (Σ_xyP_xy(x,y)y²)^1/2

Let X ~ Binomial(n, p) X = Σ X_i X_iBernuolli(p) Var(X) = Σ Var(X_i) = n Var(X₁)

Poisson random variables

Poisson means fish in french. Think number of arrivals when you see poisson If I dip a net into the water and pull it out, a poisson distribution is a good way to checking how much fish I pull out.

X ~ Poisson(λ) λ = rate P_λ)(k) = λ^k/k! e^-λ, k = 0,1,2,…

Imagine trying to take a bus in berkeley. They are supposed to be at each station every 30 minutes. However, there are times when no buses come for 45 minutes then two buses come in the 46th and 47th minute. You can model the arrival of the buses using a Poisson distribution

Where does Poisson come from? Each interval something will arrive with probability p_n = λ/n. The probability that something arrives in an interval is X_n ~ Binomial(n, p_n)

P(X_n=k) = Binomial PMF = (n k) * (λ/n) * (1-λ/n)^n-k = n(n-1)…(n-k+1) (1/n-λ)^k λ^k/k! (1-λ/n)^k as n goes to infinity, this distribution converges to a poisson distribution λ^k/k! * e^-k

Continuous Distributions, Continuous Random Variables

Password: Mirror

Conditional Distributions

P(A|C) := P(A ∩ C) / P(C)

When X,Y are discrete: their joint pmf is P_xy

P_x|y(x|y) := P_x|y(x,y) / P_y(y) = P({X=x} ∩ {Y=y}) / P({Y=y})

Since P_x|y is a pmf for each y with P<sub>y</sub>(y) > 0, we can take expectation of X with respect to it.

This is defined as conditional expectation E[X|Y=y] := Σ_x⊆X xP_x|y Once we have the value of Y, what is the value of x? We usually just write E[X|Y] to denote Σ_x⊆X xP_x|y evaluated at random variable Y. Thus, the conditional expectation is itself a random variable since it is a function of the random variable Y.

Tower Property

Most important property of Conditional expectation:

For all functions f, we can say E[f(Y)X] = E[f(Y)*E[X|Y]]
- E[f(Y)X] = Σ_y p_y(y)f(y) * Σ_xP_x|y(x|y)
- E[f(Y)X] = Σ_x,yP_xy(x,y)*f(y)x

Example: Iterated Expectation E[X] = E[E[X|Y]] We get this using the tower property by setting f(Y) = 1

Example: Let N>= 0 be an integer valued random variable. Let’s flip a fair coin N times, and let X = the number of heads. There are two sources of randomness, the number of flips and whether we get heads or tails. E[X] = E[E[X|N]] We can fix the number of coin tosses, and since half of coin tosses are heads E[X|N] = N / 2 Substitute E[N/2] Use linearity of expectation 1/2 E[N]

Var(X) = E[Var(X|N)] + Var(E[X|N]) Using the same substitution as above of E[X|N] = 1/2 Use binomial theorem for E[Var(X|N)] = E[N/4]. Variance of a binomial = ? Var(X) = E[N/4] + Var(N/2) Var(X) = 1/4 * E[N} + 1/2 * Var(N)

Recall Var(X) = E[X-E[X]²] Definition: Conditional Variance of X given {Y=y} Var(X|Y=y) = Σp_x|y(x|y)(X-E[X|Y=y])²

Just like conditional expectation, we write Var(X|Y) to denote the random variable evaluated at Y.

Theorem: Law of total variance

Var(X) = E[Var(X|Y)] + Var(E[X|Y])

Proof:

Var(X) = E[X²] - (E[X])²

= E[E[X²|Y]] - (E[E[X|Y]])²

= E[Var(X|Y)] + E[E[X|Y]²] - (E[E[X|Y]])²

= E[Var(X|Y)] + Var(E[X|Y])

Now let’s start with Var(X|Y=y)

= E[X²|Y=y] - (E[X|Y=y])²

= E[Var(X|Y) + E(X|Y)²] - (E[E[X|Y]])²

Continuous Random Variables and Continuous Distributions

Note: Random Variables need not be discrete or continuous or combinations thereof For a random variable X, we can always define its cumulative distribution function abbreviated CDF via

F_x(X) := P{X<=x}

F_x is nondecreasing
F_x(x) -> {0 if x -> -inf, 1 if x -> inf}
F_x is continuous from the right

Definition: A random variable X has continuous distribution if there exists a function f_x such that

F_x(x) = ∫_-inf^xf_x(u)du

f_x is called the density of X aka a pdf(probability density function)

For f_x to be a density, it just needs to be non-negative and the integral of the density to be equal to 1, since total probability = 1 f_x >= 0 ∫f_xdx = 1

Continous Random Variables

Good for modeling analog signals from the real world.

time we wait until a bus arrives(Exponential Distribution)
Voltage across resistor(Gaussian Distribution)
Phase of a received wireless signal(Uniform Distribution)

Continuous distributions usually described by their density. X ~ Uniform(a,b) = {1/(b-a) a <= x <=b, 0 otherwise} X ~ Exp(λ) = {λe^-λx if x>=0, 0 otherwise} X ~ N(µ,σ²) = 1/sqrt(2pi*σ)exp(-(x-µ)² / 2σ²)

Jointly Continuous Random Variables

We say X₁, X₂, …, X_n are jointly continuous if F_{X₁X₂…X_n}(x₁x₂…x_n) = P{X₁<=x₁, X₂<=x₂, … X_n<=x_n}

= ∫∫ F_{X₁X₂…X_n}(x₁x₂…x_n)du₁du₂…du_n

Example: Let a dart land uniformly at random on a 2d dartboard of radius r. The joint density will be flat over the dartboard

{1/pi*r², x² + y²} Independence: Random Variables X,Y are independent if F_xy(x,y) = F_x(x) * F_y(x) If X,Y are continuous and independent, their joint density is the product of the marginals

Expectation of continuous random variables

Same as discrete case, replace sums with integrals E[X] = ∫x f_x(x)dx

More generally: E[g(X₁…X_n)] = ∫…∫g(x₁…x_n) f_{x₁ … x_n}(x₁ … x_n)dx₁ … dx_n

Example: Var(X) = ∫(X-E[X])² f_x(x)dx

Example: X ~ Unif(a,b) E[X] = ∫ uniform density function

= ∫ x * 1 / (b-a) dx

= 1/2 * (b² - a²) / (b-a)

= 1/2 * (b+a)

Example: Var(X) = E[X²] - (E[X])² = 1/2 (b-a)²

Let r = sqrt(x² + y²) Compute the probability that the dart is in the middle half of the dartboard.

P{R<= r/2} = P{x² + y² <= r² / 4}

= E[{x² + y² <= r² / 4}]

= 1/(pi * r²) ∫ {x² + y² <= r² / 4} dx dy

Gaussian Distribution, Derived Distributions, Continuous Bayes

Discrete: Expectations come from PMF and sums Continuous: Expectations come from PDF and integrals

Examples of continuous distributions

Uniform(a,b) Exp(N) N(µ, σ²)

Exponential Distribution

Suppose we want to model a memoryless process. For example, as a switch sending packets, it doesn’t matter how long you wait for a packet to show up, the expected wait time for the next packet doesn’t change.

Mathematically: Let X be a non-negative random variable with the memoryless property. Memoryless = P{X>t+s|X>s} = P{X>t} for all t,s > 0 If I’ve waited for s seconds, the probability P{X>t} is still unchanged

P({X>t+s} ∩ {x > s}) = P({x > t}) P({x > s}) Get rid of {x > s} since if x > t + s, x must be greater than s

F(t + s) = F(t) F(s) The only unique solution to the functional equality that is a CDF(must be non-negative, monotone increasing) of the form F_x = e^-λx

If x has memoryless property, F(X) = 1 - e^-λt for some λ > 0

Gaussian Random Variables

The sum of independent effects. Height of NBA players, satisfaction of jobs, sum of voltage across resistor.

We call N(0,1), the standard normal distribution.

CDF: 1/2pi ∫exp(-µ/2)²du

P(X<= x) = P((x-µ)/σ <= (x-µ)/σ) = Φ((x-µ)/σ) Gaussians have many nice properties

If X is gaussian, then so is aX + b. If X,Y are independent gaussians, then X+Y is gaussian

Example: Let V ~ N(1, 5²) be input voltage to chip, averaged over 1 second. Our chip fails if voltage dips below 0.5 volts or exceeds 2.5 volts for any one second period

Probabilty(chip fails in 60 second duration) <= 60 * Probability(chip fails in 1 second) 60 * Probability(chip fails in 1 second) = = 60 * Probability(V < 0.5) + Probability(V > 2.5) = Φ((0.5 - 1)/σ) + (1 - Φ((2.5-1)/σ))

Example: cellphone sends a bit B ∈ {-1, 1} to tower. Tower receives Y = B + N, N ~ N(0,1). Tower makes decisions B(Y) = sign(Y) P(err | B = 1)

Given that I receive {Y=y} what is the probability that B = 1 P({B = 1}) = P(Y ∈ [y, y + 𝛿] | B = 1) P(1)

= P_B|Y(1 | y) = P_B(1) * f_Y|B(y|1) / f_y(y) This is bayes rule for one

= (1/sqrt(2pi) * exp(-(y-1)²/2)) / (1/2 * 1/sqrt(2pi) * exp(-(y+1)²)/2) + … = 1 / (1 + e^-2y)

This is b Example: Last example motivates the definition of conditional density If X,Y are jointly continuous, we define conditional density of X given Y:

f_x|y(x|y) = f_xy(x,y) / f_y(y) interpretation: is the density of X given I know {Y=y}

Bayes rule f_x|y(x|y) = f_x(x) / f_y(y) * f_y|x(y|x)

Derived Distributions

Distribution of X give by CDF F_x. Suppose we define Y = g(x) for some function g

What is the distribution of Y?

Do you really need this distribution?
If you really need the distribution, it is best to work with CDFs

Example: X continuous random variable, Y = aX + b F_y(y) = P{aX <= y-b} = P{x <= (y-b) / a if a > 0, x >= (y-b)/a if a < 0} = {F_x((y-b)/a) if a > 0, 1 - F_x((y-b)/a) if a < 0}

f_y(y) = {1/a * F_x((y-b)/a) if a > 0, -1/a * F_x((y-b)/a) if a > 0} = 1/|a| * f_x((y-b)/a)

Y = AX + b f_y(y) = 1/|det(A)|f_x(A^-1(y-b))

Information Theory and Digital Communication, Capacity of the Binary Erasure Channel (BEC)

Modes of Convergence

Weak law of large numbers: Everyone in the room takes a coin and flips it 500 times. Then we can saw the empirical frequency is between .49 and .51. Then everyone flips it 500 more times. Then the empirical frequency is between .495 and .505. However, someone might have been between .49 and .51 in the first 500 but not .495 ad .505 in the first 1000 Strong law of large numbers: The empirical frequency will go to its true probability with probability 1.

CLT is a statement about convergence in a distribution

Information Theory

Source Coding Theorem

Protocol

If I observe (x₁, …, x_n), then I describe it via bitstring (1, xxx…x) -> log(A_t/2) If I observe a sequence not in the typical set, I describe it brute force via bitstring

What is the performance(average number of bits needed per symbol observed) for this scheme? 1/n * E[number of bits in representation] ≤ 1/n (2 + n(H(x)+E))P(x₁…x_n) + 1/n (2 + nlog(x)) * P(x₁…x_n) ≤ H(x) + E/2 + 4/n + log|x| * probability of sequence not being in typical set

Results

Descriptions using ≤ H(x) + E bits per symbol on average exist
Lossless descriptions using ≤ H(x) don’t exist
Huffman encoding: If I know p_x, I can design a Huffman code requiring ≤ H(x) + 1/n bits on average to compress sequences of length n

Question: How are we going to show we can compress down to the entropy? Answer: Use concentration. For a sequence(X₁, x₂, …, x_n), let the probability of observing it be p(x₁…x_n) = Π_i=1ⁿ P_x(x_i)

Theorem: Asymptotic Equipartition Property - If Xs are iid with respect to P_x, then -1/n log p(x₁…x_n) converges to H(x) in probability

Typical Set: For E > 0, for each n ≤ 1 define typical set A := { (x₁…x_n) : p(x₁…x_n) >= 2^{-n(H(x) + E)}}

Why is this called a typical set? It is a set of typical sequences.

Question: Suppose I have N objects. What is the max number of bits I need to represent each object? Answer: max of log(N) bits

Poisson Processes

Markov models: Map events that have memory Poisson Processes: Random arrivals: customers coming into a store, beta particles detected by geiger counter, vehicles passing a tollbooth Counting Processs: Starts at 0, right continuous, integer valued Poison Process: a counting process with Exp(λ) interarrival times

Thm: If N_t is a Poisson process with rate λ, then N_t ~ Poisson(λt) P{N_t = n} = e^-λt * (λt)^n / n!

Proof: P{N_t = n} = P{T_n <= t < T_n+1} = E[1_{{T_n} <= t} 1_{t < T_n+1}]

Increments of Poisson process are stationary: N_t+s - N_s = N_{t + tau} - N_tau

Poisson processes have independent increments: if t_{0_{< t_{1_{< … < t_{k_{=> increments N_t1 - N_t0, N_t2 - N_t1, … , N_t1 - N_t0 are independent}}}}}}

Thm: If N_t is a counting process with independent stationary increments and N_t ~ Poisson(λt), then N_t ~ Poisson Process(λ)

Conditional distribution of Arrivals Thm: Conditioned on {N_t = n}, (T₁, T₂, …, T_n) = (Unif(1), … Unif(n))

Example: Let cars pass through a tollbooth according to Poisson Process(λ). Question: What is the probability that no cars pass in 2 minutes P{N₂ = 0} = (λ2)^0 * e^(-2λ) / 0! = e^(-2λ)

Question: If 10 vehicles pass in 2 minutes, what is the distribution of vehicles that passed in the first 30 seconds? Binomial(10, 1/4)

Merging(superposition) and Splitting(thinning)

Merging: If N_1,t is poisson process with rate λ and N_2,t is an independent poisson process with rate u, then N_1,t + N_2,t ~ PP(λ+u)

Splitting: Independently mask arrival with 1. Example: Let vehicles pass through a tollbooth with PP(λ). Let 1/3 of the vehicles be cards and 2/3s be trucks

If 10 vehicles pass in the first 2 minutes, what is the distribution of cars that pass in the first 30 seconds?

Random Graphs and Inference

G ~ G(n,p): G is a graph with n verticies where each edge appears indepedently with probability p.

Monotone Graph Properties have sharp thresholds: P {G ~ G(n,p) has property P} = {0 if p < threshold n , 1 if p > threshold n} There is some phase transition phenomena.

Ex: p = {graph G has >= 1 edge} Monotone property. Claim: t(n) = 1/n^2 Proof: Let x = # of edges in G, p = c / n^2 What is the probability G has zero edges? P{G has zero edges} = P{X=0} = (1-p)^(n C 2) = e^(-c/2) This evaluates to basically 1 if c << 1 and 0 if c >> 1

Goal: For property p={graph G is connected}, show that t(n) = (log(n)/n) If lambda > 1, then P{G~G(n,p) is connected} -> 1 as n->infinity If lambda < 1, then P{G~G(n,p) is connected} -> 0 as n->infinity

Lemma: If X is a random variable, then P(X=0) <= Var(x)/(E[X])^2 Proof: Var(x) = E[X-E[X]]^2 = P(X=0) * E[X-(E[X])^2 | X=0] + P(X!=0) * E[(X-E[X])^2 | X=0] = P(X=0) * E[X]^2 + P(X!=0) * E[(X-E[X])^2 | X=0]

Case lambda < 1: Will show: P{G has isolated vertex} -> 1 {G has isolated vertex} C {G is disconnected} X = Σ Ii = # of isolated verticies, where Ii = {0 if i not isolated, 1 if i isolated} Ii ~ Bern(q) where qi := (1-p)^(n-1) Var(X) = Σ Var(Ii) + Σ Covar(Ii, Ij) = nq(1-q) + n(n-1)Cov(I1, I_2)

Where Cov(I1, I2) = E[I1I2] - (E[I])^2 P{G has no isolated vertices} = P{X=0} <= (nq(1-q) + n(n-1) * pnq^2/(1-p))/(n^2q^2) <= (1-q)/nq + p/(1-p) (1-q)/nq goes to zero since nq -> infinity p/(1-p) goes to zero as p -> 0

P{G disconnected} = P{U_k=1^{n/2} {There exists a set of k vertices separated from the rest of G}} Apply union bound again P{(n C k) P {vertices 1, …, }}

Inference

Given data, how do I choose the model that generated the data X is the state of nature, often parameter or hypothesis -> P_{Y|X} -> Y produces some observation(data) X may or may not be the random variable If it is, then the distribution of X is called a prior(Bayesian Inference)

Inference

Goal: Infer X from Y X -> Model -> Y

Reasonable Approaches:

MLE(X|Y) = argmax P(Y|X)
Binary Hypothesis testing. X(Y) = 1 if L(Y) > lambda, 0 if L(Y) < lambda

For any test X:Y, there are two fundamental error rates, false positives and false negatives

False Positive: P{X(Y) = 1 | X = 0}

False Negative: P{X(Y) = 0 | X = 1}

Question: Given a constraint on type 1 error rate, how do we find the test that minimizes type II error rate? Solution: Solve X = argmin P{X(Y) = 0| X=1} Answer: Randomized threshold tests are optimal: Neyman-Pearson Theorem Given Beta, the optimal decision rule is X(Y) = 1 if L(Y) > lambda, 0 if L(Y) < lambda, Bernoulli(gamma) if L(Y) = lambda Where lambda and gamma are chosen such that P{X(Y) = 1 | X=0} = beta

B = P(L(Y) >= lambda | X=0) = P(Y/sigma >= 1/(2sigma) + philog(lambda|X=0)) = 1 - phi(1/(2sigma) + sigmalog(lambda))

Question: Where does randomization enter the picture Answer: Deterministic threshold tests define and lie on error curve, but don’t necessarily sweep the whole thing Ex: If Y is discrete, say binary-valued, then L(Y) takes at most 2 values Takes two tests and randomizes between them

Threshold tests are optimal

Estimation under mean squared error Goal: Estimate X based on Y under some loss function X -> Model -> Y

Intro to Networking

Thu, 25 Aug 2022 00:00:00 GMT

Lecture 1 - Intro to the internet, protocol, architecture

What is the internet?

Goal: transfer data between end hosts
Infra: TCP, IP, DNS, etc.
Applications: facebook, google, twitter. <- don’t give a shit about this in class
End hosts: laptops, phones, pacemakers
Switch: help end hosts communicate. ie router, phone on hotspot
ISP: aws, at&t, UC Berkeley

Protocol: specification of messages that communicating entities exchangeSpecifications

Useful for getting someone on board
If you design the protocol poorly you will spend lots of time debugging

Architecture

Network vs the Internet- How do you interconnecy
What ifno should googogle share via amazon

Protocol Design Factors

Incentives for ISPs, topology, path selection, diagnostics, customers
Enormous diversity and dynamic range with respect to parameters(packet loss, bandwidth, communication latency, types of endpoints, different users)
How do you differentiate whe interoperability relies on a common protocol?
Upgrading the internet not an option
Speed of light => cannot go faster from berkeley to NY than 27.5 milliseconds
Portions of the internet fail(software, switches, links, network interfaces)

The protocol design paradigm for the internet

Decentralized control: Software defined networking(SDN), dsdn
Best effort service model
Route around trouble
Dumb infra with smart endhosts: NFV: Richer in network services
e2e design principle: edge computing
layering
federation via narrow waist interface

How to prioritize goals? What are the fundamental constraint? How to decompose a problem?

Best Effort: Packet Switching, just send packets and hope for the best Reservations: Circuit switching, end hosts reserve bandwidth when needed

Which is better? Circuit vs Packet Switching?

Factors to consider:

Abstraction to applications
Efficiency(at scale)
Handling Failures(at scale)
Complexity of implementation(at scale)

As an application developer, circuits offer better application performance due to reserved bandwidth and is more predictable and understandable. This is also very intuitive in support of business models(ie charge based on use). However,

Smooth vs Bursty Applications

Definition: Difference between peak and avg transmission rate
Small speak to avg ratio(3:1) - ie voice
Bursty ratio(100:1) - Ie Data applications, web browsing
Phone network uses reservations, internet does not
Circuit switching spends some time to setup/teardown circuits, inefficient when you don’t have much data to send

Failures

Circuit Switching

Network must do all the things needed for packet switching
Endhosts must detect failure, teardown old reservations, send new reservation request
All endhosts must do this

Packet Switching

Link goes down, then what
Network detects failures, recalculates routes
Transient overload: When a point in time when you receive two packets at the same time

Resource Reservation Protocol

Packet switching is the default
Used to setup state
You can also buy a dedicated circuit: ie MPLS circuits, leased lines,
- Often used by enterprises from one branch location to another
- very expensive(10x higher than normal connection)
- statically set up, long lived, and per user(not per flow)

Circuit vs Packet Switching a History

1970s-80s used packet switching, well suited to bursty file transfer applications
1980s-90s believed we’d need circuit switching. Envisioned that voice/live TV would be the Internet’s true killer app

Life of a Packet

Source has data to send to destination
Chun it into packets: each packet has payload and header
Packet travels along a link
Arrives at a switch, switch forwards to next hop(or buffer or drop packet)
Repeat last step until we reach destination or packet is dropped

Designing the Internet Top Down

How to solve a problem?

Define the problem
Decompose the problem - Modularity(barbara liskov ted talk)
- Decomposing system into smaller units, providing a separation of concerns
- Motivation: Musk wants to send bezos a hand written letter. The letter goes from Musk -> aide -> fedex -> fedex -> aide -> Bezos.
- Each level understands the other people on their level
Assign tasks to entities

Internet Modularization

Lowest Layer of Abstraction
Physical transfer of bits
Best effort local packet delivery
Best effort global packet delivery
Reliable/Unreliable data delivery <- Layer where we care about packets
Applications
Highest Layer of abstraction

Local vs Global Delivery

Ethernet networks only speak to Ethernet networks Optical Transport networks only speaks to optical networks.

What is a Protocol?

An agreement between parties on how to communicate
Defines syntax and semantics
Exist at multiple layers
Each layer of the stack has multiple protocols except the network layer, which only has IP
- Each layer depends on the layer below, supports the layer above, and is independent of the same layer protocols

Why Layering?

Innovation can proceed in parallel #OpenInnovation
Different communities can innovate, ie L7(app devs) vs L1(chip designers)
The end host must implement all layers
The network must implement L1-L3, L4+ is for reliability and reliable data transport
Applications think about data, nic/driver cares about packets, network stack translates between these two
Ethernet doesn’t know how to talk to OTN, Ethernet goes to IP which sends it to OTN which sends it to other OTN which sends it to IP which sends it to Ethernet

Layer Encapsulation

Take data, add app Header(ie HTTP)
Add transport header(ie port number)
Add Network header(ie host address, ip)
Add L2 header(ie ethernet address)
Now your packet should be on wire and have a header from all layers!

Architectural Wisdom

End to End arguments in system Design
The Design Philosphy of the DARPA internet protocols

End to End Principle

Guides the debate about what functionality the network does or doesn’t implement
Should we implement reliability in the network?
Solution 1: Implement reliability at each step(network must implement reliability)
- Problem: Cannot be perfectly reliable. What happens if a component has a bug or fails between two steps.
Solution 2: End to end check and retry(does not assume network is reliable)
Sometimes you want reliability in the network to increase performance

Questions?
Difference between packet switching and circuit switching?
How do we do reliable packet delivery?

Borrowing Protocols

Thu, 30 Dec 2021 23:46:37 GMT

Asset Price Movements and Liquidations

Synthetic Asset protocols ie DAI, Mirror, Synthetix, Yeti

Needs oracle that provides prices of collateral asset
Can mint new DAI when O_t * a * c >= b where O_t is the oracle price, a is the amount of collateral, c is the collateral factor(usually less than 1), b is the amount of synthetic tokens
If the oracle updates and cause the above invariant not to hold, a liquidation of the position is allowed(This is very bad for the person who took out their position!!)

For the multi-collateral case, Ot, a, c, b are all vectors and the following invariant is used `sum over collaterals(Ot * a * c) >= b`

If you have a mixture of two assets(ie 50% eth and 50% bitcoin), as long as the oracle prices are not perfectly correlated, there is a smaller chance that the collaterals move in a way to get you liquidated.

Arguments for not doing this?

Might be hard to set the parameters for O_t, a, c, b, but gauntlet already does this for collateral ratios on compound and aave

Why is everything correlated?

Things are empirically correlated and the why doesn’t really matter.

People who invest in an asset class want to be diversified, if the bitcoin in their portfolio goes up, they want to rebalance to stay diversified so they sell part of their bitcoin and buy eth, causing the eth price to go up.

Risk Modeling for Protocols with Liquidations

Example

Max Loan Size Supply vector x collateral ratio * prices Asset | Collateral factor ETH | .825 DAI | .825 WBTC | .7

Mechanics of Liquidation

When liquidation is allowed, liquidators repay the borrowed amount and gets a portion of the collateral + some bonus. THe liquidator immediately market dumps the collateral + bonus.

There’s a routing library in Julia that Bain cap wrote(doesn’t support curve/univ3)

Cascading Liquidation Feedback Loop

Liquidations -> Market Dump -> Price Drop -> More Liquidation

Algorithm:

Input current state of lending markets, exchanges, prices While True: Check for liquidations If liquidations: Route them over exchange graph, update exchange states, update prices to post dump prices

Why? Allows us to calculate structural component expected loss vector and variance covariance matrix based on actual state of the chain Allows us to better understand price trajectories. Can sell to people who care

Questions

What’s the intuition behind multi-collateral synthetics being less likely to liquidate? What about non-stablecoins?

Cosmos

Thu, 30 Dec 2021 23:46:37 GMT

Cosmos

Most blockchains have:

Shared execution layer
Smart contracts

What do I mean when I say Cosmos?

Not Cosmos Hub
Not $ATOM
Cosmos SDK: Tendermint Consensus + IBC Modules + other stuff

Philosophy

Instead of shared execution layer, you want application specific chains.
Benefit 1: You have control
- Precompiles - code that denotes complicated computations cheaper on chain. If you don’t have a precompile, you need to use the evm. E.g. cryptography. Defined in clients Cacn technically use .com domain as ENS, but this is extremely expensive to do DNSSEC onchain since cryptography isn’t on chain. Might cost $1000 some money to verify
- Novel consensus/Protocol Design. Osmosis wants to prevent MEV from happening; they want to use threshold encryption to do so. Thus, Osmosis can embed this in their chain
Benefit 2: Blockspace Monopoly
- Vertical Scaling brings problems like state bloat, computation power. Solana drops Merkle trees from client design to improve L1 scaling.
- Cosmos Approach: Instead of scaling vertically, you can create your own blockchain and scale horizontally. If each application has their own separate throughput, Polygon doesn’t get rekt bc people are playing sunflower games.

IBC

relies on instant finality
TCP/IP - not a bridge
easy to add Cosmos based blockchains and other blockchains with {easy way to create light client proofs, Relayer}
peg zone
Relayer: trustless way to send data from one blockchain to another. Think of it as a cable that can transmit data from one chain to another. Security comes from the light client proof, relayers have no verification. End Goal: Each wallet is its own relayer. Relayers also get transaction fees in the future. Currently, relayers are subsidized via the Interchain foundation(Ethereum foundation equivalent)
No flash loans, flash mints, flash anything via IBC bc that requires atomicity.
Can do token transfers, cross chain swaps

IBC Research

Quick notes

Bc of the blockchain trilemma{decentralization, scalability, security}, applications will use multiple blockchains to achieve the trilemma
To communicate between blockchains
Deploy contract on Ethereum: meant to communicate with validators
Send package to validators
Validators communicate with solana blockchain Eg If transferring USDT, you lock USDT on ethereum contract and mint synthetic USDT on solana blockchain Validator: Who runs the validators between chains?

IBC

Handle Authentication, ordering data packets between blockchains, can use
Recall above, the package needs to be validated
Do you need to use IBC + a bridge? Mental model: IBC is a validator. (Who runs the validator?) Goal: Manage communication between two modules. This involves sending/receiving packets with establish connection

How IBC works

Properties of IBC: Data confidentiality + Legibility: Send as little data to validator as possible + validator needs to understand the data
Reliability: Relayer + blockchain works independently, there is a chance of dropping packets. This is managed by a sequence number per packet.
Flow Control: Depends on ledger, IBC does not have overriding rule
Authentication: Whenever IBC is to be used, data must be first on the originating blockchain. - IBC handler(validator?) verifies the data
Statefulness: Three parts - clients, connection, channel
Clients: info about state of both blockchains connection: identifiers/data structures agreed by both blockchains
Channel: info about encoding info, state, sequence numbers
Multiplexing: smart contracts can use single IBC connection. Generally one to one but IBC supports one to many and many to one
Relayer - keeps watch of state chain from origin chain(ie Ethereum in example above)

Use Case

Why switch networks when purchasing insurance on other chain?

If vault and pool are connected with IBC, user can buy insurance from any chain while spending from mainnet wallet.

Questions?

Confirmation time is 15 seconds on ETH, Bitcoin is 10 minutes, can we still use IBC? Need to wait 10 minutes?

What about reorgs? There’s never true finality on proof of work chains right? Solved via validators?

Learning - Learn more about cosmos Document - Publish technical articles, explainer articles, based on learning Buidling - play with cosmos ibc, a bit tough scotty doesn’t believe in us :(, Spinning up dapp chain, running a relayer

Questions

Subnets vs Cosmos

There is an expectation that you share security with Avalanche. Problem is that shared security isn’t free, you need to pay avalanche validators for this.
Tendermint is novel: instant finality + 2/3 network validators needs to be aligned. Instant finality is huge because it simplifies multichain consensus. In a proof of work blockchain, by the virtue of a 51% attack, blocks are not final since you can end up with uncle blocksreorgs
- This is bad for bridges because a transaction where you burn a token to mint it via a bridge to a different chain can be reverted due to a reorg.
- bridge UX gets killed
- Multihop dexes can take a very long time because you need to wait for probabilistic finality.
Are there Oracles natively in Cosmos?
- No current oracles, but can built oracle chain <— big brain project right here

Fixed Yield

Thu, 30 Dec 2021 23:46:37 GMT

Fixed Yield Notes

Zero coupon bond: Worth 1 dollar at some point in the future. It is called a zero coupon bonds because there is no discount, you can see the fixed income rate by looking at how many zero coupon bonds you can buy.

If you can buy 102 zero coupon bonds that are redeemable in one year for 100 dollars, your risk free rate after 1 year is 2% Equivalent to zero coupon bond

Why Fixed Income?

Variable yields means uncertainty with to what your portfolio value will be in the future
Fixed income is powerful when applied over multiple rates over a long time period. The randomness of the rates leads to a log normal distribution with a Fat tail, and nothing in this fat right tail can be used by institution.
- ie pension fund only needs to pay back pensions, the long tail is not useful if they have already achieved the returns. Thus, I can sell this long tail and get some more guarenteed returns
- Look into central limit theorem for how the distribution is generated

End Users

Institutions, people who want higher expected yields
Institutions observe the price of the zero coupon bond
Financial engineering to get yield + principal tokens(see element finance)
Institutions buy the zero coupon bond at a certain price. They wait 1 year or however long then redeem it

There must be some people who are willing to take on additional risk to get higher expected returns Will likely be complete overlap between fixed income product and people that buy protection

How does the product work?

Take yield bearing token aUST(anchor UST) and split it into paUST(Fixed Yield) and yaUST(Random yield). The paUST is always worth 1 dollar at expiration. The yaUST’s value depends on what the Anchor yields look like between creation and expiration. Higher anchor yields = higher yaUST value

Users can deposit UST
Get aUST
Deposit aUST into fixed income vault
Protocol splits aUST into paUST(redeems to 1 UST at expiration) and yaUST(variable yield at expiration)

If you want to get fixed income, sell your yaUST and buy paUST in a DEX If you want higher expected yields, sell your paUST and buy yaUST in a DEX

Fixed Income AMMs

How do we design an AMM for a fixed yield product?

As the maturity approaches, the principal get closer and closer to a dollar since when it matures, you can redeem it for a dollar.
Interest rates tend to be more stable than asset prices, thus the curve should be flatter than x * y = k

Yield space AMM

x^(1-t)+y^(1-t)=k Can chop off half of reserves since zero coupon bonds are always cheaper

can shift the decay rate to non-linear decay rate
if there are multiple competing zero coupon bonds(same asset but different)
zero coupon bond on dai, zero coupon bond on usdc, zero coupon bond on usdt deposited into stableswap AMM, then take the LP token and put that into our x^(1-t)+y^(1-t)=k AMM
For multiple maturities, you can create a nested AMM structure to trade between 3 month, 6 month, and 9 month maturities
- create a 3 month, UST Pool, then take the {3 month, UST} LP token and pair it with the 6 month principal token {6 month, {3 month, UST}}, then take the {6 month, {3 month, UST}} LP token and pair it with the 9 month principal token to create a 3 layer nested AMM {9 month, {6 month, {3 month, UST}}}
- Wouldn’t this have less liquidity for later expiries?

Questions?

How do we solve the fungibility problem? Treat them as different assets and add a market maker between them. As time to maturity approaches, the market maker should shift from a constant product market maker(x * y = k) to a stableswap (x^3 + y^3 = k) Can also do nested AMMs
If I buy now and the expiration period is 6 months from now and my friend buys tomorrow and the expiration period is 6 months from tomorrow, how do we manage the UX? All 6 month fixed yields expire on the same day

Mirror Protocol

Thu, 30 Dec 2021 23:46:37 GMT

Mirror

Overcollateralized synthetic asset Deposit bluna, luna, ust Mint some index(ie price of oil, mTesla)

Mirror Risks

Governance Risk
- Create formal governance proposal, hide sus code in it, hope it passes before people realize(Same think that rekt beanstalk finance)
- Mirror v2 has a system of incentives to achieve quorum
Oracle Risks
- S&P 500 oracle gave incorrect price and a large position was liquidated(250-500k)
Structural/Economic Risk

Geometric Brownian Motion

Problem: continuous in time but users don’t act continuous in time but rather per block. The agent can continue to add more collateral as the price moves down to prevent liquidations.

Jump Process

Poisson point process: gives the arrival of the jumps
- The model of the arrival in each independent event. The probability that a certain event arrives within a given interval is poisson distributed. Modeled by probability mass distribution, lambda tells you how often the jump happens on average.
Jumps: the prices at each jump

Model

Trader mints mS&P by supplying bLUNA as collateral, they are required to supply at least collateral amount >= collateral factor* mint AMount * collateral price / synthetic price
If no jumps occur in interval, no risk of liquidation because no price changes
If one jump occurs, check if the jump was less than threshold.
If two jumps occur, check if the first jump was less than threshold.
If n jumps occur, need to check if any jump was less than threshold

Questions

1.How does risk harbor provide protection against Mirror Liquidations?

MEV

Thu, 30 Dec 2021 23:46:37 GMT

Broke: MEV is consensus mechanism that is the source of issues Woke: Problem is from application layer, this is similar to the issues in the HFT space.

People try to randomize arrival of orders, make fees, etc. There was a good paper that prompted people to use batch auctions instead of central limit order books CLOB: orders come in serial time, processed serially in order of arrival. This is the same as the transactions ordering in ethereum, whoever comes in first gets executed first. Batch Auction: Instead of processing everything in continuous time one by one, wait and batch transactions and process them together. - For all transaction that came in this block, store them and execute them next block

To submit an order on a dex, you submit an order and a slippage tolerance. If the slippage tolerance is too lenient, you can get front run for up to the slippage tolerance.

This leads to people looking elsewhere for execution markets, ie flashbots. Since flashbots only has a certain percent of mining power, your order will execute slower than if you put your transaction in the standard pool.

Example:

Bob wants to buy 100 ETH. Order size +100 ETH
Joe wants to sell 100 ETH. Order size 0 ETH
Smitty wants to buy 10 ETH. Order size +10 ETH The batch order of buy 10 ETH would then be submitted

Overview of MEV

Frontrunning
arbitrage
Liquidations
Generalized Frontrunning: replacing profitable tx with

Liquidations

Whoever gets the liquidation first gets the money
Liquidate position, market dump tokens

Options

Thu, 30 Dec 2021 23:46:37 GMT

Options

Why do we need a characteristic function and how does it relate to Options? Options + probability are deeply related. If you know characteristic function of the option, you can compute the option price.

Benefit of characteristic functions: Allows for more flexibility in pricing risk, volatility. Increases flexibility with regards to pricing. “Correlation structure is easier to work with in characteristic function” - Prof Resnick

Characteristic function

Motivation: This stuff can be calculated on chain using Fast Fourier Transform if we have a characteristic function Characteristic function: Alternative way to express info about random variable

Recall cumulative density function (Probability that random variable is less than a given value) E[e^(itX)] - corresponds to the fourier transform, which is a good way to model things

Can express characteristic function as sum of the moments? E[X^n] term ~ price of a power future with power n. The price of a power perpetual implies something? <- not sure what it implies

Sum of two random variables => product of characteristic functions ^ What’s the use case here?

Delta is useful for market makers - If we can compute delta easily, we can check things on chain? ^Spoiler alert, we can using fast fourier transform

Two sides: uninformed flow + market makers on options(PhD physicist/math/stats)

If you are little boi and make options flow and sell via black scholes, jump is gonna rek you with diffusive, lattice, monte carlo bullshit.
When you price something wrong, they can arb you immediately and this is hard to fix since you wrote this in a smart contract
This makes on chain options difficulty

Questions?

What will be computed on chain and what will be computed offchain? We should implement characteristic function as simpler function on chain using fft(divide and conquer algo- O(nlog(n))
What is N in O(N log(N))? Can be tuned, higher it is better the approximation, similar to resolution.
What is an FFT? https://www.youtube.com/watch?v=spUNpyF58BY Thanks matt :D
What other protocols have power futures? Squeeth, Tracer, https://zeta.markets/. The later two aren’t really power futures though, they are just multipliers right?
Are these markets priced by spot? Do they reflect alpha in spot? Squeeth is underfunded, so people buy squeeth to hedge volatility future. Selling squeeth gives you positive EV, not enough liquidity in market for other side. No one wants to sell volatility. If there was a liquid options market, then you can hedge squeeth. In theory you can buy options and sell squeeth, and volatility, skewness hedged and set as options market maker.

Quanto Derivatives

Derivative that is settled in a different reference currency.

Anchor protection settled in usdc vs anchor protection settled in ust Why do the premiums cost a different amount.

Why isn’t there another instrument that could close the gap in premiums between these products? You are paying for more protection

When the euro collapsed, greece, italy, spain defaulted. Two types of cds, one denominated in euros and one denominated in usd. The usd premiums are much more expensive than the cds premiums denominated in euros because if greece defaulted on their loans, the euro is more likely to collapse as well

Effect of lower price of euros

Oracles

Thu, 30 Dec 2021 23:46:37 GMT

Motivation

From the previous stablecoins post, Ampleforth’s AMPL is one example of a stablecoin that requires an oracle to properly function. Another example is flight insurance, how does the blockchain know when a flight has been delayed? Some other DeFi applications that need data include sports betting, wrapping other cryptocurrencies(bitcoin), synthetic assets(stocks), and undercollateralized lending.

The blockchain does not have a connection to the internet: How do we get external data to contracts?

Building an Oracle

Basic Oracle Design

Build an oracle into the consensus protocol.

What if the miner lies or cheats?
- Use multiple miners and use the data of a randomly selected miner.
What if the sending node lies or cheats?
- Use digital signatures and take the majority answer.
What if the sending node goes down?
- Allow for backup transmission.
What if the web site with data goes down?
- Use multiple websites.

This results in a decentralized oracle with good liveness, what if there are malicious attackers in the system?

What if nodes disagree?
- Take the majority value
What if nodes report different numerical values?
- Use the median. Given a minority of bad values, a median is an honest value or bounded by honest values.

Remaining Problems

These problems are usually solved at the protocol level, so solutions will differ based on the specific implementation.

How do we ensure that a majority of nodes are honest?
- How do we ensure nodes get paid for their service?
- How do we ensure that oracle reports are mined by miners in a timely way?

Implementations of Oracles

Decentralized Exchanges (DEXes)

Decentralized Exchanges enable on chain trading of various asset pairs. As a result, the exchanges have the current valuation of the asset price, since mispricing the asset would result in an arbitrage opportunity. Thus, we can use the decentralized exchange price as the oracle price!

Pros: Instant response and composable with other contracts! Also, there is an economic assurance that the oracle is correct. Cons: Only price information can be returned and the oracle can be manipulated.

Decentralized Exchange Price Attack

Situation: bZx, a lending protocol, used the Kyber exchange as a price oracle. Their ETH loans were backed with sUSD collateral based on Kyber price. Attack:

The attacker sold ETH for sUSD on Kyber to drive down the ETH/sUSD price
The attacker borrowed ETH for sUSD cheaply on bZx since the oracle reported that ETH is worth a lot less sUSD than it once was.
The attacker ran away with the loaned eth, making 2381 ETH(673k USD)

Countermeasure: Use time weighted average oracles(TWAP) so attackers can’t drive down the price for any significant periods of time.

Verifiable Random Functions (VRFs)

Problem: Since everything is public on the blockchain and we can’t introduce any external data without an oracle, there is no randomness since we could simulate what will happen on chain by reading the mempool and running the transactions on a local node.

Idea: The oracle uses a secret key to generate randomness. The randomness is verifiable and unpredictable.

Oracle Privacy

Problem: Alice wants to buy flight insurance for her trip from San Francisco to New York, but how can she do that without everyone knowing that she’s flying from San Francisco to New York?

Partial Solution: Use a trusted execution environment. This ensures integrity, as no one, not even the owner of the computer with the trusted execution environment can tamper with an application executing in it. This also ensures confidentiality, as no one, not even the owner of the computer with the trusted execution environment can learn about the data running in the environment.

Problem: Hardware based trusted execution environments have serious vulnerabilities. ie Intel SGX Solution: Use a DECO, a trusted execution environment alternative built using cryptographic techniques!

Remaining Questions

Why do we first learn about the mean in school then the median? Or at least this is what I learned in elementary school.
How does a DECO work?

An EVM developer's guide to Solana

Thu, 30 Dec 2021 23:46:37 GMT

Solana

What are the differences between EVM and solana

Bitcoin is based on UTXO, Ethereum uses the accounts model
Solana also uses the accounts model

EVM accounts vs Solana

Ethereum/EVM uses Externally Owned Accounts + Smart contract
nonce, balance, codehash, storage root(256 bit hash of root node of merkle tree that encodes storage)
Solana only has 1 type of account Subtypes are data accounts and program accounts
Lamports = wei
owner = program owner, default is owned by system
executable = whether this account can process instructions(true = program account, false = data account)
data = raw data byte array stored by account
rent_epoch = Next epoch that this account will owe rent(If you run out of rent, the account gets purged) Program accounts only store logic in data, not state - which is stored in a separate account owned by this account

Special type of accounts in Solana: PDAs

A smart contract in solana needs to store state outside of program, they don’t want to store this using the same ed2559 elliptic curve to avoid collisions and someone getting the private key to this program in the future, so it uses a separate curve

Transactions

EVM has two types of transactions, sending ether vs calling smart contract

can only call single function

Solana allows you to call multiple instructions in the same program Need to pass all accounts that would be part of the transactions

Example: how tokens work

Ethereum

I have 10 USDC, in the USDC contract, the mapping called _balances
- mapping(address => uint256) private _balances
- _balances[my address] = 10

Solana

Because the state of a program is just some logic, every token can reuse the same token program logic
image.png

Random Solidity Notes

Thu, 30 Dec 2021 23:46:37 GMT

Solidity 101 Notes

A ‘^’ symbol prefixed to x.y.z in the pragma indicates that the source file may be compiled only from versions starting with x.y.z until x.(y+1).z. For e.g., “pragma solidity ^0.8.3;” indicates that source file may be compiled with compiler version starting from 0.8.3 until any 0.8.z but not 0.9.z. This is known as a “floating pragma.”
NatSpec Comments: NatSpec stands for “Ethereum Natural Language Specification Format.” These are written with a triple slash (///) or a double asterisk block(/** … */) directly above function declarations or statements to generate documentation in JSON format for developers and end-users. It is recommended that Solidity contracts are fully annotated using NatSpec for all public interfaces (everything in the ABI). These comments contain different types of tags:

@title: A title that should describe the contract/interface

@author: The name of the author (for contract, interface)

@notice: Explain to an end user what this does (for contract, interface, function, public state variable, event)

@dev: Explain to a developer any extra details (for contract, interface, function, state variable, event)

@param: Documents a parameter (just like in doxygen) and must be followed by parameter name (for function, event)

@return: Documents the return variables of a contract’s function (function, public state variable)

@inheritdoc: Copies all missing tags from the base function and must be followed by the contract name (for function, public state variable)

@custom…: Custom tag, semantics is application-defined (for everywhere)

Stablecoins

Thu, 30 Dec 2021 23:46:37 GMT

Variables to account for in DeFi/Stablecoins

Interest Rates in CeFi and DeFi
Cryptocurrency prices
Collateral Ratio
DeFi Protocol Fees
Blockchain Transaction fees
Number of users
Blockchain transaction throughput

Exogenous: A variable determined outside the model that is imposed on the model. In MakerDao’s DAI, the asset price of the collateral(ETH/USDC) is independent of the MakerDAO system.

Endogenous Variable: A variable determined by the model. In Synthetix, arbitrary derivatives are backed by SNX(a project internal token)

Fractional Reserve: When a bank only has a fraction of its deposits backed by cash and available for withdrawal. A bank’s reserve ratio = Money held by the bank / Money deposited by bank customers

What is Stability?

Stability is a relative metric. USD is stable compared to bitcoin but the price still fluctuates with respect to other assets.
We can measure volatility, the standard deviation of returns, and the worst drop over a timeframe. Some fiat, for example euros or pounds, have a volatility of 6-12%. However, volatility alone does not capture the range of prices.

Stable vs Pegged Coins

Stablecoins include USDC, USDT, DAI, UST, etc. These coins are simply USD derivatives and are usually reserve based, collateral based, or algorithmic.
Pegged coins include wBTC, renBTC, sETH, stETH. They are pegged to the price of an asset, usually via the same reserved based, collateral based, or algorithmic strategies used by stablecoins.

How do Stablecoins work?

MakerDAO’s DAI

The minter deposits ETH to open a collateralized debt position (CDP) where 150% collateral mints 100% debt. The collateral in this case could be ETH, and the debt create is represented as the DAI Stablecoin.
The minter can withdraw DAI to use for anything they want.
To get their deposited ETH back, the minter needs to pay back the same amount of DAI they withdrew to unlock their ETH.

Synthetix sUSD

The minter deposits SNX to mint a derivative. 600% collateral mints 100% derivative. The collateral in this case is always SNX, and the derivative could be something like sUSD or even something like sTSLA shares.
The minter can withdraw their derivative. In the case of sTSLA, price oracles report the external prices to the blockchain, which entitles the derivative to a portion of the collateral based on the current price of TSLA shares.
The minter can pay back their sTSLA to unlock SNX

Ampleforth (AMPL)

Ampleforth has no collateral, it is an algorithmic stablecoin The price oracle constantly updates the AMPL/USD price. There are three cases the oracle reports:

If each AMPL is worth more than 1 USD, wallet balances that hold ample increase. For example, if you had one AMPL in your wallet and each AMPL is now worth 2 USD, you now have two AMPL in your wallet. This is known as expansion.
If each AMPL is worth less than 1 USD, wallet balances that hold AMPL decrease. For example, if you had one AMPL in your wallet and each AMPL is now worth .5 USD, you now have .5 AMPL in your wallet. This is known as contraction.
If each AMPL is worth exactly 1 USD, no action is taken. This is known as equilibrium.

Remaining Questions

Even though AMPL is 1:1 USD, why is it worth anything? It isn’t backed by collateral like DAI/USDC so what can you do with AMPL? How is AMPL minted?

The Merge

Thu, 30 Dec 2021 23:46:37 GMT

The Merge

Transition from Proof of Work to Proof of stake

Proof of Work

Looking for some amount of leading zeroes, aka your hash should be less than some value

Proof of Stake

Blocks constructed by leaders proposes block to committee, committee confirms the block
Beacon chain: currently running in parallel to main POW chain and holds staking balances for ETH POS
Will be merged into the main chain

Sharding

Sharding splits a database into separate shards so nodes don’t have to store full database
Shards initially will only be a data layer, doesn’t support execution
Sharding can be used for execution via ZK rollups bc only proof needs to be stored

Difficulty Bomb

Hardcoded in eth, increases difficulty for block production, making it virtually impossible for miners to produce blocks
Will hit when the merge takes place via the bellatrix hard fork and POS validators take over
Shards would launch later

Ropsten Drama

Difficulty Bomb hits when the cumulative difficulty reaches a certain point. On ropsten, someone brought a ton of hash power to increase cumulative difficulty which made the difficulty bomb hit early
Beacon chain wasn’t yet there, normal chain can’t produce blocks bc difficulty bomb
On May 26th, there was a 7 block reorg. Reorgs are bad because they break many security assumptions on dapps, like oracles. Crypto Twitter suspects that this is due to people using different clients, some outdated, some current

Consecutive Block Production By Leaders

In POW, it’s extremely unlikely for two miners to form more than one block in a row without a significant portion of the network hash power(selfish mining)
Leaders in proof of stake would know that they are going to form a block in advance, so they would know when they could form two blocks in a row, and thus cause a reorg

File Systems

Sat, 20 Nov 2021 23:46:37 GMT

What does a storage system need?

Reliability: User data should be safely stored even if a machine’s power is turned off or the OS crashes.
Large Capacity and Low Cost It takes 350 MB for an hour of music, 1 GB for 300 photos, and 4GB to store an hour long video. Many individuals own 1 TB or more of storage for personal files.
High performance: Programs must have quick access to data to satiate users(ie computer starting up, youtube serving video, amazon processing orders)
Named data: Data must be organized for future retrieval, the easiest way is to name the file.
Controlled Sharing: Users want to share stored data, but only with the right people.

How does a file system help?

Non-volatile storage is much slower than DRAM, also, access must be done in coarse grained units, 512 bytes or more at a time.

Goal: High performance
- Physical Characteristic: Large cost to initiate I/O access
- Organize data placement with files, directories, the free space bitmap(free map), and placement heuristics so that storage is accessed in large sequential units
- Use Caching to avoid accessing persistent storage
Goal: Named data
- Physical Characteristic: Storage has large capacity, survive crashes, and is shared across programs
- Support files and directories with meaningful names
Goal: Controlled Sharing
- Physical Characteristic: Device stores many users’ data
- Include access control metadata with files
Goal: Reliable Storage
- Physical Characteristic: Crashes can occur during updates. Storage devices can fail. Flash memory cells can wear out
- Use transactions to make a set of updates atomic
- Use redundancy to detect and correct failures
- Move data to different storage locations to evenly wear out the disk

Why is it important to know how file systems work even if I’m not building a file system?

Performance: Even though file systems allow existing bytes in a file to be overwritten, inserting new bytes may require rewriting the entire file. Thus, autosaving may take as much as a second.
Corrupt Files When overwriting the existing file with updated data, an untimely crash can leave the file in an inconsistent state, containing a combination of old and new versions.
Lost files If instead of overwriting the document file, the applications writes to a new file, then deletes the original file, then moves the new file to the original file location, an untimely crash can leave the system with no copies of the document at all.

The File System Abstraction

An OS abstraction that provides persistent named data.

File: named collection of data in a file system. It is an arbitrary size, consisting of metadata and data. From the POV of the file system, a file’s data is just an array of untyped bytes.
Directory: Provides names for files. Contains a list of human readable names and a mapping from each name to a specific file or directory
Hard link: Mapping between a name and the underlying file
Soft link: Mapping from a file name to another file name. Unfortunately, this can cause dangling links: ie you link /a to point to /b which points to hi.txt then you unlink /b, /a will dangle)
Volume: A collection of physical storage resources that form a logical storage device. This is usually an abstraction that corresponds to a logical disk.
Mounting: Create a mapping from some path in the existing file system to the root directory of the mounted volume’s file path. Used when plugging in USB drive to your computer

The File System API

Creating and Deleting Files

Create() creates a new file that has initial metadata but no other data and it creates a name for the file in a directory
Link() creates a hard link(new path name for existing file). After calling link(), there should be multiple path names that refer to the same underlying file. You cannot call link() on a directory
Unlink() removes a file from its directory. If there are multiple links to a file, unlink() removes the specified name. If there is only one link to a file, unlink() also deletes the underlying file and frees the resources.
mkdir() and rmdir() creates and deletes directories.

Open and close

open() a process calls open() to get a file descriptor it can use to refer to the open file.
close() releases the open file record in the OS

File access

read() starts from the process’s current file position and advances it by the number of bytes successfully read or written. Reads bytes
write()starts from the process’s current file position and advances it by the number of bytes successfully read or written. Writes bytes
seek() changes a process’ current position for a specified open file
mmap() establish a mapping between a region of the process’s virtual memory and some region of the file so that memory loads and stores to that virtual memory region will use the kernel’s file cache or triggers a Page fault exception which causes the kernel to fetch the desire page from the file system to memory
munmap() remove mapping created by mmap()
fsync() Ensures all pending updates for a file are written to persistent storage before the call returns. Used to ensure that updates are durable and will not be lost in case of crash. If fsync() is called twice, the first is written to persistent storage before the second.

Software Layers

API and Performance

System calls and libraries

Application libraries wrap syscalls mentioned above in the file system API to add additional functionality such as buffering.

Block Cache

Since storage devices are much slower than DRAM, the OS has a block cache that caches recently read blocks and buffers recently written blocks.

Prefetching

When a process reads the first two blocks of a file, the OS may prefetch the first 10 blocks. When the predictions are accurate, it can reduce latency from future reads(by returning the contents in the cache), reduce device overhead(by replacing a large number of small requests with one large one), and improve parallelism(by allowing hardware to process multiple requests at once in parallel).

Some disadvantages of prefetching include cache pressure(each prefetched block might displace another block in the block cache which might be more useful), I/O contention(prefetching costs IO, other requests might need to wait behind prefetch requests), and wasted effort(the prefetched blocks are not actually used).

Device Drivers

Used to translate the high level abstractions implemented by the OS and IO devices.
Layering helps simplify OS by providing a common way to access various classes of devices.

Memory Mapped IO

IO devices typically have a controller with a set of registers that can be written and read to transmit commands and data to and from the device.
Memory Mapped IO maps each device’s control registers to a range of physical addresses on the memory bus. Reads and writes by the CPU to this physical address range go to the IO device’s controllers instead of main memory

DMA

Most IO devices transfer data in bulk. Rather than requiring the CPU to read or write each word of a large transfer, IO devices can use direct memory accesses(DMA).
DMA allows the IO device to copy a block of data between its own internal memory and the system’s main memory.
The OS uses memory mapped IO to set up a DMA transfer, then the device copies data to or from the target address without additional processor involvement
After setting up a DMA transfer, the OS must not use the target physical pages for any other purpose until the DMA transfer is done.

Lifecycle of a Disk Request

When a process issues a system call like read() to read data from disk into the process’s memory, the operating system moves the calling thread to a wait queue.
Then, the operating system uses memory-mapped I/O both to tell the disk to read the requested data and to set upDMA so that the disk can place that data in the kernel’s memory. - The disk then reads the data and DMAs it into main memory; once that is done, the disk triggers an interrupt.
The operating system’s interrupt handler then copies the data from the kernel’s buffer into the process’s address space.
Finally, the operating system moves the thread the ready list.
When the thread next runs, it will returns from the system call with the data now present in the specified buffer.

Files and Directories

Motivation: How do we go from a file name and offset to a block number? Naive Approach: The file system is a dictionary that maps keys(file name and offset) to values(block number)

Challenges with building a file system

Performance File systems need good spatial locality, where blocks that are accessed together are stored sequentially. The naive approach would just map block numbers in random places.
Flexibility File systems allow apps to share data, and thus must let people access small files, large files, short-lived files, and anything in between.
Persistence File systems must maintain and update both user data and internal data structures through OS crashes and power failures
Reliability File systems must store data for long periods of time

File System Implementation Overview

Most file systems have directories, index structures, free space maps, and locality heuristics.

Directories A way to map human readable file names to file numbers
Index structures A way to translate the file number to locate the blocks of the file. This is usually some form of tree.
Free space maps Tracks which storage blocks are free and which are in use as files grow and shrink. Also, blocks returned must have good spatial locality. Usually implemented as bitmaps
Locality Heuristics Policies that group data to optimize performance. Some file systems group by directory, others periodically defragment their storage by rewriting existing files so that each file is stored in sequential blocks.

Directories

Purpose Translate human readable names to internal file numbers in a mapping
Implementation Stored as a file
The root directory’s file number is agreed upon ahead of time, for the Unix Fast File System it is 2.
To read /home/albert/hi.txt we search the root directory by reading the file associated with file number 2. In file2 , we search for the name home and find that /home is stored in file number 880. In file 880, we search for the name albert and find /home/albert is stored in 526. In file 526, we search for hi.txt, we find /home/albert/hi.txt is in file number 469
The directory API is different from the standard open/read/close/write, since the file name to file number mapping cannot be corrupted
The syscalls that modify directories are mkdir, link, unlink, and rmdir. Mkdir and rmdir create and delete the directory, link("hi.txt", "new.txt") creates a hard link to an existing file, unlink("hi.txt") removes that hard link.
Processes can read the contents of directory files using the standard file read syscall

Hard vs Soft Links

Hard Links Multiple file directory entries that map different path names to the same file number. File systems must ensure that a file is only deleted when the last hard link is removed.
File systems use reference counts that count the number of hard links to each file, starting with 1 at creation. Each call to link() increases the reference count by 1 and each call to unlink() decreases the reference count by 1
Soft Links are directory entries that map one name to another name
If the file system supports hard links, storing file metadata in directory entries would be problematic since whenever the file size changes, all the file’s directory entries need to be updated

Files

Purpose: Translate a file number to the blocks that belong to the file Goals:

Put data blocks sequentially to maximize spatial locality
Provide efficient random access to any file block
Limit overhead to be efficient for smaller files
Be scalable to support large files
Provide a place to store metadata, ie reference count, owner, access control list, size

File System	Index Structure	Free Space Management	Locality Heuristics	Index Structure Granularity
FAT(USB drives)	Linked List	FAT array	Defragmentation	Block
FFS(Unix)	Tree(fixed, assymmetric)	Bitmap(fixed)	Block groups, reserve space	Block
NTFS(Windows)	tree(dynamic)	Bitmap in file	Best fit, defragmentation	Extent
ZFS(Copy on write)	tree(copy on write, dynamic)	Space map(log-structured)	Write anywhere, block groups	Block

FAT Filesystem

Techniques Uses a extremely simple index structure(linked list). FAT stands for file allocation table, its array of 32 bit entries in a reserved area of the volume. Each file corresponds to a FAT entry in the array, which is either the last block or contains a pointer to the next block.
Directories map file names to file numbers. The file’s number is the index of the file’s first entry in the FAT. We can then parse the linked list to find the rest of the file’s blocks.
Free space tracking If data block i is free, then FAT[i] = 0
Locality Heuristics Some FAT implementations use a next fit algorithm that sequentially scans the file allocation table starting from the last entry and returns the next free entry. This fragments the file, so there is often a defragmentation tool that reads files from their existing locations and rewrites them for better spatial locality.
Limitations: Poor locality(badly fragmented files), Poor random access(need to sequentially traverse the FAT entries), limited metadata and no access control, no support for hard links, limitations on volume and file size(max file size if 4GB), and a lack of support for reliability techniques(does not support transactional updates).

Unix Fast File System(FFS)

Techniques FFS uses a carefully structured tree that locates any block of a file that is efficient for both large and small files. Each file is a tree with fixed size blocks as its leaves. Each file’s tree is rooted in an inode that contains the file metadata.
- Typically the file’s inode contains 15 pointers: the first 12 pointers are direct pointers that point directly to the first 12 data blocks of a file
- The 13th pointer in the inode is an indirect pointer which points to an array of direct pointers.
- The 14th pointer in the *inode is a double indirect pointer which points to an internal node which points to an array of indirect pointers, each pointing to an array of direct pointers.
- The 15th pointer in the inode is a triple indirect pointer that contains an array of double indirect pointers.
- All of the file system’s inodes are located in an inode array. The file’s file number, aka inumber is an index into the inode array.
- This asymmetric setup allows for efficient support for both large and small files.
Free Space Management FFS allocates a bitmap with one bit per storage block, the ith bit in the bitmap indicates whether the ith block is free or in use.
Locality Heuristics FFS uses block group placement and reserved space
- block group placement divides the disk into block groups so the seek time between any blocks in a block group will be small
- Each block group holds a portion of the metadata structures
- FFS puts a directory and its files in the same block group
- Within a block group, FFS writes to the first free block in the file’s block group. This helps locality in the long term since fragmentation is reduced
When the disk’s space is almost full, the block group heuristic will perform poorly since new writes will be scattered randomly around the disk. Thus, FFS reserves a portion of the disk’s space and presents a slightly reduced disk size to applications

Windows New Technology File System(NTFS)

Instead of using fixed trees like FFS, NTFS uses extents and flexible trees

Extent variable sized regions of files that are stored in a contiguous region on the storage device
Flexible tree and master file table Each file in NTFS is represented by a variable depth tree. The extent pointers for a file with a small number of extents can be stored in a shallow tree, deeper trees are only needed if the file becomes badly fragmented.
Master File Table(MFT) stores an array of 1KB MFT records, each of which stores a sequence of variable size attribute records.
Resident attribute Stores its contents directly in the MFT record
Non-resident attribute Stores extent pointers in its MFT record and stores its contents in those extents.
MFT record Includes standard information(creation time, last modified, owner ID), file name, and attribute list
Four stages of file growth
1. Small files can have their contents included in the MFT record as a resident data attribute.
2. Normal file data has a small number of extents tracked by a single non-resident data attribute.
3. Large files combined with a fragmented file system can make a file have so many extents that the extent pointers will not fit in a single MFT record. These files can have multiple non-resident data attributes in multiple MFT records, with the attribute list in the first MFT record indicating which MFT record tracks which range of extents
4. If the file is huge, the file attribute list can be made non-resident, allowing arbitrarily large numbers of MFT records
Free Space Map NTFS stores all metadata in a dozen files with low file numbers. For example, file number 5 is the root, file number 6 is the free space bitmap, file number 8 contains a list of the volume’s bad blocks.
Locality Heuristics NFTS uses best fit, where the system tries to place a newly allocated file in the smallest free region large enough to hold it. NTFS also has a defragmentation utility

Copy on Write (COW)

When updating an existing file, COW file systems do not overwrite existing data or metadata, instead, they write new versions to new locations.

Motivations
- Small writes are expensive. Large sequential writes is much better than small random writes, and the gap will continue to grow since bandwidth grows faster than seek time/rotational latency
- Small writes are expensive on raid since we need to read old data, read old parity, write new data, and write new parity. The number of times we read/write to parity bits grows linearly with the number of reads/write but not the size of the reads/writes
- Large DRAM caches can handle essentially all file reads, thus the cost of writes dominate performance so optimizing write performance is the number one priority.
- Flash storage works better with copy on write: there is no need to clear the erasure block and it writes data to a different location instead of overwriting the current data wearing out the flash drive evenly
- Since old data isn’t overwritten, we can support versioning
Implementation: We store inodes in a file rather than in the inode array. All the file system’s contents are stored in a tree rooted in the root inode, when we update a block, we write the block and all of the blocks on the path from it to the root to new locations.

ZFS file system

uberblock the root of the ZFS storage system, ZFS has an array of 256 uberblocks and rotates successive versions
dnode similar to inode, a file represented by a variable depth tree whose root is a dnode and leaves are data blocks
Free space map per block group space maps. ZFS maintains a space map for each block group, has a tree of extents, and log structured updates.
Locality Heuristics On writes, ZFS ensures sequential writes and batched updates.

File and Directory Access Walkthrough

Goal: Read the file /foo/bar/baz Steps:

Read the root directory / to determine /foo’s inumber
Open and read file 2’s inode. Since FFS stores pieces of the inode array at fixed locations on disk, given a file’s inumber it is easy to read the file’s inode
From the root directory’s inode, extract the direct and indirect block pointers to determine which block stores the contents of the root directory.(block 48912)
Read that block of data to get the list of name to inumber mappings in the root directory to find /foo has inumber 231
Since we know /foo’s inumber, we read inode 231 to find where /foo’s data blocks are stored-block 1094 in this example.
Read block 1094 to get the list of name to inumber mappings in the /foo directory to find that the directory file /foo/bar has inumber 731.
Follow similar steps to read /foo/bar’s inode and data block 30991 to find /foo/bar/baz’s inumber is 402
Read /foo/bar/inode to get the data blocks 89310, 14919, 23301
Usually, data blocks are cached so we don’t need to repeat this process.

Questions

What does mmap() do?

Reliable Storage

Sat, 20 Nov 2021 23:46:37 GMT

Why do we need reliable storage?

Physical devices may fail, they may wear out, they may become damaged. Large organizations see annual disk failure rates of 2-4%, which means that over 10 years 30% of the data in the disk would be gone
Two threats to reliability
- Operation Interruption: A crash or power failure in the middle of a series of related updates leaves the data in an inconsistent state. Ie crashing when moving a file so we now have two copies
- Solution Create transactions for atomic updates.
- Loss of stored data: Failure of non-volatile storage media can cause stored data to become corrupted or disappear
- Solution Create redundancy for failures, ie RAID or using multiple layers of checksums

Reliability, Availability,

Reliable: Can the system perform the intended function? A system can be available but not working properly. Can increase reliability by testing system.
Available: Can the system respond to a request? Also known as the percentage of time when the system is up. Can achieve higher availability with deploying apps across multiple geographically distant servers using load balancing to reroute requests to healthy servers
Durability: Is the data still in the system? A system is durable even if you can’t access the data(ie bury a usb underground). Can achieve higher durability by making backups, storing in different geographical area, performing checksums.
A data block replicated across 100 machines is highly reliable and highly available for reads, but not highly available for writes.

Transactions + Atomic Updates

Problem: When the system crashes in the middle of several updates, the data in the system might not make sense or leaves it in a vulnerable. For example, bob is transferring 100 dollars to alice and the bank database crashes, it could have crashed when the bank pulled 100 dollars out of bob’s account but before it sent the 100 dollars to alice’s account. Now, bob just lost 100 dollars
Naive Solutions: The unix fast file system would control the order of updates such that if a crash occurred, it could scan the disk during recovery and identify and repair inconsistent data structures. Now if this sounds kind of sketchy, it is and there were three main problems with it.
- Extremely slow recovery: When the machine reboots, it has to scan all of its disks for inconsistent metadata structures.
- Slow Updates To ensure the unix fast file system updates were analyzable, they had to be ordered correctly, making it hard to parallelize the stream of requests to storage devices
- Complex reasoning We need to consider all possible failure scenarios to make sure that it is always possible to recover the system to a consistent state.
Application Level Approaches The POSIX api first writes updates to a temporary file, then renames the temporary file to atomically replace the previously stored file.

The Transaction Abstraction

Problem You are updating a website and you want to update files stored in /prod with files stored in /testing. However, you don’t want users to see the intermediate step where some docs in /prod are old and some are new.
Solution The transaction first does all of its changes, then commits the transaction. If at any time the transaction fails before the commit, we roll back everything done in the transaction. The benefit here is that transactions are ACID.(Yes this sounds like a dumb acronym they teach you in 6th grade to memorize the dinosaurs but this is actually an extremely well thought out acronym. ACID = transaction)
- Atomicity All or nothing, either a transaction commits or the system is not changed.
- Consistency The transaction moves from one legal state to another. All invariants must hold at the start of the transaction and during the commit.
- Isolation The updates within a transaction cannot interleave with other transactions. Even if multiple transactions concurrently execute, the results will be that T is executed entirely before T` or vice versa
- Durability Once a transaction is committed, the only way to change state is with another transaction. Must survive crashes!
- Transactions are just critical sections with durability.

Transaction Implementations

Problem: We want to group related writes in a single, atomic transaction but for disks and other hardware, the largest atomic operation is a single write(sector or page). Thus, we need a way to combine multiple writes into a single atomic operation.

Idempotence: The property where an operation can be applied multiple times without changing the result beyond the first application.

Idea: A transactional system can store all of the transaction’s intentions, and the updates will only be made after all of the transaction’s intentions are written down. If we store all updates as idempotent operations, we can also start off where we finished during a crash

Redo Logging

An implementation of transactions using four phases
1. Prepare Append all planned updates to the log.
2. Commit Append a commit to the log, indicating the transaction has been committed
3. Write-back Write values in log to disk
4. Garbage Collect Remove transaction records from log
If the system crashes in the middle of the transaction, we scan sequentially through the log and depending on the type of record we do different things. If the record is an
1. Record Type = Update add this record to a list of updates for the specified transaction
2. Record Type = Commit Write back all of the transaction’s logged updates to disk
3. Record Type = Roll-back Discard list of updates for specified transaction
4. Record Type = End Discard all update records for transactions without commit records.
Implementation Details
1. Multiple transactions might be executing at once, meaning that we need an identifier to show which transaction the log record belongs to
2. The write-back phase can be asynchronous, meaning we can minimize latency in the commit phase and write-back multiple things at once. However, this makes crashes take longer and the log needs to take more space in the disk.
3. Since log records are idempotent, we can reapply an update from a redo log multiple times.
4. Restarting in recovery is ok, we just continue from where we last left off.
5. Once write-back completes, the space in the log can be reclaimed
6. All transaction updates must be on the log before the commit, all commits must be on disk before the write backs, all write backs must be on disk before the transaction log records can be garbage collected.
Performance: redo logging can have performance better than update in place, especially for small writes. Four factors allow efficient redo logging:
- Log updates are sequential, so writing to spinning disks can be very fsat.
- Write back is asynchronous, so transactions using redo logs can have good response time and throughput
- The only barriers are when updates are logged and before the commit is logged, and after the commit is logged and before write-backs begin
- Group commits combine a set of transaction commits into one log write to amortize the cost of initiating the write

Isolation and Concurrency

To fulfill ACID properties, we need to ensure all of its properties. Redo logging only ensures durability, we also need isolation if there are multiple transactions executing at once.

Naive Solution: Two phase locking divides a transaction into two phases, the expanding phase where locks can only be acquired and the contraction phase where locks can be released but not acquired.

Two phase locking results in serializability, meaning the execution is equivalent to processing the transactions one at a time.
If we encounter deadlocks, we can roll back a transaction and restart at some later time.

Transactions in File Systems

If the file system crashes when updating data, it could be left in an inconsistent state. Thus, all file system updates should be done in transactions.

Logging: Include all updates to disk in transactions
Journaling: Apply metadata updates via transactions. Apply file updates normally. Ths is because file updates are idempotent
Copy-on-write: Does not overwrite data in place, but rather writes data then replaces pointer.
- Copy on write systems also usually have batch updates, where you batch multiple updates into one atomic group
- This also has an intent log, a redo log that tracks updates between large batch updates

Error Detection and Correction

Problem: All data storage devices will fail Solution: Use layered error detection and correction: checksums for devices, error correcting codes, RAID architectures, end to end correctness checks.

Types of Storage Device Failures

Sector and Page Failures

Problem: Data in the sector of a disk is lost. Flash page failures are the same thing but on SSDs. Cause: The magnetic coating on spinning disks get scratched off, machine oil coats a sector of the spinning disk, the disk head gets too far from the surface. In flash storage, a page can get worn out from using it too many times. Mitigation:

Error correcting codes: Encode data with redundant metadata that tells you if a sector becomes damaged
Remapping: remap failed sectors to good ones Pitfalls:
Assuming that non-recoverable read error rates are negligible. A bit error rate of one sector per 10**14 bits means in a 2TB read there is a 10% chance of error
Assuming non-recoverable read error rates are constant. Specific workloads can wear out some parts more than others
Assuming failures are independent. Errors are usually correlated by time and space
Assuming uniform error rates. Some models are just worse than others

Device Failures

Problem: Device stops being able to read or write to any disk sector. Cause:

For disks: Disk heads become damaged, power surge damages electronics.
For flash devices: Wear out and device runs out of space for remapping Mitigation: Notify the device reading the disk that there is an error Pitfalls:
Trusting advertised failure rates. Big tech lies
Failures are usually correlated
A mean time to failure is not the same as its useful life
Failure rates are not constant, there are devices with disk infant mortality, and devices that die after wearing out
Ignoring warning signs
Devices don’t behave identically, even if manufactured identically.

RAID Multi disk redundancy for error correction

Idea: Spread data redundantly across multiple disks in order to tolerate individual disk failures.
Mirroring: The system write each block of data to two disks and can read data from either disk. If one fails, read from the other disk
Rotating Parity: store several blocks of data on several disks and protect those blocks with one redundant block stored on yet another disk.
- Each disk has 1 / # data blocks of its space dedicated to parity and is responsible for storing (# data blocks - 1) / # data blocks.
Striping data: a strip of several sequential blocks is placed on one disk before shifting to another disk for the next strip. Thus, requests larger than a block but smaller than a strip require just one disk to seek then read/write.
Recovery: If a disk suffers a sector failure, it reports it when there is an attempt to read the sector then remaps the damaged sector to a spare and reconstructs the lost sector from the other disks and rewrites it to the original disk.
Ways RAID can fail
- Both disks fail
- One disk fails and the other disk’s sector fails
- Both disk’s sectors fail
  
  Improving RAID Reliability
There are three main things we can do to improve reliability: increase redundancy, reduce non recoverable read error rates, reduce mean time to repair
To increase redundancy, duplicate it to 3 disks instead of 2.
To reduce non-recoverable read error with scrubbing, read the entire contents of disk, find sectors with unrecoverable read errors, and reconstruct lost data from remaining RAID disks
To reduce non recoverable read error rates with more reliable disks, use higher quality disks
To reduce mean time to repair, use hot spares: disks that are idle but plugged into a server so that if one disk fails, the hot spare can be automatically activated
Reduce mean time to repair with declustering: split the reconstruction of a failed disk accross multiple disks.
Pitfalls: assuming uncorrelated failures, ignoring risk from latent errors, not implementing scrubbing, not having a backup,

Free Resources at UC Berkeley

Wed, 10 Nov 2021 23:46:37 GMT

Physical Resources

Free Printing: Open Computing Facility(171 MLK Student Union). 100 pages of free prints per semester. Need cal id to create an account
If you want to build cool stuff with electronics
- Jacobs Design Institute. Useful for getting a product/UI/UX Designer
- Supernode

Tech Resources

If you want free website hosting
- The Open computing facility will host your websites for free. (https://www.ocf.berkeley.edu/docs/)

Financial Resources

If you don’t know the difference between a credit and debit card
- Bears for financial success(Financialaid.berkeley.edu/bffs). Useful for Identity theft, credit, debt, credit cards.
If you do research or want to do research:
- Office of Undergraduate Research and Scholarships(Research.berkeley.edu) Useful for research, scholarships related to research

Startup Resources

List of “accelerators”
UC Launch, Big Ideas, Citrix-Banatao, https://startup.berkeley.edu/, https://www.dormroomfund.com/apply

CS Classes Resources:

CS61A
http://alvinwan.com/publications/abcSICP.pdf

Before Classes Start:

Figure out who is the best GSI(Graduate Student Instruction)
- There’s a big gap between the best and work GSIs/UGSIs at Cal. Try to figure out which GSI is the best(usually a combination of semesters on course staff, if they are a head TA, if they have resources for the class that aren’t official class materials <- this one is huge because it shows that they care). Attend all of their discussion, labs, and office hours and if you do, you pretty much have at least an A- in the class.

When moving to a new apartment

Take a slow moving video before/right after moving in to prevent getting sued for damages

Osmosis Office

Osmosis has an office in berkeley, iykyk

Input and Output

Mon, 25 Oct 2021 23:46:37 GMT

Motivation

Without I/O, computers are useless. However, there are thousands of devices, each one slightly different. How do we standardize the interfaces? Devices can be unreliable(media failures/transmission errors/ packet loss), how do we make them predictable, fast, and reliable?

Bus

Common set of wires for communication among hardware devices + associated protocols to transfer data
Allows for read, write
Connects n devices over a single set of wires, connections, protocols O(n^2) relationships with 1 set of wires
Cons: Only one transaction at a time

How does the Processor Talk to the Device?

The CPU contains a set of registers that can be read and written
Port Mapped I/O: in/out instructions
Memory Mapped I/O: load/store instructions

Optimal Parameters for I/O

Data granularity: Some devices provide single bytes at a time(keyboards), while other provide whole blocks (disks, networks)
Access Pattern: Some devices must be accessed sequentially(tape), others can be accessed randomly (disk, cd). Some devices require continual monitoring while other generate interrupts when they need service

Transferring Data To/From Controller

Programmed I/O: Each byte transferred via processor in/out or load/store
- Pros: Simple hardware and software
- Cons: Consumes processor cycles proportional to data size
Direct Memory Access: Give controller access to memory bus and asks it to transfer data blocks to/from memory directly
Blocking Interface: When system reads data, sleep the process until the data is ready. When the system writes data, sleep the process until device is ready for data.
Non-blocking Interface: Return quickly from read/write with count of bytes transferred. It may return nothing and write may write nothing.
Async Interface: When reading data, take pointer to user buffer, return immediately since the kernel will fill the user buffer and notify user. When writing data, get the pointer to user buffer, return immediately. The kernel will read the data and notify the user.

Advanced Memory Management

Mon, 25 Oct 2021 23:46:37 GMT

Zero Copy I/O

Problem: Operating systems commonly stream data between user level programs and physical devices. However, it takes a ton of time to do so since we move data from hardware to kernel then userspace or vice versa.

Potential Solutions that don’t work: We could move the application into the kernel directly so the data doesn’t need to be copied from kernel to userspace. (This is not secure) We could also modify syscalls to allow applications to modify the data stored in the kernel buffer. (This isn’t flexible, it doesn’t allows for modifications)

Solution 1: Use the process’ page table The application aligns the user level buffer to a page boundary which is then provided to the kernel. Thus, a page-to-age copy from user to kernel space can be simulated by changing page table pointers instead of physically copying memory.

Virtual Machines

A way for a host OS to run a guest OS as an application process.
They are widely used on client machines to run non-native applications, are used in data centers to allow a machine to be shared by multiple machines

Virtual Machine Page Tables

There are two sets of page tables, guest and host
The host OS provides a set of page tables to constrain the execution of the guest OS which is thinking it is running on a real OS but it is actually running on virtual memory.
The host translates the guest virtual addresses into physical addresses
The guest OS manages page tables for the guest processes, exactly the same as if it was on real hardware. This is because the guest OS thinks it is on real hardware.

When the host OS transfers control to the guest OS, everything works as expected.

When the guest OS transfer control to the guest process, this is a privileged instruction and the hardware processor will first trap back to the host. However, we can’t use the guest OS page table, since it is using virtual addresses instead of physical addresses. Nor could we use the host OS page table since the guest OS should not have permission to modify that content. Thus, we will need to use Shadow Page Tables.

Shadow Page Table

A composite page table that represents both the guest and host page tables.

To keep the shadow page table up to date, the host needs to check if any shadow page tables need to be updated before it changes a page table entry.
To keep the shadow page table up to date, the host also sets the memory of the guest page tables as read only. Thus, the guest OS traps the host every time it makes a change to a page table entry. When trapping, the host updates both the guest page table and the corresponding shadow page table.

Transparent Memory Compression

Problem: Multiplexing multiplexors is hard. For example, a data center usually has a single physical machine running many virtual machines at the same time(ie a computer hosting multiple web sites)
When a guest OS is running two copies of the same program or process, the host kernel has no way to share pages between the two instances.
Solution: Commercial Operating systems run a scavenger in the background that looks for common pages that can be shared among multiple virtual machines. Once identified, the host kernel provides an illusion that each guest has its own copy.

Case 1: Multiple copies of the same page Multiple virtual machines might share a zeroed page, it would be wasteful to create a new zeroed page for each virtual machine that needs one. The host OS instead allocates a read-only zero page, which traps the host kernel when modified and allocates a new zeroed physical page for the guest OS.

They scavenger runs in the background looking for zeroed pages

Case 2: Compression of unused pages Pages can differ by small amounts, this lets the host kernel add a layer in the memory hierarchy to save space. Instead of evicting a relatively unused page, we can save its delta with regards to the original page, and the full copy the delta is from stays read only.

If the original page is modified, the delta can be quickly recompressed or evicted as necessary.

Fault Tolerance

All systems break, but how can we manage memory to recover application data structures after a failure, not just user file data?

Checkpoint and Restart

Naive Solution: Periodically save the program’s internal data to a file.
General Solution: The OS uses the virtual memory system to provide application recovery as a service Implementation:
Suspend each thread executing in the process and save its state. (Suspending threads is to avoid race conditions)
Store a copy of the contents of the application memory on disk (this is the checkpoint)
After a failure, resume execution by restoring the contents of memory from the checkpoint and resume each of the threads.

Use address translation to minimize the amount of time the system is stalled during a checkpoint. We can mark the application pages as copy on write to do so.

Process Migration: The ability to take a running program on one system, stop the execution, then resume it on a different machine.

Recoverable Virtual Memory

Unfortunately, checkpoint + restart is an expensive operation that we should use sparingly. Is it possible to provide the application the illusion of persistent memory, so that the contents are restored to a point not long before failure?

Implementation:

Take a checkpoint so that some consistent version of the application data is on disk
Record a log of every update that the application makes to memory.
Once things crash, read the checkpoint and apply the correct changes from the log.

Pros: This is how text editors save their backups. Cons: Information that is not on the current version of the file can be leaked!

Incremental Checkpoint: Created by stopping the program and saving a copy of any pages that have been modified since the previous checkpoint.

How much work is lost during a crash is a function of how quickly we can write an incremental checkpoint to disk.

Deterministic Debugging

Debugging a concurrent program is hard, we don’t control the scheduler, thus depending on how threads are scheduled program runs may vary.
Debugging an OS is even harder, the OS uses widespread concurrency but also makes it difficult to tell input from output.
Implementation: Use a virtual machine abstraction to allow for debugging. The host kernel mediates the initial state, the input data from I/O devices, and precise timing of interrupts.
For multicore systems, we also need to track the order in which threads go through critical sections in addition to the initial state, the input data from I/O devices, and precise timing of interrupts

Security

The operating system needs to protect itself and other applications from malicious or buggy implementations.

Virtual Machine Honeypot: Run the suspect application in a virtual machine constructed for the purpose of running suspect code. We can delete the virtual machine if the code is suspect.

User Level Memory Management

Operating Systems now provide users the ability to:

Decide where to get missing pages. (local disk, non-volatile memory, remote memory, etc.)
Which pages can be accessed
Which pages should be evicted.

Storage Devices

Persistent Storage devices like SSDs or Magnetic Disks can hold much more memory but are much slower than DRAM.

Magnetic Disk

Description: Each disk drive holds one or more platters, which are the CD looking things that hold up the magnetic material. When the disk drive powers up, the platters are constantly spinning on the pole. The disk head reads/writes data by sensing or introducing a magnetic field on a surface. Since the platter is spinning very fast, it generates a layer of rapidly spinning air, which lets the disk head float close to the magnetic material without touching it. If the disk head does touch the material, this is known as a disk crash. This can happen when you drop a running magnetic disk.

The data itself is stored in 512 byte sectors, the disk drive must read and write the entire sector even if it only wants to change a single byte in the sector.

Track Buffering: Improves performance by storing sectors that have been read by disk head but not yet requested by the OS.
Write acceleration: Stores data to be written to disk in the disk’s buffer memory and acknowledges the writes to the OS before the data is written to the platter. The data is actually written later. Of course, if power is lost before the disk’s buffer memory has been stored, then data might be lost.
Tagged command queueing: Issue multiple concurrent requests to disk and the disk processes the requests out of order in an optimal manner.

Access and Performance

To go from a request to data, the disk must first seek the right track, wait for the desired sector to rotate to the head, then transfer the blocks. The equation for this is disk access time = seek time + rotation time + transfer time

Seek Time: Move the arm over the desired track(each track is kind of like a different lane on a track in track and field). The minimum seek time is the time it takes for the head to move from one track to an adjacent one. The maximum seek time is the time it takes for a disk to move from the innermost track to the outermost track. The average seek time is the average seek time across all possible pairs of tracks on a disk, this is usually approximated to the time it takes to seek a third of the way across a disk
Rotation Time: After the disk head and found the correct track, it must wait for the target sector to rotate under it. The waiting time is known as the rotational latency. However, most disks start reading the entire track’s sectors into their buffer memory.
Transfer time: After the disk head is on the target sector, the disk must transfer the data from the target sector to the buffer memory(for reads) or from the buffer memory to the target sector(for writes). Then, it transfers the buffer memory to main memory for reads or vice versa for writes. The surface transfer time is the time to transfer one or more sequential sectors from a surface once the disk head begins reading. Thus, the surface transfer time is usually faster for outer tracks than inner tracks since outer tracks have a higher linear velocity.

Flash Storage

Flash storage is now used for laptops, machine room servers, phones, cameras, thumb drives, and everything in between. Flash storage is a type of solid state storage, it has no moving parts and stores data using electrical circuits. It has much better random I/O, use less power, and is less vulnerable to physical damage.

Each flash storage element is a floating gate transistor. Since the floating gate is surrounded by an insulator, it can hold an electrical charge for months or years without requiring power. Even though it is not connected to anything, it can be charged or discharged via electron tunneling by running a sufficiently high-voltage current near it. Thus, the floating gate state can be detected by applying an intermediate voltage to the transistor control gate that will only be sufficient to activate the transistor if the floating gate is changed.

Access and Performance

There are three operations we can perform on flash storage.

Erase erasure block: Before flash memory can be written, it must be erased by setting each cell to a logical one. Flash memory can only be erased in large units known as erasure blocks. This is a slow operation, taking several milliseconds.
Write Page: Once erased, NAND flash memory can be written on a page-by-page basis, where each page is typically 2048-4096 bytes. This takes tens of microseconds.
Read Page: NAND flash memory can be read on a page by page basis. Reading usually takes tens of microseconds.

Calculations

Without remapping
Time per write = (sizeof erasure block / page size) * (page read time + page write time) + block erase time With remapping Time per write = block erase time / (sizeof erasure block / page size) + page write time Remapping amortizes the cost of flashing an erasure block.

Durability

Over time, the high current loads from flashing and writing memory causes the circuits to degrade. Eventually after a few thousand to few million erasures, a given cell will wear out and no longer reliably store a bit.

Thus, flash devices use:

Error correcting codes: Each page has some extra bytes to protect against bit errors in the page
Bad page and bad erasure block management: If a page or erasure block has a manufacturing detect or wears out, the firmware marks it as bad and stops storing data on it.
Wear leveling: Rather than overwriting a page in flash, the flash translation layer remaps the logical page to a new physical page that has already been replaced. Thus, we wear out the flash device evenly
Spare pages and erasure blocks: Manufacturers may include spare pages and spare erasure blocks to provide extra space for wear leveling and to allow for bad page and bad erasure block management without shrinking the capacity of the storage device.

The future for memory devices

Spinning disk vs flash storage: Textbook is outdated, flash storage is almost always better
Phase change memory: Use a current to alter the state of glass between amorphous and crystalline forms which represents the data bit(0 or 1). Apparently it has better write performance and endurance than flash, though right now storage density isn’t that great.
Memristor A circuit element whose resistance depends on the amounts and directors of currents that have flowed through it in the past. The goal is for each core on a 32 core processor chip to have a few gigabytes of stacked memristor memory

Questions

Caching

Fri, 22 Oct 2021 23:46:37 GMT

Examples of Caches

TLBs: Caches recent results of address translation from page tables.
Virtually addressed caches: Stores memory value associated with a virtual address
Physically addressed caches: This is a part of the virtually addressed cache. Each entry stores the memory value associated with a physical memory location.
DNS caches, Web content caches, web search, email clients, incremental compilation, just in time translation, virtual memory, file systems, conditional branch prediction all use caches.

Design Challenges of Caches

Locating the cached copy: Check if the value you are looking for is in the cache
Replacement Policy: Which elements should be replaced when the cache is full?
Coherence: What should we do when the cache is out of date?

Naive Cache Implementation

An (address, value) pair stored in a map. When looking for something, we check if it exists in the map and return the value if it does(aka a cache hit). If it doesn’t this is a cache miss.
The cost of retrieving data out of the cache must be less than fetching the data from memory. The likelihood of a cache hit must be high enough to be worth the effort.
Two sources of predictability are temporal locality and spatial locality. Programs tend to reference the same instructions and data they recently referenced(Temporal Locality). Programs also tend to reference data that is near recently referenced data.
Latency(read request)=Prob(cache hit) × Latency(cache hit)+ Prob(cache miss) × Latency(cache miss)
Write through cache: All updates are sent immediately onward to memory
Write back cache: Updates are stored in the cache and only sent to memory when the cache runs out of space.

Memory Hierarchy + When to use Caches

Fundamental tradeoff: the smaller memory is, the faster it can be. The slower memory is, the cheaper it can be.

Working Sets

A sufficiently large cache will have a high cache hit rate, since if you can fit all the memory and data, the miss rate will be 0.
Most programs have an inflection point where a critical mass of program data can just barely fit inside the cache. This is known as a working set
Thrashing: Occurs when a cache is too small to fit inside the working set.

Zipf Model

Motivation: New data is constantly being added to the web, it is likely that your cache is outdated when you request something from it. There is no working set for the web, a small amount of pages might be useful(gmail, facebook, youtube) but there is a huge amount of web pages that are only visited from time to time.
Zipf can be described as Frequency of visits to the kth most popular page ∝ 1 / kα and can model the popularity of library books, the population of cities, salary distribution, number of facebook friends, etc.
Heavy tailed distribution: A significant number of references will be the most popular items, but a substantial portion will be less popular ones.

Cache Design

Fully associative: Addresses can be stored anywhere in the cache, there is no ordering. When looking for an address, we check every entry in the cache: there is a cache hit if any of the entries match. Cons: This is really slow as caches becomes bigger.
Direct mapped: Each address is hashed then the data stored at the location of the hash. Cons: If two different addresses hash to the same entry, the system will thrash.
Set associative: The cache hashes the address to get a location, then checks the entry in each table in parallel. It returns the value if any entries match the address.

Replacement Policies

Motivation: If we look up an address and find that it is a miss, how do we choose which memory block to replace?

Random

How it works: Choose a random block to replace.
Pros: Randomly evicting a block is fast, thus it is good for first-level hardware caches. Non-pessimal, randomness will never be the worst possible choice on average.
Cons: Unpredictable, it might counteract an application that tries to take advantage of the cache.

First in First Out(FIFO)

How it works: Evict the cache block or page that has been in memory the longest.
Pros: Each page spends an equal amount of time in the cache.
Cons: Imagine a program that cycles through an array, where the array is too large to fit into the cache. Then the cache is useless.

Optimal Cache Replacement (MIN)

How it works: Replace whichever block that is used farthest in the future.
Pros: This is optimal. (aka the best possible algorithm)
Cons: Impossible, MIN requires knowledge of the future.

Least Recently Used (LRU)

How it works: Evict the block that has not been used for the longest period of time. In software, this is done with a linked list. In hardware, it is approximated with the clock algorithm
Pros: Takes advantage of temporal locality
Cons: Weak when the task is repeated scans through an array, the optimal strategy there is FIFO

Least Frequently Used(LFU)

How it works: Discard the page used least often
Pros: Good in web caching. Ie just because we recently looked at spas in Iceland doesn’t mean we will frequently look at spas in Iceland. Frequency is a better approximation than time.
Cons: If an item is referenced repeatedly for a short period of time, this will stay in the cache long after it should have been evicted.

Belady’s Anomaly

Intuitively, adding space to a cache should always improve or maintain the cache hit rate. However, Belady’s Anomaly says this isn’t the case.

For some replacement policies like FIFO, adding more space will drastically change the contents of the cache, thus making the hit rate worse.

Memory Mapped Files

Demand Paging: Applications can access more memory than is physically present on the machine by using memory pages as a cache for disk blocks. Memory Mapped File: The program directly accesses and modifies the contents of the file, treating memory as a write-back cache for disk.

Pros

Transparency: The program can write directly into the file as if it was memory
Zero copy I/O: The OS does not need to copy file data from kernel buffers into user memory and back, it can just change the program’s page table entry to point to the physical page frame containing that part of the file.
Pipelining: The program can operate on data in the file as soon as page tables are set up, no need to wait for the entire file to be read into memory.
Interprocess communication: Two processes can share memory via a memory mapped file.
Large Files: The only size limit of a memory mapped file is the size of the virtual address space.

Implementation

OS provides a system call where the kernel initializes a set of page table entries for that region of virtual address space, setting each entry to invalid.

When the process gives an instruction that touches an invalid mapped address:

The TLB miss occurs, triggering a full page table lookup in hardware.
The hardware finds the page table entry and finds that it is invalid, causing a hardware page fault into the OS kernel.
The OS kernel converts the address to a file offset.
The OS kernel allocates an empty page frame and reads the required file block into the empty page frame.
The disk interrupts the processor when the file block has been read and the scheduler resumes handling the page fault exception.
The kernel updates the page table entry to point to the page frame allocated for the block and sets the entry to valid.
The OS resumes execution of the process.
The TLB misses, trigger a full page table lookup.
The hardware returns the page frame to the processor, loads the TLB with the new translation, evicts the previous TLB entry, then uses the translation to construct a physical address.

To create the empty page frame that holds the incoming page from disk, the OS:

Selects a page to evict using the current replacement policy.
Finds page table entries that point to the evicted page, usually through a core map.
Set each page table entry to invalid, to prevent other processes from using the evicted page while it is being evicted.
Copy back any changes to the evicted page.

Approximating LRU

Keeping a linked list to track when pages were last used in hardware is prohibitively expensive, thus, we will use the clock algorithm as an approximation instead.

Clock algorithm

The OS scans through the core map of physical memory pages. 2. For each page frame, it records the value of the use bit and clears the use bit. 3. The OS does a shootdown for each page frame that has its use bit cleared.

A common usage of the collected values of the use bit is the not recently used/kth chance algorithm where the kernel picks a page that has not been set for the last k sweeps of the clock algorithm.

Virtual Memory

By backing every memory segment with a file on disk, we create virtual memory.

Self Paging

Problem: There is a malicious program that tries to grab as much memory as possible by using multiple threads to cycle through memory and add those pages. Solution: Self paging assigns every process its fair share of page frames. As the system starts to page, it evicts the page from whichever process has the most pages allocated to it. Cons: If there are two processes, one with a working set of 2/3 of memory and the other with 1/3 of memory, the ideal split is for the first process to get 2/3 of memory and the other to get 1/3 of memory. Unfortunately, self-paging will results in both process splitting the memory, causing the first process to thrash

Swapping

Problem: Sometimes, there is simply too much data in all of the process’ working sets to effectively run them at once. If we try to run things normally, we will end up thrashing with every process. Solution: We can evict an entire process from memory (aka swapping) Cons: The process that just got evicted from memory will be sad :(

Future

Low latency backing store: Things that were once on hard disk are moving onto SSDs, things that are on SSDs will move onto even faster storage devices.
Variable page sizes: We will move toward smaller effective page sizes due to low latency solid state storage and cluster memory.
Memory aware applications: Applications can be tuned to how much memory is in the first level cache to increase performance.

Questions

How does set associative caches work?
How does virtual memory work?

Cross Validation and Regularization

Thu, 21 Oct 2021 23:46:37 GMT

The train-test Split

Training Data: Used to fit model
Test Data: Used to check generalization error
How to split? Randomly, Temporally, Geographically (Usually Random)
What size? Larger training set = more complex models, Larger test set = better estimate of generalization error (Usually 90%-10% or 75%-25%)

Recipe for Successful Generalization

Split data into training(90%) and testing set(10%)
Use only training data when training, use cross validation to test generalization(do not look at testing data!)
Commit to the model and train once more using only training data
Test the model using testing data. If accuracy is not acceptable, return to step 2
Train on all available data and ship it

Regularization

Parametrically controlling the model complexity Naive Solution: Find the best value of theta which uses less than B features. Unfortunately, this is an NP hard combinatorial search problem.

L0 Norm Ball Ideal for feature selection but combinatorically difficult to optimize
L1 Norm Ball Encourages sparse solutions and is convex!
L2 Norm Ball Spreads weight over features(robust) and does not encourage sparsity
L1 + L2 Norm(Elastic Net) Compromise, need to tune two regularization parameters

Standardization: Ensure each dimension has the same scale and is centered around 0 Intercept Terms: Typically doesn’t regularize the intercept term

Ridge Regression

Model: Y = X * theta Loss: Squared Loss Regularization: L2 regularization Objective Function Squared Loss + added penalty

Note that there is always a unique optimal parameter vector for Ridge Regression!

Lasso Regression

Model: Y = X * theta Loss: Squared Loss Regularization: L2 regularization Objective Function Average Squared Loss + added penalty

Note that there is NO closed form solution for the optimal parameter vector for LASSO so we must use numerical methods like gradient descent

Questions

Inference

Thu, 21 Oct 2021 23:46:37 GMT

Inference

Moving from premises to logical consequences
Induction is inference from a particular premise to a universal conclusion
Statistical inference: Using data analysis to deduce properties of underlying distribution

Prediction vs Inference

Prediction: Using our model to make predictions for unseen data. (Given attributes of house, how much is it worth?) We don’t care about how
Inference: Using our model to draw conclusions about the underlying true relationships between our features and response. (How much extra will a house be worth if it has a view of the river?) We care about model parameters that are interpretable and meaningful

Statistical Inference

Draw conclusions about a population parameters given only a random sample
Parameter: some function of population, ie population mean
Estimator: some function of a sample, whose goal is to estimate a population parameter, ie sample mean. Estimators are random variables
- Bias of an estimator: difference between estimator’s expected value and the true value of the parameter being estimated.
- Variance of an estimator: Expected squared deviation of an estimator from its mean

Bootstrapping

Idea: treat our random sample as a population and resample from it

Psuedocode

collect random sample of size n(aka bootstrap population)
initialize list of estimates
repeat 10,000 times:
    resample with replacement from bootstrap population
    apply estimator f to resample
    store in list
list of estimates is the bootstrapped sampling distribution of f

The median cannot be accurately drawn from bootstrapping
If the sample is too small, bootstrapping won’t work

Confidence Interval

What does a confidence interval p% mean?
- If we take a sample from the population and compute P% confidence interval for the true population parameter, and repeat this many times, our population parameter will be in our interval P% of the time.
to compute confidence interval(s,f,P), approximate sampling distribution of f using sample s. Choose middle P% of samples from this approximate distribution
A 95% confidence interval does not mean that there is a 95% chance that the population parameter is in the interval, either the population parameter is in or isn’t

Bootstrapping Model Parameters

Our estimate for theta depends on what our training data was.
We want to think about all of the different ways that our training data and our parameter estimate could have come out

We want to test whether a feature has any effect on the outcome. This works for linear and logistic regression models with any number of features.

Estimate theta1 each time
Make confidence interval for theta 1 and see if 0 is in the interval
If yes, theta 1 is not significantly different than 0
If no, theta 1 is significantly different than 0

Multicollinearity

If features are related to one another, it might not be possible to have a change in one while holding the others constant
Multicollinearity: Where a feature can be predicted fairly accurately by a lienar combination of other features.
- Doesn’t impact model predictability, only interpretability
Perfect Multicollinearity: one feature can be written exactly as linear combination of other features

Summary

Estimators are functions that provide estimates of true population parameters
We can bootstrap to estimate the sampling distribution of an estimator
Using the bootstrapped sampling distribution, we can compute a confidence interval for our estimator
- This gives a rough idea of how uncertain we are about the true population parameter
- Only valid if the original random sample is representative
The assumption when performing linear regression is that there is some true parameter theta that defines the linear relationship between features X and response Y
- We can use bootstrap to determine whether or not an individual feature is significant
Multicollinearity arises when features are correlated with one another
Supervised Learning = We have an X and Y
Unsupervised Learning = We only have x, want to learn about X

Questions

What is confidence interval

Logistic Regression

Thu, 21 Oct 2021 23:46:37 GMT

Linear Regression

Goal: Predict a number from a set of features Output: A real number Process: determine optimal parameters by minimizing some average loss and something with a regularization penalty.

Classification

Goal: Predict a categorical variable(ie win or lose, disease or no disease, spam or ham) Types: Binary Classification(Two classes, ie spam or not spam), Multiclass classication(type of animal)

Logistic Regression Model

P(Y=1|x) = 1 / (1 + e^((-x^t)_theta)) = sigma(x^T _ theta) The probability of an observation belonging to class 1 given x(vector of features) = Linear regression function on linear transformation of a linear combination of x(vector of features)

Thus, the logistic regression is sometimes known as a generalized linear model since it is a non-linear transformation of a linear model.

Pitfalls of Squared Loss functions for logitisic regression

Can get trapped in a flat region if the surface is not convex Solution: Use Cross-entropy Loss! Empirical Risk for logistic regression when using cross-entropy loss
R(theta) = -(1/n)sum from i=1 to n( y_i * log(logistic regression(X_i^T * theta)) + (1 - y_i) * (1 - logistic regression(X_i^T * theta)))

Advantages of Cross-entropy loss

Loss surface is guaranteed to be convex
More strongly penalizes bad predictions
Has roots in probability and information theory

Logistic Regression

Good for outliers since we wrap our data into a logistic function call
L2 and L1 Loss Motivation: We want to measure the size of the error(difference between prediction and data), and for it to be indifferent to sign.
To measure the difference between two probability distributions, the cross entropy loss is useful
Use a threshold to classify a point on the logistic curve into a graph, this turns our decision rule into a classifier

Evaluating Classifiers

The most basic evaluation for a classifier is accuracy. This is widely used, but in the presence of class imbalance it doesn’t mean much(since if we classify all the pictures as dog and dogs compose of 95% of the dataset our accuracy is 95%)
Types of classification errors: True positive and True negatives(correct when classify observations as positive or negative), False positive(False alarm), False negative(We failed to detect)
Confusion matrix: Gives us true + false positives and negatives for a given classifier

Precision and Recall

Accuracy = (True positive + True negative) / number of data points
- This tells us what proportion of points did our classifier classify correctly
Precision = True positive / (True positive + False positive)
- Of all observations that were predicted to be 1, what proportion is actually 1? Penalized false positives
Recall = True positive / (True positive + false negative)
- Of all observations that were actually 1, what proportion did we predict to be 1? Penalizes false negatives
- Can achieve 100% recall by making the classifier output 1 regardless of input, but makes precision low
Generally, a higher classification threshold for positives equals fewer false positives, increases precision.
On the other hand, a lower classification threshold has fewer false negatives, increases recall.

Visual Metrics

The only thing we can change after getting our model is to play around with our classification threshold
Accuracy vs threshold: Threshold too high = false negatives, Threshold too low = false positives
Precision vs threshold: Threshold increases = fewer false positives, precision tends to increase(not always)
Recall vs threshold: Threshold increases = more false negatives, recall tends to decrease

Other Metrics

False Positive Rate = False Positives / (False Positives + True Negative)
- What proportion of innocent people did I convict
True Positive Rate = True Positives / (True Positives + False Negatives)
- What proportion of guilty people did I convict? aka recall
ROC Curve(Receiver Operating Characteristic): plots the false positive rate vs true positive rate
AUC(Area under curve): Area under ROC curve, best possible = 1, worst possible = 0.5(randomly guessing)

Questions
Can a threshold be a sloped line os is this useless?

Data 100 Variance

Thu, 21 Oct 2021 23:46:37 GMT

Random Variable: A numerical function of a random sample, a statistic. Expectation: Weighted average of the values of X, where the weights are the probabilities of the values Variance: Expected squared deviation from the expectation of X, the units of variances are the units of X squared.

Interpretation of Variance

Var(X) = E[X^2] - (E[X])^2
The main use of variance is to quantify chance error
Chebyshev’s inequality: The vast majority of the probability is around the expectation plus minus a few standard deviations
If X is centered, ie E[X] = 0, then Var(X) = E[X^2]

Linear Transformations

E[aX + b] = aE[X] + b
Var(aX+b)= a^2 * Var(X)
SD(aX+b) = |a|SD(X)
Var(aX+b)=Var(aX)
A shift by b units does not affect spread
The multiplication by a does affect spread

Standardization of random variables

X in standard units = (X-E[X]) / SD(X)
X in standard units measures the number of SDs from expectation
It is a linear transformation of X
E[X_su] = 0, SD[X_su] = 1
E[X_su^2] = Var[X_su] = 1

Variance of a sum -> Covariance

Var[X+Y] = Var[X] + Var[Y] + 2*Covariance
Covariance = 2E[(x-E[X])(Y-E[Y])] =
Covariance is 0 if X and Y are independent
To get right of units for covariance, scale it by the standard deviation to get correlation
Correlation = Cov[X,Y] / (SD[X] * SD[Y])
Uncorrelated random variables is when covariance = 0

I.I.D. sample sum

independent and identically distributed
draws at random with replacement from a population are i.i.d.
E[S_n] = n _ sample mean, Var[S_n]=n(std dev)^2, SD[Sn] = sqrt(n) * std dev

Model Risk

Mean squared error of prediction
Model risk = E[(Y-Y(x))^2], Y(x) is a model that we are using
Chance Error: Due to randomness alone in the new observations
Bias: Non random error due to model being different from the true underlying function

Bias and Overfitting

Overfitting: small differences in random samples which leads to large differences in the fitted model
Overfitting solution: Reduce model complexity
Model risk = std dev^2 + (model bias)^2 + model variance

Questions

Address Translation

Tue, 12 Oct 2021 23:46:37 GMT

Motivation

Address Translation allows for process isolation, interprocess communication(via sharing a common memory region)
Shared code segments(ie sharing a library between programs)
Program initialization(starting program before all code is loaded from memory to disk)
Efficient dynamic memory allocation (only allocate memory for what is needed)
Cache management (change how programs are positioned in physical memory to improve cache efficiency)
Program Debugging (address translation prevents the rewriting of the code section, making things easier to debug)
Efficient I/O (allows data to be safely transferred between user mode apps and I/O devices)
Memory Mapped Files (map files into address space so contents of files can be referenced with program instructions)
Virtual memory (fool the application into thinking that there is more memory than is physically present on computer)
Checkpointing + Restarting (Can restart program from the saved state)
Persistent Data Structures (Region of memory that survives program + system crashes)
Process Migration (can move executing program from on server to another, ie load balancing)
Distributed Storage (Turn network of servers into shared memory using address translation)

Possible Places to put address translation implementation

Specialized hardware in system
trusted compiler, linker, or byte-code interpreter
application translates pointers
hybrid, with address translations in software and hardware

Simple Address Translation

At its core, it is a black box that translates a virtual address to a physical address
Possible implementations could be an array, a tree, a hash table, or pretty much anything

Goals of an address translation black box:

Memory Protection - Limit access of a process to certain regions of memory(ie code)
Memory Sharing - Allow multiple processes to share memory (ie program code, file)
Flexible Memory placement - Put process anywhere to pack physical memory + use caches better
Sparse Addresses - Programs have multiple dynamic memory regions such as the heap for data objects, a stack for each thread, and memory mapped files
Runtime lookup efficiency - Fetching addresses should be fast
Compact Translation Table - Black box should not take much memory
Portability - Works on all types of architectures

Symbol Table and Dynamically Linked Library(DLL)

Problem: Multiprogramming requires address translation for absolute addresses, such as goto the start of the procedure or loading a global variable.
Old Solution: Early OS used relocating loaders, after picking an empty region of physical memory, the loader would modify instructions that used an absolute addresses. The modifications were tracked in the symbol table.
Current Solution: Generate the symbol table, which tracks which values need to be modified when files are assembled together by the linker. Also, commercial operating systems use a dynamically linked library, which links a library into a program when the program first calls into the library.

Flexible Address Translation

The simplest model of memory management is base + bound. Each process has a base and bound to put everything between.

Segments

Segmentation: Instead of keeping only one pair of base and bound, we can instead keep an array of base and bounds.
Segments are contiguous in memory, however, the different segments are not contiguous to each other, this would defeat the point. There will likely be gaps in the segments.
A segmentation fault is a failure to a part of memory outside the legal segments.
Segments are powerful, they are used in x86 and allows two processes to share code as their two segment tables point to the same base and bounds for their segments. They can also be used to share library routines.
Fork: Makes a copy of the parent’s segment table then copy the data there.
Zero on reference: The operating system allocates memory for the heap but only zeroes out the first few kb of the segment. It then sets a bound register to limit the program to the zeroed part of memory. If the program wants more memory, the OS will zero out existing memory before resuming execution.
Cons: There is a large overhead for managing the large number of variable size memory segments. Also, external fragmentation can occur where the small chunks between the segments have enough free space for a new application but the free space is not contiguous.
Questions left: How much memory should we allocate per segment?

Paged Segments

Paged Memory: Memory that is allocated in fixed sized chunks called page frames.
Page Table: Similar to segment table,its entries contain pointers to page frames. No need for a bound since all page sizes are the same.
To share memory between processes, we set the page table entry for each process sharing a page to point to the same physical page frame using a core map - a data structure that tracks info about each physical page frame such as which page table entries point to it.
Data breakpoint: request to stop the execution of a program when it references or modifies a memory location. It is implemented by making that location read only, so that every change can check if the instruction causing the write on read-only memory exception is the breakpoint.
Cons: While physical memory management is easier, virtual address space becomes more challenging since compilers expect the stack to be contiguous(for virtual addresses) and of arbitrary size.
Questions: How big should a page frame be? A larger page frame wastes space if not all memory inside frame is used. A small page frame means there are tons of page frames, meaning that the page table is very large.

Multi Level Translation

Instead of using an array for lookup, we can use a tree instead for the following reasons.
Efficient Memory Allocation: Use a bitmap to store free space
Efficient Disk Transfers: Hardware disks are partitioned into fixed size regions known as sectors, we can make page size a multiple of the sector to simplify transfers, loading, reading, and writing.
Efficient Lookup: Use a translation lookaside buffer to make lookups fast
Efficient Reverse Lookup: Easy to go from physical page frame to set of virtual addresses. This is crucial for the infinite virtual memory illusion.
Page protection and sharing granularity: Each page table entry has its own access permissions

Paged Segmentation

Only two levels in the tree.
Each segment table entry points to a page table, which points to the memory backing that segment.

Multi Level Paged Segmentation

Use a Global Descriptor Table, aka a segment table.
For 32 bit systems, First 10 bits-page directory, Second 10 bits-second level page table, final 12 bits-offset within a page
For 64 bit systems, only the first 48 bits are used

Portability

Motivation

Main memory density is increasing by almost a bit per year, for a multi-level page table to map all of memory, an extra level of the page table is needed every decade to keep up with main memory size.

For things like copy-on-write, zero-on-reference, fill-on-reference, we need a different view for hardware and virtual address space.

We also need:

List of memory objects: We need to known which memory region represents program code, library code, shared data, copy-on-write, or memory-mapped file.
Virtual to physical translation: In exceptions and system call parameter copying, the kernel needs to translate virtual to physical addresses to check if invalid page is truly invalid. (it could be just not yet loaded or doing some copy-on-write magic)
Physical to virtual translation: The core map. Which processes map to a specific physical memory location so that when the kernel updates a page status, it also updates every page table entry that refers to that page.
OSX uses a hash table rather than a tree for software translation data. This hash table is known as an inverted hash table

inverted page table entry = {
  process or memory object ID,
  virtual page number,
  physical page frame number,
  access permissions
}

Efficient Address Translation

Motivation

Page tables, Segmentations, etc. all require multiple extra memory references before reaching the physical memory location. This greatly increases the overhead per instruction.

Thus, we can use a cache to improve performance. A cache stores certain data to increase the speed of future requests, usually by taking advantage of temporal locality or spatial locality.

Translation Lookaside Buffer (TLB)

Problem: Instructions are usually next to each other both in virtual memory and physical memory, it would be pointless to load instructions one at a time from physical memory since they are on the same page in physical memory.
TLB: A small hardware table that contains results of a recent address translation. Each entry maps a virtual page to a physical page.

TLB entry = {
          virtual page number,
          physical page frame number,
          access permissions
            }

On a lookup, the TLB hardware checks all of the entries against the virtual page. If there is a match, the TLB returns the physical address. (TLB hit) If none of the entries match, the hardware does the full address translation and is added to the TLB, usually replacing the least recently used entry.

The cost for a TLB lookup is
Cost (address translation) = Cost (TLB lookup) + Cost (full translation) × (1 - P(hit))

Superpages

Superpage: Set of contiguous pages in physical memory that map to the same contiguous region in virtual memory. For example, two 4 Kb pages that are next to each other form an 8 Kb superpage.
Pros: Reduces the number of TLB entries needed since one superpage can map many pages.
Cons: Complicates the OS by requiring the system to allocate chunks of memory in different sizes.
Implementation: Each entry in the TLB has a flag signifying whether the entry is a page or superpage. When matching against a superpage, the TLB ignores the offset within a superpage and only considers the most significant bits.
Uses: When the computer display is being withdrawn, a normal TLB isn’t big enough to hold all the entries. Large Matrices in scientific code can be sped up via superpages.

TLB Consistency Issues

Process context switch: The virtual addresses of the old process are no longer valid. Otherwise, the new process would be reading the old process’ data structures, causing it to crash or even worse, reading sensitive info such as passwords stored in memory.

We need to get rid of the TLB’s copies of the old process’ page translations and permission. We can flush the TLB(discard all contents), or flush the tagged TLB, where entries contain the process ID that produced each translation.

tagged TLB entry = {
  process ID,
  virtual page number,
  physical page frame number,
  access permissions
}

Permission Reduction: When the OS modifies an entry in a page table, the special purpose hardware keeps the cached data consistent. Software is needed so a single modification can affect multiple TLB entries and the OS ensures the TLB does not contain an incorrect mapping.

When adding permissions to virtual address space, the TLB can be left alone. When reducing permissions to a page, the kernel checks if the TLB has a copy of the old page. If it does, the kernel must update permissions.

TLB Shootdown: Multiprocessors might have a cached copy of a translation in its TLB. Thus, whenever a page table entry is modified, the corresponding entry in every processor’s TLB must be discarded. This requires interrupting the processor. (very expensive!)

Virtually Addressed Caches

Stores a copy of the contents of physical memory, indexed by the virtual address.
On modern processors, these are split up located near the processor core, one for instruction lookups and one for data.
Process context switch, permission reduction, and shootdown are all issues with virtually addressed caches.
Memory address alias: Different processes have different virtual addresses that point to the same memory location. Each process has its own TLB entry for that memory, and the virtual cache stores a copy of the memory for each process. However, when a process modifies its own copy, how does the system know to update the other copy?

The solution is to store the physical address with the virtual address in the virtual cache.

Physically Addressed Caches

Consulted as a second level cache after the virtually addressed cache and the TLB but before main memory.
Once the physical address of the memory address is located, we look at the Physically Address cache to see if there is a match. Thus we can avoid going to main memory

Software Protection

Naive software only protection: A machine code interpreter implemented in software simulates each instruction and only runs it if the addresses in the page table are permitted. (this is extremely slow!)

Goals of software protection:

Simplify Hardware: Is it possible to have eliminate hardware address translation to limit hardware complexity and runtime overhead from computers.
Application level protection: A browser wants to protect itself and the OS from malicious code in applications it runs.
Protection inside the kernel: Running third party code inside the kernel is dangerous, we need to protect the code in software.
Portable Security: Protection as a part of the runtime system allows us to download and run different applications without feat that the application will corrupt the OS.
Thus, we want a sandbox to execute untrusted code so that it can work without harming the rest of the system.

Single Language OS

Restrict all applications to be written in a single, carefully designed programming language(like solidity -> EVM?)
For example, Javascript in modern web browsers is sandboxed. If a JavaScript program attempts to call a procedure that does not exist or reference arbitrary memory locations, the interpreter causes a runtime exception and stop the program.
Cons: Using a single language based software protection relies on trusting the interpreter. Since interpreted code is slow, many interpreted systems put their functionality into system libraries that are compiled into machine code and run directly on the processor. A JavaScript program could cause a library routine to overwrite the end of a buffer for a buffer overflow attack.

Thus, many systems use both software and hardware protection.

Language Independent Software Fault Isolation

Motivation: Single Language OS requires trusting every compiler for every possible language. Is there a way we can isolate application code in a language independent fashion without using hardware?
Implementation: Use a base + bound method. First, insert machine instructions into executable and check that each address is in base + bound. Second, use control + data flow analysis to remove checks that are not strictly necessary for the sandbox to be correct.

Checking that each address is in base + bound

For store + load instructions, just check to make sure that the addresses are within the correct region of data.
For indirect branch instructions, make sure the program cannot branch outside of the sandbox except for predefined entry + exit points.

Control + Data flow analysis

This allows us to eliminate extra inserted instructions if they are not needed. Possible optimizations include:

Loop invariants: If a loop strides through memory, we can use math to ensure all memory accesses in loop will be in the protected region.
Return Values: If static analysis can prove a procedure does not modify the return program counter on the stack, the return can be made safely.
Cross procedure checks: If code analysis can prove a parameter is always checked before it is passed as an argument to a subroutine, it does not need to be checked inside the subroutine.

Sandboxes via intermediate code

Instead of creating a sandbox for x86 or ARM code, we can check a form of intermediate code and run that into the sandbox(kind of like a virtual machine)

For example, the JVM can be used to safely contain java code.

Future

Very large memory systems: Ever increasing memories will require larger TLBs to hide the depth of the lookup trees.
Multiprocessors: TLB consistency will be more and more expensive as more processors will be added.
User level sandboxes: Each application has its own sandbox which is enforced in hardware.

Questions

Queueing Theory

Tue, 12 Oct 2021 23:46:37 GMT

Queueing Theory

Response time depends non-linearly on the rate that tasks arrive at a system. For all intents and purposes, we assume that the system is work-conserving(all tasks that arrive are eventually serviced). Secondly, we are using FIFO scheduling.

Definitions

Server: A server is anything that performs tasks. ie web server, processor in a client machine, cashier in supermarket
Queueing delay(W): The total time a task must wait to be scheduled. aka wait time
Number of tasks queued(Q): number of tasks on the queue
Service time(S): The time to complete a task assuming no waiting. aka execution time.
Response time(R): Queueing delay(W) + Service time(S)
Arrival rate(lambda): Average rate at which new tasks arrive
Arrival process: When tasks arrive and whether arrivals are bursty or spread evenly over time.
Service Rate(u): Number of tasks the server can complete per unit time when there is work to do.
Utilization(U): Fraction of time the server is busy. U = lambda / u if lambda < u, else U = 1 if lambda >= u
Throughput(X): Number of tasks processed by the system per unit of time. X = U * u We can rewrite this as X = lambda if U < 1, else X = u if U=1
Number of tasks in the system(N): The number queued + number receiving service. N = Q + U

Little’s Law

Conditions: Applies to any stable system where the arrival rate matches the departure rate. It relates the average throughput, response time, and number of tasks in the system. N = X * R

Response Time vs Utilization

Higher Utilization normally implies higher queueing delay and higher response times.
Higher Utilization also increases the risk of overload(where requests grow faster than they can be processed, thus queues and wait times grow without bound)
Can predict average response time from the arrival process and service time but it is difficult
Goal: make tradeoffs between higher utilization and better response time. In the fast, higher utilization was favored since computers were expensive, now, it is better to have a better response time since computers are cheaper than humans.
Best case(Evenly spaced arrivals): Suppose we have a set of fixed sized tasks that arrive equally spaced from one another. As long as the rate at which tasks arrive is less than the rate at which the server completes the tasks, there is no queueing since the server finishes the previous request just in time for the next arrival.
Suppose arrivals come in every 1 ms and the server can process each task in 1ms. If at some point a single extra request arrives, every other request will need to wait 1ms for that request.
Worst case(Bursty arrivals): Suppose a group of tasks arrive exactly at the same time. The average wait time increases linearly as more tasks arrive together.
Exponential arrivals: A useful model for understanding queueing behavior is the exponential distribution. This can describe the time between tasks arriving and the time it takes to service each task. An exponential distribution of a continuous random variable with a mean of 1 / lambda has the probability density function `f(x) = lambda * e ^ -(lambda * x) The exponential distribution is memoryless, ie no matter how long we have already waited for the event, the likelihood of that event occurring remains the same.

What if Questions

What if we changed the scheduling policy from FIFO to something else?

If the distribution of arrivals or service times is less bursty than an exponential(ie evenly spaced or Gaussian), FIFO will deliver nearly optimal response times. Round robin will be worse than FIFO
If task service times are exponentially distributed but individual task times are unpredictable, the average response time is exactly the same for Round Robin as for FIFO.
If task lengths can be predicted and there is variability in service times, Shortest Job first can improve average response times, particularly if arrivals are bursty
Real world systems usually have more bursty arrival times than the exponential distribution, their distributions are known as heavy tailed distributions since it has more long tasks. SJF is better than Round Robin which is better than FIFO for heavy tailed distributions
SJF can improve average response times but increases response times for long tasks since long tasks will only complete immediately before an idle period. As utilization increases, the idle periods become increasingly rare.

What if workloads vary with the queueing delay?

Arrival rates and service times are not always independent of queueing delay. For example, a online store becomes overloaded and slow during the holiday shopping season. Rather than continuing to browse, some customers may get fed up and leave, reducing the arrival rate of requests for individual web pages.

What if we had multiple servers?

Banks have multiple tellers, grocery checkouts have multiple cashiers. Clearly, there are often efficiency gains from having separate queues.
On the other hand, a single queue is always better than separate queues when focusing on average response time because of variations in how long each task takes to service. One queue could be idle while another queue has multiple tasks

If a processor is 90% busy serving web requests and we add another processor to reduce the load, how much will that improve average response time?

Unfortunately, there is not enough info to say since each web request needs processing time and disk I/O and network bandwidth.
If arrival times and service times are exponentially distributed and independent of the system response time, the overall response time is the sum of the response times of the components `response time = sum for all arrival and service times(S / (1 - U))

What are some conclusions we can draw from real world systems?

Almost all real world systems exhibit some randomness in their arrival process or service times
Response time increases with increased load
System performance is predictable across a range of load factors if we can estimate the average service time per request.
Burstiness increases average response time.

Real Time Scheduling and Overload Management

Tue, 12 Oct 2021 23:46:37 GMT

Real Time Scheduling

Motivation

The software that runs the brakes in your car or autopilot on an airplane must work when called for, you can’t hit the brakes to stop after you have crashed your car. The change in value of a specific task over time is shown in the below graph.

Three techniques to increase likelihood that threads run before their deadline

Over provisioning: Make sure real-time tasks only use a fraction of the system’s total processing power. This is like taking 2 units when you know you can handle 16 units.
Earliest Deadline First: Choose the task that has the earliest deadline first. However, this won’t always work with the given tasks due to it requiring both I/O and computation. Thus, always break tasks down into shorter units and use the deadlines of the shorter units.
Priority Donation: The problem of priority inversion occurs when a higher priority thread must wait for a lower priority thread to complete its work. This can be solved via Priority Donation, where a higher priority thread that is waiting for a lower priority thread donates its priority to the lower priority thread, allowing it to run and finish using that resource.

Questions to ask

For priority donation, do you actually change priority values by subtracting from the priority of the higher priority thread and adding to the priority of the lower priority thread or do you just set the lower priority thread’s priority to the priority of the higher priority thread?

Virtual Memory and Address Translation

Tue, 12 Oct 2021 23:46:37 GMT

General Address Translation

Motivation

We want to store addresses in a different place than the computer expects. We can then use address translation to move between our what the computer thinks is storing stuff and the actual stuff.

Useful things we can do with Address Translation

Process Isolation: Protect the operating system kernel and other applications against buggy or malicious code. This is done by limiting memory references by applications so that some applications cannot access that memory. Also, address translation can be used by applications to construct sandboxes for third party extensions.
Interprocess Communication:

Segmentation

Translation happens on every instruction fetch, load, or store.
Each virtual address space has holes, leading to efficient segmentation for sparse address spaces
When is it OK to address outside valid range? This is how the stack grows
What if not all segments fit in memory? One solution is swapping, where we save the segment of the current process to disk then switch to the next process. A better way is to only keep parts of memory we need.

Problems with Segmentation

Must fit variable sized chunks into physical memory, there could be multiple gaps but no segment is small enough to fit in any of the gaps.
Segmentation may move processes multiple times to fit everything.
Limited options for swapping to disk

Solutions to Fragmentation

Paging - Allocate physical memory in fixed size chunks where every chunk of physical memory is equivalent. The pages should be smaller than segments

Implementation: Use a page table per process, which resides in physical memory and contains a physical page and permission for each virtual page(valid bits, read, write, etc.) The offset from virtual address is copied to physical address, the virtual page number is all the remaining bits.

Usage: Page sharing is used in the kernel region of every process which has the same page table entries. Also used when different processes run the same binaries, user level system libraries, and shared memory segments between different processes. (Can actually share objects between processes as long as the page is mapped into the same place in the address space)

Page Table

Page Table Context Switch: What need to be switched on a context switch? Page table pointer and limit.
Pros: Simple memory allocation, easy to share Cons: Poor when address space is sparse, table is really big

Implementation: Page tables are maps from Virtual Addresses to Physical Addresses. A simple implementation is a very large lookup table, but can also be implemented via Trees or Hash tables

What is in a Page Table Entry(PTE):

Pointer to the next level page table or actual page
Permission bits: valid bits, read-only, write-only, read-write
Invalid PTE can imply region of address space is actually invalid or page/directory is just somewhere else other than memory

Using the Page Table Entry(PTE):

Used in demand paging: keeps only active pages in memory, place others on disk and mark PTEs as invalid
Used in Copy on Write: Unix fork gives copy of parent address to child. Address spaces disconnected after child created. Implementation - make copy of parent’s page tables, mark entries in both sets of page tables as read-only, page fault on write creates two copies
Used in Zero Fill on Demand: New data pages stay zeroed, mark PTEs as invalid, page fault gets zeroed page.

Multi-Level Translation

Pros:

Only need to allocate

Paging - Caching and TLBs, Demand Paging

Problem: A single level page table can only have so many addresses, how do we get more space?

Solution: Use a multilevel page table, where the first page table points to the second page table which points to the third page table

Pros: Only need to allocate as many page table entries as we need for applications.(Sparse address spaces are easy) Easy memory allocations, Easy Sharing at segment or page level(needs additional reference counting)

Cons: one pointer per page, page tables need to be contiguous, two or more if more than 2 levels lookups per reference(very expensive!)

Problem: Given a virtual address, how to map to physical address? Solution: Use a hash table aka Inverted Page Table whose size is independent of the virtual address space. The size is directly related to the amount of physical memory.

Cons: The complexity of looking things up in hardware in a hash table makes things hard. There will also be cache issues since each thing in the hash table is far from everything else. Total size of page table = number of pages used by program in physical memory

Address Translation Comparison

Technique	Advantages	Disadvantages
Simple Segmentation	Fast context switching since the segment map is maintained by the CPU	External Fragmentation
Paging(Single-Level)	No external fragmentation, fast + easy allocation	Large table size, internal fragmentation
Paged Segmentation	Table size ~ # of pages in virtual memory, fast + easy allocation	Multiple memory references per page access
Multi-Level Paging	Table size ~ # of pages in virtual memory, fast + easy allocation	Multiple memory references per page access
Inverted Page Table	Table size ~ # of pages in physical memory	Complex Hash function required, no page table cache locality

How is Translation Accomplished? - The Memory Management Unit (MMU)

The MMU must translate virtual addresses to physical addresses for every instruction fetch, every load, and every store.

Implementation:

In the first level of the page table, Read PTE from memory, check that its valid, merge address
In the second level page table, read and check the first level, then read and check and update the PTE
Repeat for the n levels in the page table

Problems: This is really slow, since a single load/store will require n reads, n being the number of levels in the page table.

Solution: Caching - take advantage of the principle of locality to present the illusion of memory as the cheapest technology.

Temporal Locality: Keep recently accessed data items closer to processor. Spatial Locality: Move contiguous blocks to the upper levels.

How to make Address Translation Fast?

Cache the results of recent translation(the results of the MMU). Cache the PTE extra info, physical frame number, using the virtual page number as the key

Translation Look Aside Buffer (TLAB): The name of the cache we use for translations. The TLAB records recent Virtual Page Number to Physical Frame number translation.

If the Virtual Page Number is present, return the Physical Address.
If not, the use the MMU to find and save the result of the physical address.
- Instruction accesses spend a lot of time on the same page since their accesses are sequential
- Stack accesses have definite locality of reference,
We can have a multilevel TLAB with different size/speeds

Sources of Cache Misses

Compulsory: Cold start or context switch - the first access to a block. Not much we can do here. Capacity: Cache cannot contain all blocks accessed by the program. Solution - increase cache size. Conflict/Collision: Multiple memory locations mapped to the same cache location. Solution - increase cache size or increase associativity. Coherence(Invalidation): Other processes eg I/O updates main memory, while the copy of main memory in the cache is not updated since it was some other process that modified main memory. THIS IS VERY COMMON IN MULTI-PROCESS PROGRAMS! False Sharing: Common cause of coherence misses.

How is a Block found in a cache?

Block: Minimum quantum of caching. Data Select field is used to select data within blocks. Many caching applications don’t have data select fields. Index: Used to Lookup Candidate in a Cache. Tag: Used to identify actual copy.

Direct Mapped Cache: Cons: Conflict Misses. Since data is stored 1:1 it is likely there is already something in that slot and we will need to evict that old piece of data

N-way set associative: N entries per cache index. Fully associative cache: Every block can hold any line, address does not include a cache index. Compare cache tags of all cache entries in parallel.

Replacing blocks on a cache miss

For Direct Mapping, there is only one possibility. For Set associative or Fully associative, do it randomly or evict the least recently used block

What happens on a cache write?

Write through: The information is written to both the block in the cache and to the block in the lower level memory. Pros: read misses cannot result in writes Cons: The processor is held up on writes unless writes are buffered. Write back: The info is written only to the block in the cache. The modified cache block is written to main memory only when it is replaced. Pros: repeat writes not sent to DRAM processor not held up on writes Cons: More complex, read miss may require writeback of dirty data

Physically Indexed vs Virtually Indexed Caches

Physically Indexed Caches (More common)
- Address handed to cache after translation, the page table holds physical addresses
- Pros: Every piece of data has a single place in the cache and the cache stays unchanged, thus a cache remains unchanged through a context switch
- Cons: The TLB is in the critical path of the lookup
Virtually Indexed Caches(Less common)
- The address is handed to the cache before translation, the page table holds virtual addresses
- Pros: The TLB is not in the critical path of lookup
- Cons: The same data could be mapped in multiple places of cache and may need to flush cache on context switch.

What TLB Organization Makes Sense?

Requirements: The TLB needs to be really fast since it is in the critical path of memory access. Also, there needs to be very few conflicts, otherwise miss times will be extremely high because we need to traverse the page table.

TLB lookup is serial with cache lookup, thus. the speed of the TLB can impact the speed of access to the cache

Demand Paging

Questions to ask

What is 0x12345678 123456 is what? 78 is the offset in the page table?
Caching in industry? What is it used for?
How do we know that the main memory is different from the cached memory when detecting coherence misses? <—Apparently very hard to do :( take CS152
Why isn’t LRU implemented using a heap

Multiprocessor Scheduling

Mon, 11 Oct 2021 23:46:37 GMT

Bankers Algorithm

Motivation: Avoid deadlocks when working with threads
Idea: When a thread requests a resource, the OS checks if it would deadlock
- If so, wait for other threads to release resources
- If not, grant the resource right away
Implementation: State max resources that thread needs. Allow thread to proceed if available resources - requested resources >= max remaining needed by any thread
Keeps system in a safe state

Precise Exceptions

At the time that an exception handler begins execution, there is a well-defined instruction address in the instruction stream for which all prior instructions have committed their results and no following instructions (including the excepting one) have modified processor state.

Meltdown Security Flaw

Many processors allowed timing windows in which illegal accesses could be performed speculatively and made to impact cache state – even though the speculatively loaded data was later squashed in the pipeline and could not be directly used.

Earliest Deadline First(EDF) scheduler

The scheduler to use for realtime systems. Runs the task with the earliest deadline first.

Beladys anomaly

Increasing number of page frames increases number of page faults for FIFO, Random Page Replacement

Multiprocessor Scheduling

Mon, 11 Oct 2021 23:46:37 GMT

Motivation

Most computers are multiprocessors. How do we use multiple cores to run sequential tasks? How do we adapt scheduling algorithms for parallel applications?

Sequential Scheduling - Naive Approach

Create a single multi-level feedback queue(MFQ) with a lock so that only one processor can read/write from it at once. Each idle processor takes the next task off the MLFQ and runs it.

Potential Problems:

Contention for MFQ lock: Many Processes are fighting over the MFQ lock
Cache Coherence Overhead: Processes need to fetch the current state of the MFQ from the cache of previous processors to hold the lock. The cache for the current processor is likely not going to be of much use since most of it will need to be updated. Also, you are fetching/updating the cache data while you are holding the MFQ lock, causes further delays.
Limited Cache Reuse Since threads run on the first available processor, that processor probably doesn’t have the necessary data in its cache.

Commercial Operating System Scheduler

Use a separate copy of the MFQ for each processor to maximize cache reuse.

Affinity Scheduling: Once a thread has been scheduled on a processor, return it to the same processor when it is re-scheduled to maximize cache reuse. Each processor has its own copy of the queue to run threads from, only rebalance the queue length differences are large enough to compensate for the cache reload for the threads we move to different queues. Unfortunately locks are still necessary since per processor data structures must be protected by locks.

Definition-Preempting a thread/process: A thread is pre-empted when a higher priority thread is placed on the ready queue and the higher priority thread runs instead of the current thread.

Oblivious Scheduling

Each thread is time sliced to available processors with no attempt to ensure threads from the same process run at the same time.

Definition-Critical Path: The minimum sequence of steps for the application to compute the result.
Definition-Spin-then-wait: If a lock is busy, the waiting thread spin-waits briefly for it to be released, and if it still busy, it blocks and looks for other work.
Pros: Great if the lock is held for short periods of time.

Potential Problems:

Critical Path Delay- Preempting a thread on the critical path will slow down the end result.
Bulk Synchronous Delay- A common design pattern is to split work into equal sized chunks, do the work on the chunks, then communicate the work done to the next stage of computation(Think Google mapreduce). However, at each time step the work done is limited to the slowest processor, since all the chunks need to be finished before moving onto the next stage computation.
Producer-consumer Delay- Another common design pattern is the producer-consumer. The results of one thread are fed to the next thread, of which the results are fed to the next thread, etc. Preempting a thread in the middle of a producer-consumer chain will stall all of the processors in the chain.
Preemption of lock holder- In spin-then-wait strategies for locks, the lock holder can be preempted-other tasks will spin-then-wait until the lock holder is rescheduled.
I/O- If a read/write happens in the kernel, the thread blocks. To use the processor when the thread is waiting, the application needs to have more threads and processors so the extra threads can run while one is waiting. However, if the thread does not block(ie a read when the file is in memory), then we cannot do time slicing since we have more threads than processes.

Gang Scheduling

Approach: Schedule all of the tasks of a program together. The application groups some work into a number of threads, and the threads run together or not at all. If the OS needs to schedule a different application or there aren’t enough idle resources, it preempts all of the processors of an application to make room

Implementation in Commercial Operating Systems: The application pins each thread to a specific processor and mark it to run with high priority. The system uses the rest of the processors to run other applications without interfering with the primary application.

Problems: As the number of processors increase, performance grows slower since some applications scale linearly with the number of processors, other achieve diminishing returns.

Thus, it is usually more efficient to run two parallel programs with half the number of processors vs time slicing the two programs and gang scheduling.

Definition-Space Sharing: Allocating different processors to different tasks. Efficient since it minimizes processor context switches. The problem occurs when the not all tasks start and stop at the same time, how do we know how many processors to use if the number changes over time?

Scheduler Activations

Make the assignment and re-assignment of processors to applications visible to applications. Each application has an execution context(aka scheduler activation). Each application is explicitly informed when a processor is added to the allocation or taken away via an upcall.

This leaves the question of how many processors should each process be assigned? This is an open research problem, there is a fundamental tradeoff between average response time and fair allocation of resources among different applications. In multiprocessors, average response time can be improved by giving extra resources to parallel interactive tasks as long as it does not cause long-running compute intensive parallel tasks to starve for resources.

Remaining Questions

What is the difference between MLFQ and MFQ? Are they the same thing just abbreviated differently?
Even if one of the waiting processors picks up the pre-empted task, a single preemption can delay the entire computation by a factor of two, and possibly even more with cache effects. Why is this the case with Bulk synchronous delay for oblivious scheduling?
Preempting a thread in the middle of a producer-consumer chain will stall all of the processors in the chain. Shouldn’t this be stall all of the processors after this thread in the chain?
Performance as a function of the number of processors, for some typical parallel applications. Some applications scale linearly with the number of processors; others achieve diminishing returns. Why is this the case?

Data 100 Modeling

Sun, 10 Oct 2021 23:46:37 GMT

Motivation

Predict unknown values based on known values

Modeling Process

Choose a Model:** Constant model Choose a Loss Function:** Squared Loss, Absolute Loss Minimize average loss across entire dataset to determine optimal parameters

Correlation

Correlation Coefficient (r): Measures the strength of the LINEAR association between two variables

r is unitless and ranges between -1 and 1
- if r = 1, x = y and all points fall exactly on the line
- if r = -1, -x = y and all points fall exactly on the line
- if r = 0, there is no linear association between x and y
r says nothing about causation or non-linear association. Remember correlation does not imply causation!

r = average of the product of x and y, both measured in standard units xi in standard units = (xi - x) / o_x

Covariance: r * stdx * stdy

Simple Linear Regression

Motivation: Want to predict value of y for and given x. A naive attempt for getting children heights given parent heights is to compute the average value of y for each x value, then use those as predicts.

Simple linear regression: y = a + bx To determine optimal a and b, choose a loss function. If the loss function is squared loss, the objective function is mean squared error(MSE).

To solve for the optimal parameters, we use the objective function and minimize the mean squared error by hand using calculus.

b = r * (oy/ox) a = ybar - b * xbar This gives us parameter estimates for x and y.

Loss Surfaces

Usually 3d, with axes being a, b and loss(y = a + bx)

Model Interpretation

Slope - measured in units of y per unit of x New data needs to be similar to original data - you cannot predict the weight of a chihuahua’s weight given a model for golden retrievers Visualize, then quantify - watch out for anscombe’s quartet

Terminology

Names for the x variable:

Feature
Covariate
Independent variable
Explanatory variable
Predictor
Input
Regressor

Names for the y variable:

Output
Outcome
Response
Dependent variable

Adding independent variables

Use a weighted sum of coefficients and input variables.

Evaluating Models

Look at Mean Squared Error(MSE) or Root Mean Squared Error(RMSE)
Look at the correlations
Look at a residual plot

Root Mean Squared Error: Square root of the mean squared error. RMSE is in the same units as y. A lower RMSE indicates more accurate predictions. It is impossible to lower the RMSE just by adding features using the same data

R squared: Used to measure the strength of the linear association between our actual y and predicted y. aka coefficient of determination.

R^2 = variance of fitted values / variance of y

Ordinary Least Squares

Multiple regression using matrix multiplication

Multiple regression is of the form y = theta_0 + theta_1 * x_1 + theta_2 * x_2 + ... + theta_p * x_p We can restate this as a dot product y = x^T * theta

Design Matrix

Motivation: the mean squared error involves all observations at once, it would be nice to express our model in terms of all observations, not just one. We can put them into a design matrix.

Rows: Correspond to observations. e.g. all features for data point 3 Columns: Correspond to features. e.g. feature 1, for all data points

Residuals

Residuals are the difference between an actual and predicted value, in the regression context. We use the letter e to denote residuals, e_i = y_i - yhat_i

The mean squared error is equal to the mean of the squares of its residuals. We can stack all n residuals into a vector, called the residual vector. residual vector = true y values - predicted y values

Residuals are orthogonal to the span of X. If our model has an intercept term(when our design matrix has a column of all 1s)

The sum and mean of the residuals is 0
The average true y value is equal to the average predicted y value

Residual Plots:

With simple linear regression with only 1 independent variables, we plot residuals vs x
In the general case, use residuals on y axis vs fitted values on x
A good residual plot has no pattern, if there is a curve, this is a sign that transformations or additional variables can help
A residual plot should have a similar vertical spread throughout the entire plot. If it doesn’t there are probably issues with the accuracy of the predictions

Unique Solutions

There is always at least one model parameter that minimizes average loss.
Constant models with a squared loss: a unique solution always exists
Simple linear model with a squared loss: Any non constant value has unique mean, SD, correlation coefficient
Constant model with absolute loss: Unique when there is an odd number y values, if there is an even number of y values, there are infinitely many solution.

Invertability of X transpose * X

Invertible iff it is full rank
X transpose * X and X have the same rank
Thus, X^T * X is invertible iff X has rank p + 1 (full column rank)

Real World Example - Fairness in Housing Appraisal

Situation

The cook county assessor’s office is in charge of assessing property values in order to determine property taxes.

Problem

The biased property value assessment resulted in a regressive tax, where rich people paid less and poor people paid more. In addition, rich people appealed more often than poor people, resulting in an even greater reduction of property tax.

Solution

Ask a Question: What do we want to know? How to fairly value things for tax purposes. What are our metrics for success? Have both fairness and transparency in projections.
Data Acquisition and Cleaning: What data do we have and what do we need? Housing Sales data between 2013-2019, Property Characteristics-ie age, bedrooms, baths, etc.How will we sample more data? Is our sample representative?
Exploratory Data Analysis and Visualization: What attributes are most predictive of sales price? Which are potentially problematic? Is the data predictive of sales price?

Takeaways

Accuracy is a necessary, but not sufficient condition of a fair system.
Fairness and transparency are context-dependent
Learn to work with contexts and consider how your data analysis will reshape them
Keep in mind the power and limits of data analysis?

Probability and Generalization

Random Variables

Random Variable: Represents a numerical value determined by a probabilistic event.

Probability Mass Function

The distribution of a random variable X provides the probability that X takes on each of its possible values(discrete)
The probabilities for all possible values of random variable X in a Probability Mass Function must sum to 1
Each individual probability for a given value X must be between 0 and 1. Joint Distributions: Probability of two or more random variables taking on a specific set of values. Ie P(X=0, Y= 10) = (0.5) ** 10 for coin flips where X is heads and Y is tails Marginal Distribution: A way to go from the joint distribution to the distribution for a single variable. Ie consider all possible values of Y that can simultaneously happen with X and sum over all of the joint probabilities.
∑y∈Y P(X=x,Y=y) = P(X=x)

Independent Random Variables: Any two random variables are independent if and only if knowing the outcome of one variable does not alter the probability of observing any outcomes of the other variables.

Expectation and Variance

Expectation

The long run average of a random variable, also known as the expected value or expectation of a random variable.
E[X] = ∑ x∈X x ⋅ P(X=x)

Linearity of Expectation:

Use when working with linear combinations of random variables. This holds true even when the random variables are dependent on each other.
E[X+Y] = E[X] + E[Y]
E[cX] = c * E[X]
E[X−Y] = E[X] − E[Y]

However, E[XY]=E[X]E[Y] is only true when X and Y are independent random variables.

Variance

The variance of a random variable is a description of the variable’s spread, or how far values are apart from each other.
Var(X) = E[(X − E[X])**2]
Var(X) = E[X**2] − (E[X])**2
Var(aX + b) = a**2 * Var(X) Is true if X is a random variable
Var(X + Y) = Var(X) + Var(Y) Holds true if X and Y are independent

Covariance

If the covariance is positive, the random variables are positively correlated(ie move in the same direction for stocks). If the covariance is negative, the random variables are negatively correlated. A covariance of 0 indicates the variables are independent.

Cov(X,Y) = E[(X − E[X]) * (Y − E[Y])]
Cov(X,Y)= E[XY] − E[X]*E[Y]

Risk

Risk: Statistical risk is known as the expected loss, or the expected value of the model’s loss on randomly chosen points from the population.

R(θ) = E[(X − θ)**2]
To minimize risk, use R(θ) = E[(X − E[X]) ** 2] + (E[X] − θ) ** 2
R(θ) = Bias + Variance = (E[X] − θ) ** 2 + E[(X − E[X]) ** 2]
A low variance means the random variable will likely take a value close to θ, while a high variance means the random variable will take a value far from θ

Empirical Risk Minimization

Since calculating the expected value of X requires complete knowledge of the population, since expected value is defined as probability X takes a specific value * that specific value.
We can use a large random sample instead of the population when calculating the expected value of X
Thus we can approximate E[X] ~ mean(x)
Therefore, the empirical risk is the risk from using the large random sample instead of the population

Multiple Linear Regression

Questions to Ask

Uniprocessor Scheduling

Sun, 10 Oct 2021 23:46:37 GMT

Motivation

Given multiple tasks, how do we decide which one to do first?

Uniprocessor

Everything discussed below is work conserving - ie never leaves processor idle

Fifo/FCFS

Perform each task in the order it arrives. Run each task until it finishes

Pros: No overhead, fair since every task waits their turn, simple
Cons: If a long task is scheduled before a short task, convoy effect dramatically raises average wait time. Average response time depends on task size distribution

Shortest Job First (SJF) / Shortest Remaining Time First (SRTF)

Perform the shortest job first. Run each task till it finishes

Pros: Optimal strategy
Cons: Impossible, Increases the variance of average response time, Starvation can occur due to too many short jobs coming in which blocks off the longer jobs on the queue

This shows that there is a tradeoff between reducing average response time and the variance of average response time

Round Robin

Tasks take turns on the processor, if the task is not completed in the given time quanta, it goes to the end of the queue
Short time quanta: Too much time spent context switching, no work will be done Long time quanta: Tasks spend a long time waiting for a turn Infinite time quanta: Same as FIFO Zero time quanta: SJF

Simultaneous multithreading(SMT)/Hyperthreading: Switch between two processors on a cycle by cycle basis

Pros: No starvation
Cons: Context switching takes time, need to evict cache in addition to registers of processor

Max min fairness

Iteratively maximize the minimum allocation given to a particular process until all resources are assigned

If all processes are compute bound, this turns into round robin.

If some processes don’t use entire share of processor becasue they are short running or IO bound, give those processes their entire request and redistribute the unused portion to remaining processes. Recursively distribute unused portions until nothing left. When no remaining requests can be fully satisfied, divide the remainder equally among all remaining processes.

Pros: Good for IO bound tasks, no starvation
Cons: Computationally expensive

Multi Level Feedback

Extension of Round Robin but uses multiple round robin queues, each with a different priority level and time quantum. Higher priority level queues go before lower priority queues, higher priority levels have shorter time quanta than lower levels.

Meant to favor short tasks over long ones. New tasks enter at top, drops to lower queue when it uses up its time quanta. If task yields the processor bc it is waiting for IO, it stays on the same level, and is removed when task completes.

Increases Priority when process receives less than its fair shares, reduces priority when process has more than fair share

Goals: Run short tasks quickly, Low Overhead, no starvation, fair, defers maintenance tasks

Bridges

Sun, 22 Aug 2021 23:46:37 GMT

Quick Primer on 4 different bridges

Please note that some of this will be outdated. This also does not include layerzero, wormhole, synapse, abacus, NEAR’s rainbow bridge, and Ren, plus countless multisig bridges.

This is a brief intro to Axelar, Celer, Gravity, and Nomad.

Axelar Network

Summary:

Axelar network is built from the ground up using Cosmos SDK to interoperate between Cosmos chains, EVM chains, Bitcoin, and other PoS and PoW chains. Satellite is the cross-chain asset transfer app built on top of the network and has amassed over $37m TVL and 100k cross-chain transactions since launching in February. It has also recently become the official canonical bridge for Osmosis and plans to further expand its reach into the Cosmos and EVM ecosystem.

Satellite currently supports Avalanche, Polygon, Terra, Osmosis, Moonbeam, Fantom, Ethereum and Cosmos Hub with plans to support all major EVM chains over the next couple months.

Security:

The Axelar network isolates functionalities in modules at the Cosmos SDK level, e.g., one module for bitcoin, another for EVM chains, another for IBC transfers, etc.

There is also minimal reliance on smart contracts and are only used for key-rotations and mint/burn operations.

Costs:

Users only pay the fee in the asset they transfer on the source chain, and all other fees (relaying / intermediary / destination chains) are covered by services around the network.

App: https://satellite.axelar.network/

Celer Network

Summary:

Celer Network is a multi-chain operating system built with Cosmos SDK. Celer released the world’s first EVM-supported inter-chain message SDK. and has a track record of no hacks for any product since Celer originated in 2018.

Users have single-click UX with fast confirmation of the bridge between source and destination chains and with 2,500-3,500 daily active users comfortable with using Celer Network.

Security:

Costs: cBridge charges the gas fee on the destination chain to cover the operation costs of validators. The fee is not related to the size of the transaction, unlike many other bridging solutions.

There are also no platform fees charged by the Celer Network for token asset bridging.

App:

https://cbridge.celer.network/#/transfer

Gravity Bridge

Summary:

The Gravity Bridge blockchain is the trustless, neutral bridge between Ethereum and the Cosmos ecosystem. The principal focus of the Gravity community is on providing the most effective and secure bridge possible, instead of a DeFi application on the local chain.

The Gravity Bridge protocol provides strong censorship resistance guarantees and the bridge will remain live as long as the chain remains live, i.e. with more than 2/3 of validators active.

Currently supports Ethereum and has plans to support additional EVM chains before the end of 2022.

Security:

The Gravity Bridge, like all bridges, has two components. A Solidity contract on Ethereum and a CosmosSDK module on the Gravity Bridge blockchain.

While there’s nothing set for an insurance fund at the moment, the $GRAV token holders can choose to use some funds from the large community pool to help recuperate stolen funds.

Costs:

Robust relaying reward systems combined with reduced gas costs creates extremely fast & reliable transfers with a decentralized, self-funded, relayer community

App:

https://spacestation.zone/

Nomad

Summary:

Nomad is an interoperability protocol for generalized message passing. At its core, it uses an optimistic mechanism with fraud proofs to reduce the overhead of sending messages between chains and requiring 100+ validators. Nomad is designed to be non-custodial, trust-minimized and will be permissionless and decentralized in the future. With their unique watcher system, they forgo the need of 100+ validators and need only 1 good faith watcher to prevent fraudulent transactions.

Unlike validator-based bridges, Nomad channels only require one “validator” and as a result, are easy to scale and deploy rapidly. It optimizes for high security, low cost, with the trade-off of mild latency to ensure fraud proofs can be confirmed on-chain.

Nomad is already running in production on Ethereum and Moonbeam, and has $70M+ TVL in contracts, and over$ 300M in total volume bridged. By the middle of Q2, Nomad will also support Arbitrum, Optimism, Polygon, BSC, Avalanche, and Fantom.

Evmos will officially use Nomad and Connext as its bridging solutions, with Nomad as the settlement layer and Connext as the liquidity layer. Connext and Nomad have a deep partnership that enables the modular interoperability stack.

Security:

Nomad is composed of multiple agents — an updater, relayers, processors, and watchers. The most important is the updater, whose job it is to “validate” or update new roots.

The updater is the sole “validator” in the system, and Nomad prevents liveness issues by downtime slashing + rotation, similar to that of a validator system. Relayers are like IBC relayers, and processors simply prove inclusion of messages on replicas — these are both permissionless roles. The watchers observe updaters to ensure that they are behaving correctly, and flag fraud if observed. Watchers allow Nomad to have a trust model where there only needs to be one honest actor, rather than a majority.

Currently, the bridge is running in a permissioned environment with the core team running all updaters and watchers, but agents will become decentralized as Nomad continues to expand into the market.

Nomad has partnered closely with Connext to provide a “fast liquidity” option. Connext uses liquidity pools to allow instant bridging whenever enough liquidity is available. If Connext’s liquidity network can’t offer sufficient liquidity, Nomad will default to its locking-and-minting layer, with the 30 minute dispute window.

Nomad has been audited back in October 2021 and is currently in the middle of a Quantstamp audit — with no flagged issues thus far.

Costs:

Nomad doesn’t have to pay 100+ validators to maintain the bridge and therefore requires minimal fees. Users will only need to incur network fees of the underlying chains.

The off-chain agents are currently run by the Nomad core team who are subsidizing fees. As Nomad progressively decentralizes agent operations, new actors will probably begin to charge fees in exchange for relaying transactions across chains.

The only cost Nomad charges is the underlying gas fees to submit transactions on various networks plus any premium agents add on top. For fast chains with lower gas fees, the cost should be in the order of cents to a couple dollars.

There are no platform “toll” fees on Nomad, and they’re making an effort to reduce gas and network fees by implementing optimizations around enqueuing messages and processing.

Users can use Metamask to send their transaction, and after a pre-defined latency for the chain pair (~30 mins for Ethereum), will receive their tokens on the destination chain. Nomad subsidizes the relaying and processing costs, and only requires users to pay network fees with no additional cost.

App:

https://app.nomad.xyz/

Osmosis

Sun, 22 Aug 2021 23:46:37 GMT

Osmosis

Proof of stake with decentralized exchange app to provide liquidity, swap, and stake the tokens of blockchains from across the cosmos ecosystem
Automated market maker connects to other tendermint based blockchains
OSMO holders can delegate their assets to coinbase cloud public validator to earn a share of rewards (self staking and earn fees through token delegated to their validators)
Innovating in liquidity pools by using osmosis’ governance processes and interoperable builder toolkit. OSMO native token in osmosis used for governance and paying transaction fees.

Superfluid staking

If you deposit two supported tokens into pool can stake liquidity share to validators on the token’s chains
Deposited asset will not only earn a share of fees from LP swap transactions but also earn rewards for helping to secure the chain
Can provide liquidity to the osmosis AMM and participate in the native ecosystem of the tokens they hold by staking
- Example: OSMO<> AKT pool’s LP tokens can secure both Osmosis and Akash networks

Participating in Osmosis

Validator nodes participate in osmosis consensus, propose blocks, and verify blocks proposed by other validators.
Over time validators will be able to incur more responsibility in the ecosystem, such as acting as price oracles or bridges
- Full Nodes: Interact with the osmosis blockchain
- Archival Nodes: Store history state of the chain
- Seed Nodes: Used to help nodes find and connect to peers
- Sentry Nodes: Used to stand between validators and rest of the network, protect validators from DDos
Validator’s chance of being selected to perform work and earn rewards for doing so is determined by weight of OSMO staked to the validator
OSMO holders may delegate their tokens to an active validator to help secure the network and earn a portion of the rewards
Validators set a commission fee between 5-100% of total rewards earned by the delegators 14 day unbonding period for delegation, unbonded OSMO do not earn rewards but are eligible for losses due to slashing. Delegators also have full voting rights within Osmosis’ governance

Rewards and economics on Osmosis

Governance token to vote, pay transaction fees, and to provide rewards to participants for providing security
Annual reward rate is 160% per annum for validators
25% of released tokens will be issued to validators and their delegators as a reward for their work daily
760,000 -> 500,000 -> 340,000 -> 250,000 per day

Three types of fees paid out to users

Transaction Fees: Any user who transact on the chain, distributed to OSMO stakers and validators
Swap Fees: Any trader swapping assets on Osmosis
- Determined by LP parameters and size of trade
- Distributed to pool’s liquidity providers
Exit Fees
- Percentage of LP shares and charged to liquidity providers who pull their liquidity out of a pool. Shares are burned and distribute their value to the other liquidity providers in the pool.
45% of tokens released per epoch are designated as rewards for liquidity providers in pools
Any designated rewards that are reduced from mining reward by governance will be directed to the Osmosis community pool instead.
Inflation on Osmosis is structured around a thirdening model with token issuance cut by ⅓ each year
100m initial supply 300 in first year 200 in second until a maximum supply of 1bn OSMO is issued

Risk of participation

Slashing: Validators committing a slashable offense will result in loss of funds and reward for validators and any delegators who staked their OSMO to that validator

Governance on Osmosis

Will add new features that rely on active participation of the community
Designed such that most efficient solution is reachable through the process of experimentation 5% of token released daily on osmosis are directed to the community pool.

Lecture 1

Mon, 18 Jan 2021 23:46:37 GMT

Lecture 1

Class Content

Will learn

Relationship between computing and gender, work, expertise, intelligence
Ability to tell stories about computing

Won’t learn

Are computers good or bad?
What is the right policy for information distribution?
What the future will look like?

Main Questions

How did programming transform from a feminine to masculine occupation?
How did computers go from a symbol of repression to a symbol of liberation?
Why do so many Silicon Valley programmers go to burning man?

Lecture 2

Plan of the course

Week 2-4 Computing before digital computers
Week 5-10 How the cold war made computers the machines they are today
Week 11-15 Contemporary internet platforms and algorithms

Questions

Lecture 1

Invalid Date

TODO

Put readings on todo list can be found on class webpage Put homeworks on todo list Midterm 1 Monday march 16 in class Discussion sections Monday-Wednesday

Core material

Find patterns in data and use them to make predictions
Models + stats help us understand patterns
Use optimization algorithms to learn the patterns

Classification

Simplest Case: you have two choices, given data, make a prediction
Knn best when few outliers

Classifying Numbers

Turn points into grid of 0s and 1s based on the color of the grid
Turn grid into a vector by flattening the vector
Create hyperplane in the n-dimensional space to group things

Testing and Validation

Train a classifier - it learns to distinguish 7 from not 7
Test the classifier on NEW images
There are two types of error
- Training set error: Fraction of training images not classified correctly
- Test set error: Fraction of misclassified NEW images, not seen during training
Outliers: Points whose labels are atypical (e.g solvent borrowers who defaulted anyway)
Overfitting: When the test error deteriorates because the classifier becomes too sensitive to outliers
Hyperparameters: Most ML algorithms have a few hyperparameters that control over/underfitting. eg k in k-nearest neighbors

Select classifiers by validation

Validation Set: Hold back a subset of the labeled data
Train the classifier multiple times with different hyperparameter settings
Choose setting(hyperparameter + learning algorithm) that works best on validation set

Now, we have 3 sets:

Training set: Used to learn model weights
Validation set: Used to tune hyperparameters, choose among different models
Test set: Used as FINAL evaluation. Test set kept in vault, ran once, at the very end

Kaggle.com

Runs ML competitions, including our HWs
We use 2 data sets:
- public set labels available during the competition
- private set labels known only to Kaggles

Albert Su

Showerthoughts

Useful stuff

Not Useful stuff

Longer stuff

Personal Project Ideas

Completion Time: 10 years

Completion Time: 1 year

Completion Time: 1 month

Completion Time: 1 week

Reading List

To Read

Read

Kinesis Labs

Things to do when...

Ideas on Tech

Short form

University Related

DeFi

Internships

Learning Tech Stuff

Misc

Long form

Cool projects from eth hackathons

Dynamic Fees Postmortem

Kinesis Dynamic Fees Postmortem

Key Points

The Issue

Triaging the Issue

Next Steps

Redeploying Contracts

Sending Rewards

Public Announcement to Migrate Funds

Set Rewards Per Second to Zero

TODOs on Kinesis’ End

Artificial Growth - Giving out 100k in 60 days to grow faster

Appendix

Crypto

Everything I’m still bullish on

Bullish on Team and Idea

Projects to be built on top of Eigenlayer

Bullish on Idea

Bullish on Team

Other

Everything I’m bearish on

Bearish on Team and Idea

Bearish on Team

Bearish on Idea

Other

Things you might not have know

TVL is fake

Random list of protocols that probably have an exploit or two in them

Tiers of DeFi Teams

List of Novel or Interesting Defi Protocols you probably haven’t heard of

Lessons from a UC Berkeley 4.0

Midterm 1

One Day Before the Exam, you haven’t started studying. What do you do?

Why do you succeed when other berkeley students fail?

Do you forget stuff after classes are over?

Translating skills from class to work?

Homeworks

Piazza

Have you ever gotten worse than an A in a class?

Probability in Electrical Engineering and Computer Science

Probability: Math + a way of thinking about uncertainty

All of the rules of probability you are likely familiar with are consequences of the three axioms.

Lecture 2: Conditional Probability, Independence, Random Variables

Bayes Rule

Conditioning also allows us to usefully decompose intersections of events.

Independence

Conditional Independence

Random Variables

Probability Distributions

Joint Distributions

Independent Random Variables

Expectation - Used for discrete random variables

Linearity of Expectation

Sum of Independent Binomials, Variance, Covariance, Correlation Coefficient, Entropy

General Strategy for Expectation

Tail sum formula: Another way to compute expectation without using linearity of expectation.