Welcome (back) to IST 380 !
Today: putting the programming in
Data Science Programming…
Functions!
I have a sinking
feeling about all
of this fun?
Congratulations, Ravens!
Assignments…
Homework #1 is due tomorrow evening (2/5)
Getting started with R (tutorial + "quiz" + text)
There will be time this evening, if you'd like to use it…
started? finished? thoughts so far?
Homework #2 is due next Tuesday (2/12)
Pr #1: working through the text examples
Pr #2: writing some additional functions + a chance
to consider probability problems
Predictive?
Pr #3: writing a predictive model… It was 101
years ago!
Following up…
Following up…
http://www.propublica.org/article/everything-we-know-so-far-about-obamas-big-data-operation
Following up…
http://www.propublica.org/article/everything-we-know-so-far-about-obamas-big-data-operation
Palm trees!
R in the
NYT
http://www.nytimes.com/2009/01/07/
technology/business-computing/07program.html
Our path
g 2
… R's toolsets, in
m
m
now building gra i l l s
r o Sk
larger pieces… P
descriptive
statistics
Subject
Central Limit Expertise
Theorem
Functions
predictions I predict we'll get here…
R Reference Card
A security blanket for some of us…
RStudio An IDE that wraps R IDE like to know
what this means!
You might want to start with Chapter 9's Integrated Development Environment…
RStudio An IDE that wraps R IDE like to know
what this means!
Editable
files/scripts Live data
Plots and help
Console
interactions
You might want to start with Chapter 9's Integrated Development Environment…
summary
Descriptive statistics hist
quantile
sd, mean
state populations
Chapter 6 reviews statistical descriptions using these data
rgeom
Generative statistics runif
rnorm …
sample
replicate
distribution of samples of state populations
Chapter 7 reviews repeated sampling and the resulting distribution of means
Try it!
(1) Load in the state's populations
Is the mean or median greater? Why?
(2) Which state is closest to the median?
Extra: Which is closest to the 42% quantile?
Create a sample of 16 states from the list.
Create a distribution of 100 such samples – histogram it!
(3) Increase the number of samples until you obtain
sample-distribution mean within ~1% of the real mean.
Find 5% and 95% quantiles of that distribution.
(p<.05)
Big Ideas:
Central Limit Theorem
Law of Large Numbers
Monte Carlo Methods
Central Limit Theorem
the mean of a large number of independent random
variables, each with finite mean and variance, will be
approximately normally distributed.
state populations means of 4000 samples
(size 16) from the states
Central Limit Theorem
Take N samples from this population and find the
mean of each one. For large N, those sample means
will form a bell curve around the true mean
state populations means of 4000 samples
(size 16) from the states
Central Limit Theorem
Take N samples from this population and find the
mean of each one. For large N, those sample means
will form a bell curve around the true mean
o n e
s i s of …
h i
T l ca s e
c i a
spe
state populations means of 4000 samples
(size 16) from the states
Law of Large Numbers
in the limit, the average of the results obtained from a large
number of random trials of a process will converge to the
expected value for that process
chances of rolling doubles?
Law of Large Numbers
whose take-home message is: Try it lots of times and
just see what happens!
Monte Carlo Methods
The two Monte Carlos
Making random
numbers work for you!
Monte Carlo casino, Monaco
Monte Carlo
methods, Math/CS
The two Monte Carlos
Stanislaw Ulam
(Los Alamos badge)
Bond. (James Bond)
Making random
numbers work for you!
Monte Carlo casino, Monaco
Monte Carlo
methods, Math/CS
Monty Hall
Let's make a deal '63-'86
Sept. 1990
inspiring the “ Monty Hall paradox”
0 1 42 3 4 5 6 7 8 9
A
H
0 1 42 3 4 5 6 7 8 9
A
H
Hw #2: Monte Carlo Monty Hall
… and a second example:
Both envelopes hold some positive amount of money (in a check or IOU),
but one of these two envelopes holds twice as much money as the other.
Should you switch or stay?
Functions in R (Chapter 9)
Functions in R
A function to add two inputs…
thoughts? oddities? niceties?
Functions in R
A "guessing-game" function… Let's fix it!
42 in 72-point font!
Slide credits: thanks to JHU's R. D. Peng
nested conditionals…
What's going
on here?
How are these two
conditionals different?
cat is better
than print
seq_along creates
a list of indices
Thoughts?
Could we write a Monty Hall function?
MH <- function()
… that runs one three-curtain trial?
MHall(chosen_curtain=1, sors="switch", verbose=TRUE)
… another to allow us to turn off printing?
Mhall_N(chosen_curtain=1, sors="switch", N=300)
… another to run it N times?
MystEnv_N(first_env=10, sors="switch", N=10)
… and another to try the envelope-switching?
So, what is Machine Learning?
The goal of machine learning … or
predictive statistics/analytics,
is to find a function
that yields an output from a previously-unseen input
- based on the data available about the process in question.
This week's final problem asks you to write such a
function – for the Titanic survivor dataset.
The Titanic
April 15, 1912
1502 out of the
2224 passengers
died in the sinking
What characteristics did
the survivors share?
The Data
here are the
11 columns
There are 742 rows and 11 columns in the training data.
Our goal
… is to write a function that takes in a row of new data and
outputs whether that passenger would survive (1) or not (0).
A first predictor
A second predictor
Does the data match the
famous emergency cry?
Testing our functions…
Try it!
Help is available either with hw#1
(getting started with R)
or hw#2 (writing functions/Titanic)
this evening during lab time…
Good luck with everything this week!
Lab !
CS vs. IS and IT ?
greater integration
system-wide issues
smaller details
machine specifics
www.acm.org/education/curric_vols/CC2005_Final_Report2.pdf
CS vs. IS and IT ?
Where will IS go?
CS vs. IS and IT ?
IT ?
Where will IT go?
IT ?
The bigger picture
Weeks 10-12 Weeks 13-15
Objects Final Projects
Week 10 Week 13
classes vs. objects final projects
Week 11 Week 14
methods and data final projects
Week 12 Week 15
inheritance final exam
• Neighbor's name
Data?!
• A place they consider home
• Are they working at a company now? Where?
• How many U.S. states have they visited?
• Their favorite unhealthy food… ?
• Do they have any "Data Science"
(statistics, machine learning, CS)
background?
state reminders…
• Neighbor's name Zachary Dodds
Data!
• A place they consider home Pittsburgh, PA
• Are they working at a company now? Where? Harvey Mudd
• How many U.S. states have they visited? 44
• Their favorite unhealthy food… ? M&Ms
• Do they have any "Data Science"
(statistics, machine learning, CS)
background? mostly CS for me…
• Neighbor's name Zachary Dodds
Data!
• A place they consider home Pittsburgh, PA
• Are they working at a company now? Where? Harvey Mudd
• How many U.S. states have they visited? 44
• Their favorite unhealthy food… ? M&Ms
• Do they have any "Data Science" This class is truly
seminar-style:
(statistics, machine learning, CS) we're devloping
expertise in this
background? mostly CS for me… field together.
be sure to set up your login + profile for the submission site…
