Programming in R

• R is a dialect of the S language
  • S was initiated at the Bell Labs as an internal statistical analysis environment.
  • Most well known implementation is S‐plus (most recent stable release was in 2010)
• R was first announced in 1993.
• The R core group was formed in 1997, who controls the source code of R (written in C)
• R 1.0.0 was released in 2000
• The current version is 3.5.0 (released on April 23, 2018)
  • Previous version 3.4.4 was released on March 15, 2018

• R is a dialect of the S language
  • Language designed for statistical analysis
  • Similar syntax
• Available on most platform/OS
• Rich data analysis functionalities and sophisticated graphical capabilities
• Active development and very active community
  • CRAN: The Comprehensive R Archive Network
• Source code and binaries, user contributed packages and documentation
  • More than 10,000 packages available on CRAN
• Free to use

• S-PLUS: commercial verison of S
• Gretl: open-source statistical package, mainly for econometrics
• SPSS: widely used program for statistical analysis in social science
• PSPP: free alternative to SPSS
• SAS: proprietary software that can be used with very large datasets such as census data
• STATA: proprietary software that is often used in economics and epidemiology
• MATLAB: proprietary software used widely in the mathematical sciences and engineering
• GNU Octave: free alternative to MATLAB
• Python: general programming language

• R can be installed on Windows, Mac OSX and Linux from CRAN.
• R version 3.4.3 available on LUApps at https://luapps.lehigh.edu

• From Command Line on *NIX
  • Enter R on the command line (if you have modified your PATH correctly)
• Batch Mode on *NIX
  • Use the Rscript filename.R command to execute commands from a file, filename.R

cat hello.R

## print("Hello World!")

Rscript hello.R

## [1] "Hello World!"

• RStudio is the most popular (de facto) environment for running R on all platforms.
• free and open source IDE for R. Can be installed on Windows, Mac OSX and Linux.
• user interface comparable to other IDEs or software such as MATLAB.
• more suited for development
• Version 1.1.383 available on LUApps at https://luapps.lehigh.edu

• Anaconda Python distribution is the most popular platform for Python
• It provides
  • a convenient install procedure for over 1400 Data Science libraries for Python and R
  • conda to manage your packages, dependencies, and environments
  • anaconda navigator: a desktop portal to install and launch applications and editors including Jupyter, RStudio, Visual Studio Code, and Spyder
    • install r-essentials from anaconda navigator or conda install r-essentials from the command line
• Jupyter Notebooks is an alternative to RStudio for writing scripts and workflows that you can share with others.
  • It is ideal for reproducible research or data reporting
• Visit https://go.lehigh.edu/linux to use R, Anaconda and other Linux software installed and maintained by the Research Computing group on your local Linux laptop or workstation
• Click here for instructions to run Jupyter Notebooks on Sol

• Use the console to use R as a simple calculator

1 + 2

## [1] 3

• The assignment symbol is "<-". The classical "=" symbol can also be used

a=2+3
b<-10/a
a

## [1] 5

b

## [1] 2

• install packages from CRAN, for e.g. knitr

install.packages('knitr')

• load a library, for e.g. knitr

library(knitr)

• Help from command line

?<command name>
??<part of command name/topic>

• or search in the help page in RStudio

• getwd(): display current working directory

• setwd('dir'): change current working director to dir

• R has five atomic classes
• Numeric
  • Double is equivalent to numeric.
  • Numbers in R are treated as numeric unless specified otherwise.
• Integer
• Complex
• Character
• Logical
  • TRUE or FALSE
• You can convert data from one type to the other using the as.<Type> functions
• To check the class of an object, use the is.<Type> function.

a <- 3
b <- sqrt(a)
b

## [1] 1.732051

c <- 2i
d <- TRUE
d

## [1] TRUE

as.numeric(d); as.character(b); is.complex(c)

## [1] 1

## [1] "1.73205080756888"

## [1] TRUE

• Vectors can only contain elements of the same class
• Vectors can be constructed by
  • Using the c() function (concatenate)
• Coercion will occur when mixed objects are passed to the c() function, as if the as.<Type>() function is explicitly called
  • Using the vector() function
• One can use [index] to access individual element
  • Indices start from 1

# "#" indicates comment
# "<-" performs assignment operation (you can use "=" as well, but "<-" is preferred)
# numeric (double is the same as numeric)
d <- c(1,2,3)
# character
d <- c("1","2","3")
# you can covert at object with as.TYPE
# as. numeric changes the character vector created above to numeric
as.numeric(d)

## [1] 1 2 3

# The conversion doesn't always work though
as.numeric("a")

## Warning: NAs introduced by coercion

## [1] NA

x <- c(0.5, 0.6) ## numeric
x <- c(TRUE, FALSE) ## logical
x <- c(T, F) ## logical
x <- c("a", "b", "c") ## character
# The ":" operator can be used to generate integer sequences
x <- 9:29 ## integer
x <- c(1+0i, 2+4i) ## complex
x <- vector("numeric", length = 10)
# Coercion will occur when objects of different classes are mixed
y <- c(1.7, "a") ## character
y <- c(TRUE, 2) ## numeric
y <- c("a", TRUE) ## character
# Can also coerce explicitly
x <- 0:6
class(x)

## [1] "integer"

as.logical(x)

## [1] FALSE  TRUE  TRUE  TRUE  TRUE  TRUE  TRUE

• Lots of R operations process objects in a vectorized way
  • more efficient, concise, and easier to read.

x <- 1:4; y <- 6:9
x + y

## [1]  7  9 11 13

x > 2

## [1] FALSE FALSE  TRUE  TRUE

x * y

## [1]  6 14 24 36

print( x[x >= 3] )

## [1] 3 4

• Matrices are vectors with a dimension attribute
• R matrices can be constructed
  • Using the matrix() function
    • Passing an dim attribute to a vector
    • Using the cbind() or rbind() functions
• R matrices are constructed column‐wise
• One can use [<index>,<index>] to access individual element

# Create a matrix using the matrix() function
m <- matrix(1:6, nrow = 2, ncol = 3)
m

##      [,1] [,2] [,3]
## [1,]    1    3    5
## [2,]    2    4    6

dim(m)

## [1] 2 3

attributes(m)

## $dim
## [1] 2 3

# Pass a dim attribute to a vector
m <- 1:10
m

##  [1]  1  2  3  4  5  6  7  8  9 10

dim(m) <- c(2, 5)
m

##      [,1] [,2] [,3] [,4] [,5]
## [1,]    1    3    5    7    9
## [2,]    2    4    6    8   10

# Row binding and column binding
x <- 1:3
y <- 10:12
cbind(x, y)

##      x  y
## [1,] 1 10
## [2,] 2 11
## [3,] 3 12

rbind(x, y)

##   [,1] [,2] [,3]
## x    1    2    3
## y   10   11   12

# Slicing
m

##      [,1] [,2] [,3] [,4] [,5]
## [1,]    1    3    5    7    9
## [2,]    2    4    6    8   10

# element at 2nd row, 3rd column
m[2,3]

## [1] 6

# entire i<sup>th</sup> row of m
m[2,]

## [1]  2  4  6  8 10

# entire j<sup>th</sup> column of m
m[,3]

## [1] 5 6

• Lists are a special kind of vector that contains objects of different classes
• Lists can be constructed by using the list() function
• Lists can be indexed using [[ ]]

# Use the list() function to construct a list
x <- list(1, "a", TRUE, 1 + 4i)
x

## [[1]]
## [1] 1
## 
## [[2]]
## [1] "a"
## 
## [[3]]
## [1] TRUE
## 
## [[4]]
## [1] 1+4i

• Data frames are used to store tabular data
  • They are a special type of list where every element of the list has to have the same length
  • Each element of the list can be thought of as a column
  • Data frames can store different classes of objects in each column
  • Data frames also have a special attribute called row.names
  • Data frames are usually created by calling read.table() or read.csv()
    • More on this later
  • Can be converted to a matrix by calling data.matrix()

• R objects can have names

# Each element in a vector can have a name
x <- 1:3
names(x)

## NULL

names(x) <- c("a","b","c")
names(x)

## [1] "a" "b" "c"

x

## a b c 
## 1 2 3

# Lists
x <- list(a = 1, b = 2, c = 3)
x

## $a
## [1] 1
## 
## $b
## [1] 2
## 
## $c
## [1] 3

# Names can be used to refer to individual element
x$a

## [1] 1

# Columns and rows of matrices
m <- matrix(1:4, nrow = 2, ncol = 2)
dimnames(m) <- list(c("a", "b"), c("c", "d"))
m

##   c d
## a 1 3
## b 2 4

• The class() function
• The str() function
• The attributes() function reveals attributes of an object (does not work with vectors)
  • Class
  • Names
  • Dimensions
  • Length
  • User defined attributes
• They work on all objects (including functions)

m <- matrix(1:10, nrow = 2, ncol = 5)
str(matrix)

## function (data = NA, nrow = 1, ncol = 1, byrow = FALSE, dimnames = NULL)

str(m)

##  int [1:2, 1:5] 1 2 3 4 5 6 7 8 9 10

str(str)

## function (object, ...)

• Factors are used to represent categorical data.
• Factors can be unordered or ordered.
• Factors are treated specially by modelling functions like lm() and glm()

# Use the factor() function to construct a vector of factors
# The order of levels can be set by the levels keyword
x <- factor(c("yes", "yes", "no", "yes", "no"), levels = c("yes", "no"))
x

## [1] yes yes no  yes no 
## Levels: yes no

• R has a Date class for date data while times are represented by POSIX formats
• One can convert a text string to date using the as.Date() function
• The strptime() function can deal with dates and times in different formats.
• The package "lubridate" provides many additional and convenient features

# Dates are stored internally as the number of days since 1970-01-01
x <- as.Date("1970-01-01")
x

## [1] "1970-01-01"

as.numeric(x)

## [1] 0

x+1

## [1] "1970-01-02"

# Times are stored internally as the number of seconds since 1970-01-01
x <- Sys.time() ; x

## [1] "2019-06-28 11:51:39 EDT"

as.numeric(x)

## [1] 1561737099

p <- as.POSIXlt(x)
names(unclass(p))

##  [1] "sec"    "min"    "hour"   "mday"   "mon"    "year"   "wday"  
##  [8] "yday"   "isdst"  "zone"   "gmtoff"

p$sec

## [1] 39.241

• Missing values are denoted by NA or NaN for undefined mathematical operations.
  • is.na() is used to test objects if they are NA
  • is.nan() is used to test for NaN
  • NA values have a class also, so there are integer NA, character NA, etc.
  • A NaN value is also NA but the converse is not true

x <- c(1,2, NA, 10,3)
is.na(x)

## [1] FALSE FALSE  TRUE FALSE FALSE

is.nan(x)

## [1] FALSE FALSE FALSE FALSE FALSE

x <- c(1,2, NaN, NA,4)
is.na(x)

## [1] FALSE FALSE  TRUE  TRUE FALSE

is.nan(x)

## [1] FALSE FALSE  TRUE FALSE FALSE

• For each distribution R provides four functions: density (d), cumulative density (p), quantile (q), and random generation (r)
  • The function name is of the form [d|p|q|r]<name of distribution>
  • e.g. qbinom() gives the quantile of a binomial distribution

# Random generation from a uniform distribution.
runif(10, 2, 4)

##  [1] 3.916864 3.479611 3.478209 2.797747 3.262741 3.780256 2.140962
##  [8] 3.039850 3.525697 3.797439

# You can name the arguments in the function call.
runif(10, min = 2, max = 4)

##  [1] 2.843526 2.361906 2.322919 2.327150 2.545540 3.065703 2.637677
##  [8] 2.113549 3.257024 2.765928

# Given p value and degree of freedom, find the t-value.
qt(p=0.975, df = 8)

## [1] 2.306004

# The inverse of the above function call
pt(2.306, df = 8)

## [1] 0.9749998

• Similar to other languages, functions in R are defined by using the function() directives
• The return value is the last expression in the function body to be evaluated.
• Functions can be nested
• Functions are R objects
  • For example, they can be passed as an argument to other functions

newDef <- function(a,b)
 {
     x = runif(10,a,b)
     mean(x)
 }
newDef(-1,1)

## [1] 0.2491381

• Control structures allow one to control the flow of execution.

# Comparisons: <,<=,>,>=,==,!=
# Logical operations: !, &&, ||
if(x > 3 && x < 5) {
  print ("x is between 3 and 5")
} else if(x <= 3) {
  print ("x is less or equal to 3")
} else {
  print ("x is greater or equal to 5")
}

x <- c("a", "b", "c", "d")
# These loops have the same effect
# Loop through the indices
for(i in 1:4) {
  print(x[i])
}

## [1] "a"
## [1] "b"
## [1] "c"
## [1] "d"

# Loop using the seq_along() function
for(i in seq_along(x)) {
  print(x[i])
}

## [1] "a"
## [1] "b"
## [1] "c"
## [1] "d"

# Loop through the name
for(letter in x) {
  print(letter)
}

## [1] "a"
## [1] "b"
## [1] "c"
## [1] "d"

for(i in 1:4) print(x[i])

## [1] "a"
## [1] "b"
## [1] "c"
## [1] "d"

• The while loop can be used to repeat a set of instructions
• It is often used when you do not know in advance how often the instructions will be executed.
• The basic format for a while loop is while(cond) expr

sum <- 1
while ( sum < 11 )
{
  sum <- sum + 2;
  print(sum);
}

## [1] 3
## [1] 5
## [1] 7
## [1] 9
## [1] 11

sum <- 12
while (sum < 11 ) 
{
  sum <- sum + 2;
  print(sum);
}

• The repeat loop is similar to the while loop.
• The difference is that it will always begin the loop the first time. The while loop will only start the loop if the condition is true the first time it is evaluated.
• Another difference is that you have to explicitly specify when to stop the loop using the break command.

sum <- 1
repeat
{
  sum <- sum + 2;
  print(sum);
  if (sum > 11)
    break;
}

## [1] 3
## [1] 5
## [1] 7
## [1] 9
## [1] 11
## [1] 13

sum <- 12
repeat
{
  sum <- sum + 2;
  print(sum);
  if (sum > 11)
    break;
}

## [1] 14

• The break statement is used to stop the execution of the current loop.
  • It will break out of the current loop.
• The next statement is used to skip the statements that follow and restart the current loop.
  • If a for loop is used then the next statement will update the loop variable.

x <- rnorm(5)
for(lupe in x)
 {
     if (lupe > 2.0)
         next

     if( (lupe<0.6) && (lupe > 0.5))
        break

    cat("The value of lupe is ",lupe,"\n");
 }

## The value of lupe is  -1.613799 
## The value of lupe is  -0.245662 
## The value of lupe is  -0.756709 
## The value of lupe is  -0.3906685 
## The value of lupe is  0.304653

• The apply() function evaluate a function over the margins of an array
  • More concise than the for loops (not necessarily faster)

# X: array objects
# MARGIN: a vector giving the subscripts which the function will be applied over
# FUN: a function to be applied
str(apply)

## function (X, MARGIN, FUN, ...)

x <- matrix(rnorm(200), 20, 10)
# Row means
apply(x, 1, mean)

##  [1]  0.105284758 -0.547839904 -0.006042115  0.006990364 -0.052279347
##  [6]  0.762414286 -0.310173357  0.469239193 -0.157112108  0.623160416
## [11] -0.120917410 -0.008236342  0.301077236 -0.099175667  0.350147131
## [16] -0.196845153  0.476606109  0.244559332 -0.298884989  0.396900728

# Column sums
apply(x, 2, sum)

##  [1]  3.78591243 -1.31776440 -7.70247824  6.61153743 11.62866203
##  [6]  0.07050581 -3.62419206  6.11508356  5.07354178 -1.25207674

# 25th and 75th Quantiles for rows
apply(x, 1, quantile, probs = c(0.25, 0.75))

##           [,1]       [,2]       [,3]       [,4]       [,5]       [,6]
## 25% -0.6807435 -1.8368053 -0.4507926 -0.3938400 -0.7890905 -0.2451055
## 75%  1.2454631  0.9653446  0.7507690  0.4874695  0.9047425  1.3925322
##           [,7]       [,8]       [,9]      [,10]      [,11]      [,12]
## 25% -1.2497502 -0.3328754 -0.7400952 -0.4908325 -0.4404739 -0.7955561
## 75%  0.5810956  1.4608103  0.3797403  1.4744898  0.1085194  0.1486289
##          [,13]      [,14]      [,15]      [,16]      [,17]      [,18]
## 25% -0.4210659 -0.5224930 -0.4249143 -0.3661420 -0.4012379 -0.1791038
## 75%  0.8129981  0.8372162  0.9356285  0.4550468  1.4235447  0.5326436
##          [,19]      [,20]
## 25% -0.7053883 -0.2262357
## 75%  0.1295247  0.7913577

dim(x)

## [1] 20 10

# Change the dimensions of x
dim(x) <- c(2,2,50)
# Take average over the first two dimensions
apply(x, c(1, 2), mean)

##           [,1]        [,2]
## [1,] 0.1347153 -0.07717415
## [2,] 0.1966237  0.13360976

rowMeans(x, dims = 2)

##           [,1]        [,2]
## [1,] 0.1347153 -0.07717415
## [2,] 0.1966237  0.13360976

• lapply: Loop over a list and evaluate a function on each element
• sapply: Same as lapply but try to simplify the result
• tapply: Apply a function over subsets of a vector
• mapply: Multivariate version of lapply

• The tidyverse is a collection of R packages developed by RStudio’s chief scientist Hadley Wickham.
  • ggplot2 for data visualisation.
  • dplyr for data manipulation.
  • tidyr for data tidying.
  • readr for data import.
  • purrr for functional programming.
  • tibble for tibbles, a modern re-imagining of data frames.
• These packages work well together as part of larger data analysis pipeline.
• To learn more about these tools and how they work together, read R for Data Science.

• What is Tidy Data?
  • "Tidy data" is a term that describes a standardized approach to structuring datasets to make analyses and visualizations easier.
• The core tidy data principles
  • Variable make up the columns
  • Observations make up the rows
  • Values go into cells
• library(tidyverse) will load the core tidyverse packages:

library(tidyverse)

## ── Attaching packages ────────────────────────────────── tidyverse 1.2.1 ──

## ✔ ggplot2 3.1.0       ✔ purrr   0.3.0  
## ✔ tibble  2.0.1       ✔ dplyr   0.8.0.1
## ✔ tidyr   0.8.0       ✔ stringr 1.4.0  
## ✔ readr   1.1.1       ✔ forcats 0.4.0

## ── Conflicts ───────────────────────────────────── tidyverse_conflicts() ──
## ✖ dplyr::filter() masks stats::filter()
## ✖ dplyr::lag()    masks stats::lag()

library(lubridate)

## Loading required package: methods

## 
## Attaching package: 'lubridate'

## The following object is masked from 'package:base':
## 
##     date

• Packages that are part of tidyverse but not loaded automatically
  • lubridate for dates and date-times
  • magrittr provides the pipe, %>% used throughout the tidyverse.
  • readxl for .xls and .xlsx sheets.
  • haven for SPSS, Stata, and SAS data.
• packages that are not in the tidyverse, but are tidyverse-adjacent. They are very useful for importing data from other sources:
  • jsonlite for JSON.
  • xml2 for XML.
  • httr for web APIs.
  • rvest for web scraping.
  • DBI for relational databases

• readr is to provide a fast and friendly way to read rectangular data (like csv, tsv, and fwf).
• readr supports seven file formats with seven read_ functions:
  • read_csv(): comma separated (CSV) files
  • read_csv2(): semicolon separated file and "," for decimal point
  • read_tsv(): tab separated files
  • read_delim(): general delimited files
  • read_fwf(): fixed width files
  • read_table(): tabular files where colums are separated by white-space.
  • read_log(): web log files
• Usage
  • read_delim(file,delim)
    • file: path to a file, a connection, or literal data

# read daily usage report for Sol in AY 2016-17
# usage is reported in terms of SUs used and jobs submitted for
#  serial (1 cpu), single or smp ( > 1 cpu but max of 1 node) and 
#  parallel or multi node (> 1 node)  jobs 
#daily <- read_delim('http://webapps.lehigh.edu/hpc/training/soldaily1617-public.csv',delim=";", trim_ws = TRUE)
daily <- read_delim('http://webapps.lehigh.edu/hpc/training/soldaily1718-public.csv',delim=";", trim_ws = TRUE)

## Parsed with column specification:
## cols(
##   Type = col_character(),
##   Name = col_character(),
##   Department = col_character(),
##   PI = col_character(),
##   PIDept = col_character(),
##   Status = col_character(),
##   Day = col_date(format = ""),
##   SerialJ = col_integer(),
##   Serial = col_double(),
##   SingleJ = col_integer(),
##   Single = col_double(),
##   MultiJ = col_integer(),
##   Multi = col_double(),
##   TotalJ = col_integer(),
##   Total = col_double()
## )

• the readr functions will just work: you supply the path to a file and you get a tibble back

• A tibble, or tbl_df, is a modern reimagining of the data.frame,
  • keeping what time has proven to be effective, and
  • throwing out what is not.

• Tibbles are data.frames that are lazy and surly

• Create a tibble from an existing object with as_tibble()

• create a new tibble from column vectors with tibble()

tibble(x = 1:5, y = 1, z = x ^ 2 + y)

## # A tibble: 5 x 3
##       x     y     z
##   <int> <dbl> <dbl>
## 1     1     1     2
## 2     2     1     5
## 3     3     1    10
## 4     4     1    17
## 5     5     1    26

• define a tibble row-by-row with tribble():

tribble(
  ~x, ~y,  ~z,
  "a", 2,  3.6,
  "b", 1,  8.5
)

## # A tibble: 2 x 3
##   x         y     z
##   <chr> <dbl> <dbl>
## 1 a         2   3.6
## 2 b         1   8.5

• dplyr is a grammar of data manipulation, providing a consistent set of verbs to solve the most common data manipulation challenges:

• filter() picks cases based on their values.

  

• select() picks variables based on their names.

  

• mutate() adds new variables that are functions of existing variables

  

• summarise() reduces multiple values down to a single summary.

  

• arrange() changes the ordering of the rows.
• These all combine naturally with group_by() which allows you to perform any operation "by group"

daily %>% head

## # A tibble: 6 x 15
##   Type  Name  Department PI    PIDept Status Day        SerialJ Serial
##   <chr> <chr> <chr>      <chr> <chr>  <chr>  <date>       <int>  <dbl>
## 1 User  user… EN/Mechan… pi001 Mecha… Gradu… 2017-10-01       0      0
## 2 User  user… Mechanica… pi001 Mecha… Guest  2017-10-01       0      0
## 3 User  user… EN/Mechan… pi001 Mecha… Gradu… 2017-10-01       0      0
## 4 User  user… EN/Mechan… pi001 Mecha… Gradu… 2017-10-01       0      0
## 5 User  user… EN/Mechan… pi001 Mecha… Gradu… 2017-10-01       0      0
## 6 User  user… EN/Mechan… pi001 Mecha… Gradu… 2017-10-01       0      0
## # … with 6 more variables: SingleJ <int>, Single <dbl>, MultiJ <int>,
## #   Multi <dbl>, TotalJ <int>, Total <dbl>

# Number of core hours available per month for AY 2016-17
# Oct 1, 2016: Initial launch with 780 cpu
# Mar 15, 2017: Added 192 cpus
# May 1, 2017: Added 312 cpus
# Total Available at end of AY 2016-17: 1284 cpus
# Nov 15, 2017: Added 16 cpus (himem node)
# Apr 2, 2018: Added 252 cpus
# Aug 31, 2018: Added 72 cpus
# Total Available at end of AY 2017-18: 1624 cpus
# Nov 12, 2018: Added 72 cpus
# Dec 1, 2018: Added 24 cpus
# Jan 2, 2019: Added 432 cpus
# Total Available in AY 2018-19: 2152 cpus
ay1617su <- c(580320.00,561600.00,580320.00,580320.00,524160.00,580320.00,699840.00,955296.00,924480.00,955296.00,955296.00,924480.00)
ay1718su <- c(955296.00,924480.00,967200.00,967200.00,873600.00,967200.00,1117440.00,  1154688.00,1117440.00,1154688.00,1155480.00,1169280.00)

monthly <- daily %>% 
  group_by(Month=floor_date(as.Date(Day), "month"),Name,Department,PI,PIDept,Status) %>% 
  summarize(Serial=sum(as.double(Serial)),
    Single=sum(as.double(Single)),
    Multi=sum(as.double(Multi)),
    Total=sum(as.double(Total)),
    SerialJ=sum(as.double(SerialJ)),
    SingleJ=sum(as.double(SingleJ)),
    MultiJ=sum(as.double(MultiJ)),
    TotalJ=sum(as.double(TotalJ)))
monthly %>% head

## # A tibble: 6 x 14
## # Groups:   Month, Name, Department, PI, PIDept [6]
##   Month      Name  Department PI    PIDept Status Serial Single Multi Total
##   <date>     <chr> <chr>      <chr> <chr>  <chr>   <dbl>  <dbl> <dbl> <dbl>
## 1 2017-10-01 user… EN/Mechan… pi001 Mecha… Gradu…      0     0     0     0 
## 2 2017-10-01 user… Mechanica… pi001 Mecha… Guest       0     0     0     0 
## 3 2017-10-01 user… EN/Mechan… pi001 Mecha… Gradu…      0     0     0     0 
## 4 2017-10-01 user… EN/Mechan… pi001 Mecha… Gradu…      0     0     0     0 
## 5 2017-10-01 user… EN/Mechan… pi001 Mecha… Gradu…      0     0     0     0 
## 6 2017-10-01 user… EN/Mechan… pi001 Mecha… Gradu…      0   248. 5001. 5250.
## # … with 4 more variables: SerialJ <dbl>, SingleJ <dbl>, MultiJ <dbl>,
## #   TotalJ <dbl>

monthly %>% 
  group_by(Month) %>%   
  summarize(Total=round(sum(as.double(Total)),2),Jobs=round(sum(as.double(TotalJ)))) %>%
  mutate(Available=ay1718su,Unused=Available-Total,Percent=round(Total/Available*100,2)) -> monthlyusage
monthlyusage

## # A tibble: 12 x 6
##    Month        Total   Jobs Available  Unused Percent
##    <date>       <dbl>  <dbl>     <dbl>   <dbl>   <dbl>
##  1 2017-10-01 508739.  32817    955296 446557.    53.2
##  2 2017-11-01 494221.  50844    924480 430259.    53.5
##  3 2017-12-01 563782.  37092    967200 403418.    58.3
##  4 2018-01-01 495731. 103627    967200 471469.    51.2
##  5 2018-02-01 574644.  96780    873600 298956.    65.8
##  6 2018-03-01 762476.  17533    967200 204724.    78.8
##  7 2018-04-01 729374.  35536   1117440 388066.    65.3
##  8 2018-05-01 719943.  32791   1154688 434745.    62.4
##  9 2018-06-01 629904.   7991   1117440 487536.    56.4
## 10 2018-07-01 808487.  35872   1154688 346201.    70.0
## 11 2018-08-01 911909.  32635   1155480 243571.    78.9
## 12 2018-09-01 818882.  30309   1169280 350398.    70.0

library(knitr)
monthly %>%
  group_by(PIDept) %>%
  summarize(Total=round(sum(as.double(Total)),2),Jobs=round(sum(as.double(TotalJ)))) %>%
  kable

monthly %>%
  group_by(Department) %>%
  summarize(Serial=round(sum(as.double(Serial))),SMP=round(sum(as.double(Single))),DMP=round(sum(as.double(Multi))),Total=round(sum(as.double(Total)),2),
  Jobs=round(sum(as.double(TotalJ)))) %>% arrange(desc(Total)) %>% kable

library(xtable)
monthly %>%
  group_by(Department) %>%
  summarize(Serial=round(sum(as.double(Serial))),SMP=round(sum(as.double(Single))),DMP=round(sum(as.double(Multi))),Total=round(sum(as.double(Total)),2),
  Jobs=round(sum(as.double(TotalJ)))) %>% arrange(desc(Total)) %>% xtable

## % latex table generated in R 3.4.4 by xtable 1.8-2 package
## % Fri Jun 28 11:51:43 2019
## \begin{table}[ht]
## \centering
## \begin{tabular}{rlrrrrr}
##   \hline
##  & Department & Serial & SMP & DMP & Total & Jobs \\ 
##   \hline
## 1 & Biological Sciences & 641121.00 & 1661321.00 & 95819.00 & 2398260.36 & 402386.00 \\ 
##   2 & EN/Chemical Engineering & 25518.00 & 1170727.00 & 247976.00 & 1444221.76 & 17857.00 \\ 
##   3 & EN/Mechanical Engineering & 72018.00 & 246499.00 & 982193.00 & 1300709.46 & 21965.00 \\ 
##   4 & AS/Biochemistry (CAS) & 45254.00 & 707018.00 & 9258.00 & 761529.43 & 38840.00 \\ 
##   5 & Mechanical Engr and Mechanics & 559.00 & 512775.00 & 79357.00 & 592690.50 & 6889.00 \\ 
##   6 & Chemical and Biomolecular Engr & 2255.00 & 312510.00 & 142164.00 & 456929.47 & 2758.00 \\ 
##   7 & AS/Physics (AS) & 7416.00 & 192294.00 & 123810.00 & 323519.75 & 886.00 \\ 
##   8 & EN/Computer Science (EN) & 575.00 & 293024.00 & 498.00 & 294097.32 & 806.00 \\ 
##   9 & IC/Computer Science \& Business & 0.00 & 1107.00 & 112508.00 & 113614.48 & 212.00 \\ 
##   10 & EN/ & 63218.00 & 14.00 & 0.00 & 63231.84 & 2905.00 \\ 
##   11 & Chemistry & 0.00 & 55719.00 & 0.00 & 55719.32 & 1210.00 \\ 
##   12 & AS/Molecular Biology & 42368.00 & 9833.00 & 22.00 & 52223.76 & 7397.00 \\ 
##   13 & AS/Applied Mathematics & 48764.00 & 1070.00 & 0.00 & 49834.27 & 2781.00 \\ 
##   14 & AS/Biology & 0.00 & 26231.00 & 0.00 & 26231.15 & 245.00 \\ 
##   15 & Economics & 0.00 & 24409.00 & 0.00 & 24409.44 & 165.00 \\ 
##   16 & Computer Sci and Engineering & 1733.00 & 3870.00 & 9751.00 & 15354.66 & 2329.00 \\ 
##   17 & EN/Bioengineering & 0.00 & 11213.00 & 0.00 & 11213.45 & 81.00 \\ 
##   18 & AS/Integrative Biology & 44.00 & 9293.00 & 4.00 & 9341.34 & 219.00 \\ 
##   19 & AS/Chemistry (AS) & 527.00 & 7661.00 & 0.00 & 8187.47 & 1064.00 \\ 
##   20 & Civil and Environmental Engr & 2070.00 & 5492.00 & 0.00 & 7561.14 & 1099.00 \\ 
##   21 & Physics & 5660.00 & 1738.00 & 0.00 & 7398.17 & 1099.00 \\ 
##   22 & Materials Science and Engr & 200.00 & 14.00 & 978.00 & 1192.95 & 447.00 \\ 
##   23 & Accounting & 23.00 & 305.00 & 0.00 & 328.42 & 38.00 \\ 
##   24 & EN/Electrical Engineering & 273.00 & 0.00 & 0.00 & 273.37 & 43.00 \\ 
##   25 & EN/Computer Engineering & 6.00 & 0.00 & 0.00 & 6.10 & 1.00 \\ 
##   26 & EN/Industrial and Systems Engr & 6.00 & 0.00 & 0.00 & 5.56 & 103.00 \\ 
##   27 & LTS & 5.00 & 0.00 & 0.00 & 4.53 & 1.00 \\ 
##   28 & EN/Structural Engineering & 1.00 & 0.00 & 0.00 & 0.69 & 1.00 \\ 
##    \hline
## \end{tabular}
## \end{table}

monthly %>%
  group_by(Status) %>%
  summarize(Total=round(sum(as.double(Total)),2)) -> monthlystatus
monthlystatus

## # A tibble: 5 x 2
##   Status                   Total
##   <chr>                    <dbl>
## 1 Faculty                442642.
## 2 Faculty/Staff         2463835.
## 3 Graduate Student      3952679.
## 4 Guest                  653371.
## 5 Undergraduate Student  505562.

• The goal of tidyr is to help you create tidy data.
• Tidy data is data where:
  • Each variable is in a column.
  • Each observation is a row.
  • Each value is a cell.

• Tidy data describes a standard way of storing data that is used wherever possible throughout the tidyverse.

• two fundamental verbs of data tidying:

  • gather() takes multiple columns, and gathers them into key-value pairs
  • spread(). takes two columns (key & value) and spreads in to multiple columns
  • separate(): Splitting a single variable into two
  • unite(): Merging two variables into one

daily %>% 
  filter(as.Date(Day) >= "2018-02-01" & as.Date(Day) <= "2018-03-01") %>% 
  select(Day,Name,Department,PI,PIDept,Serial,Single,Multi) %>% 
  gather(JobType,Usage,Serial:Multi) %>% 
  filter(as.double(Usage) > 100 ) -> tmp
tmp %>% arrange(Usage) %>% 
  kable

tmp %>% arrange(Usage) %>% 
  spread(JobType,Usage,fill = 0.0) %>% 
  kable

• Data Visualization
• Data cleaning/preprocessing
• Profiling and debugging
• Regression Models
• Machine learning/Data Mining
• ···

• Data visualization or data visualisation is viewed by many disciplines as a modern equivalent of visual communication.
• It involves the creation and study of the visual representation of data.
• A primary goal of data visualization is to communicate information clearly and efficiently via statistical graphics, plots and information graphics.
• Data visualization is both an art and a science.
• More details in next weeks seminar

p <- daily %>%
  group_by(Day, Status=trimws(Status)) %>%
  summarize(Total=round(sum(as.double(Total)),2),Jobs=round(sum(as.double(TotalJ)))) %>%
  ggplot(aes(Day,Total)) + geom_line(aes(col = Status))
p

Plot is very busy. There are several options to clean this up.

• summarize by week or month
• take a running average or add a smoothing function
• create separate plots for each Department using facet_wrap

p + facet_wrap( ~Status, scales = "free", ncol = 2) + theme(legend.position='none')

If a Picture is Worth a Thousand Words, Then Is a Video Worth a Million?

• What is an animation really?
  • Just a collection of images that appear at a high frequency or frame rate.
• If you have a collection of pictures, you can convert them to gif, mpeg, or any other video format using tools like ImageMagick or ffmpeg.
• R provides tools that will convert a collection of images from plots to video provided you have one of these conversion tools.

if(!require('animation')){
    install.packages('animation')
}

## Loading required package: animation

if(!require('gganimate')){
    install.packages('gganimate')
    install.packages('gifski')
    install.packages('png')
}

## Loading required package: gganimate

dailyusage_status <- daily %>%
  group_by(Day=floor_date(as.Date(Day), "week"), Status) %>%
  summarize(Total=round(sum(as.double(Total)),2),
            Jobs=round(sum(as.double(TotalJ)))) %>%
  ggplot(aes(Day,Total,group=Status)) + 
  geom_line(aes(col = Status)) +
  labs(title = 'Week: {frame_along}', x = 'Month', y = 'Usage', color = 'Status') +
  transition_reveal(along = Day)
anim <- animate(dailyusage_status, height = 300, width = 640)
anim_save("weeklystatus.gif", anim)

dailyusage_pidept <- daily %>%
  group_by(Day=floor_date(as.Date(Day), "week"), PIDept) %>%
  summarize(Total=round(sum(as.double(Total)),2),
            Jobs=round(sum(as.double(TotalJ)))) %>%
  ggplot(aes(Day,Total,group=PIDept)) + 
  geom_line(aes(col = PIDept)) +
  facet_wrap( ~PIDept, scales = "free", ncol = 3) + 
  theme(legend.position='none') + 
  labs(title = 'Week: {frame_along}', x = 'Month', y = 'Usage', color = 'Department') +
  transition_reveal(along = Day)
anim <- animate(dailyusage_pidept, height = 640, width = 960)
anim_save("weeklypidept.gif",anim)

{width=900px}

dailyjobs <- daily %>% 
  select(c(Day,Total,Jobs=TotalJ)) %>% 
  separate(Day,c("Year","Month","Day"),sep="-", convert = TRUE) %>%
  group_by(Month,Day) %>%
  summarize(Total=round(sum(as.double(Total)),2),
            Jobs=round(sum(as.double(Jobs)))) %>%
  ggplot(aes(Day,Jobs,group = Month, color=Month)) +
  geom_line() + 
  scale_color_gradientn(colors=rainbow(12)) +
  geom_segment(aes(xend = 31, yend = Jobs), linetype = 2, color = 'grey') +
  geom_point(size = 2) + 
  geom_text(aes(x = 31.1, label = month(Month, label = TRUE)), hjust = 0)  + 
  transition_reveal(Day) + 
  coord_cartesian(clip = 'off') + 
  scale_y_log10() +
  labs(title = 'Day: {frame_along}', y = 'Jobs') + 
  theme_minimal() + theme(legend.position = 'none')
animate(dailyjobs, height = 640, width = 960)
anim <- animate(dailyjobs, height = 600, width = 960)
anim_save("dailyjobs.gif",anim)

• User documentation on CRAN
  • An Introduction on R: http://cran.r-project.org/doc/manuals/r-release/R-intro.html
• Online tutorials
  • http://www.cyclismo.org/tutorial/R/
• Educational R packages
  • Swirl: Learn R in R
• Online courses (e.g. Coursera)

• Typical data analysis workflow involves
  • Obtaining the data
  • Cleaning and preprocessing the data
  • Analyzing the data
  • Generating a report
• knitr is a R package that allows one to generate dynamic report by weaving R code and human readable texts together
  • It uses the markdown syntax
  • The output can be HTML, PDF or (even) Word
• slidify is a R package that allows one to create a HTML presentation
  • You are now at the end of a slidify presentation

• Install the devtools package and load it

install.packages('devtools')
library(devtools)

• Install the slidify and slidifyLibraries package from github

install_git('git://github.com/ramnathv/slidify')
install_git('git://github.com/ramnathv/slidifyLibraries')

• Load the slidify library

library(slidify)

• Create a Slide desk

author("myslides")

• This will create a folder called myslides with files and subdirectories to create your presentation
  • assets/css/custom.css: Create your own custom css
  • assets/layouts/: Don't like the default layouts, create your own in this directory
  • libraries: files that slidify create. Do not edit the files, copy the file to the assets directory and modify it.
• To edit your presentation, edit the index.Rmd file using R Markdown
• To create slides, in the R console run the command

slidify('index.Rmd')``
# View the presentation in a web browser
browserURL('index.html')

• Do a Google search for slidify to learn more and/or see example slides.

• To use my template

git clone https://gogs.cc.lehigh.edu/alp514/slidify

• Edit the index.Rmd or create a new .Rmd file
• OR just overwrite the assets folder in your myslides folder with the one from the git repository you just cloned
• The git repository contains a script compile.R that will compile the R markdown file (for e.g. index.Rmd) into a html file (for e.g. index.html)

chmod +x compile.R
./compile.R index