R | 𝚃𝚛𝚊𝚗𝚜𝚙𝚘𝚗𝚜𝚝𝚎𝚛

Sankey diagrams for Bacteria and antibiotics

Wed, 24 Jul 2019 21:13:14 -0500

Visually Classifying Bacteria and Antibiotics

After World War II, antibiotics earned the moniker “wonder drugs” for quickly treating previously-incurable diseases. Data was gathered to determine which drug worked best for each bacterial infection. Comparing drug performance was an enormous aid for practitioners and scientists alike. In the fall of 1951, Will Burtin published a graph showing the effectiveness of three popular antibiotics on 16 different bacteria, measured in terms of minimum inhibitory concentration.

image creidt: Ask a biologist

I am reproducing this wonderful visualization from my professor( Silas Bergen.) in ggplot2, who did this in Tableau

Let’s bring the datasets,

library(tidyverse)
library(knitr)
library(kableExtra)
df <- read.csv("https://cdn.rawgit.com/plotly/datasets/5360f5cd/Antibiotics.csv", stringsAsFactors = F)
#String as Factors is a demon. Better not bring it here ! We rarely need that beast.
#There are 16 bacteria so giving them ID to reference later..
df<-df %>% mutate(ID =seq(1:16) )

kable(head(df,n = 16))

Bacteria	Penicillin	Streptomycin	Neomycin	Gram	ID
Mycobacterium tuberculosis	800.000	5.00	2.000	negative	1
Salmonella schottmuelleri	10.000	0.80	0.090	negative	2
Proteus vulgaris	3.000	0.10	0.100	negative	3
Klebsiella pneumoniae	850.000	1.20	1.000	negative	4
Brucella abortus	1.000	2.00	0.020	negative	5
Pseudomonas aeruginosa	850.000	2.00	0.400	negative	6
Escherichia coli	100.000	0.40	0.100	negative	7
Salmonella (Eberthella) typhosa	1.000	0.40	0.008	negative	8
Aerobacter aerogenes	870.000	1.00	1.600	negative	9
Brucella antracis	0.001	0.01	0.007	positive	10
Streptococcus fecalis	1.000	1.00	0.100	positive	11
Staphylococcus aureus	0.030	0.03	0.001	positive	12
Staphylococcus albus	0.007	0.10	0.001	positive	13
Streptococcus hemolyticus	0.001	14.00	10.000	positive	14
Streptococcus viridans	0.005	10.00	40.000	positive	15
Diplococcus pneumoniae	0.005	11.00	10.000	positive	16

Before proceeding further with the data manipulation we need to think about the format of the visualization. Here we will be making our visualization on the bacteria level, that means we will have information for each bacteria, their gram stain , and the concentration of drug required .

If you look at the table above, we do have all the data we need but not on the format we are thinking. We want one information per row for each bacteria unlike above where each row has all the information of each bacteria on one single row. Let’s change the format of the data,

key_value = df %>% gather("Drug","Concentration",Penicillin:Neomycin,-Bacteria)
kable(head(key_value))

Bacteria	Gram	ID	Drug	Concentration
Mycobacterium tuberculosis	negative	1	Penicillin	800
Salmonella schottmuelleri	negative	2	Penicillin	10
Proteus vulgaris	negative	3	Penicillin	3
Klebsiella pneumoniae	negative	4	Penicillin	850
Brucella abortus	negative	5	Penicillin	1
Pseudomonas aeruginosa	negative	6	Penicillin	850

okay so, now what we need to do is add a minimum concentration information for each bacteria for each stain type. so basically a column on the gathered table above. The only thing to keep note of is that here we should group all these bacteria and select the minimum concentration. We could have done this first[basically for eacg ] and gather like above but this is my thought process.

df_min<- key_value  %>% 
  group_by(Bacteria) %>% summarise(Min = min(Concentration))
kable(head(df_min))

Bacteria	Min
Aerobacter aerogenes	1.000
Brucella abortus	0.020
Brucella antracis	0.001
Diplococcus pneumoniae	0.005
Escherichia coli	0.100
Klebsiella pneumoniae	1.000

so now, let’s join this df_min dataframe from above with df to have that minimum information in the dataframe.

df<- inner_join(df,df_min,by = "Bacteria")
df<- df %>% mutate(Best = case_when(
  Penicillin == Min~ "Penicillin",
  Neomycin == Min~ "Neomycin",
  Streptomycin == Min~ "Streptomycin"
))

Now, since the data is ready and in the format we want,

kable(head(df))

Bacteria	Penicillin	Streptomycin	Neomycin	Gram	ID	Min	Best
Mycobacterium tuberculosis	800	5.0	2.00	negative	1	2.00	Neomycin
Salmonella schottmuelleri	10	0.8	0.09	negative	2	0.09	Neomycin
Proteus vulgaris	3	0.1	0.10	negative	3	0.10	Neomycin
Klebsiella pneumoniae	850	1.2	1.00	negative	4	1.00	Neomycin
Brucella abortus	1	2.0	0.02	negative	5	0.02	Neomycin
Pseudomonas aeruginosa	850	2.0	0.40	negative	6	0.40	Neomycin

Okay, this step might be a little unintuitive but if we think with grammer of graphics philosophy this will make sense.

seq1 <- rep(1:16,each=100)
seq2 <-rep(seq(-6,6,length=100),16)
newdat <-data.frame(ID=seq1,T=seq2)
write.csv(newdat,"new_data.csv",row.names=FALSE)

We are making a new dataframe that has data point for the sigmoid curve(you can just draw sigmoid curve in R but this way it is linked with our data with ID)

#Joining the data by ID
final_df<-inner_join(df,newdat,by = "ID")
kable(head(final_df))

Bacteria	Penicillin	Streptomycin	Neomycin	Gram	ID	Min	Best	T
Mycobacterium tuberculosis	800	5	2	negative	1	2	Neomycin	-6.000000
Mycobacterium tuberculosis	800	5	2	negative	1	2	Neomycin	-5.878788
Mycobacterium tuberculosis	800	5	2	negative	1	2	Neomycin	-5.757576
Mycobacterium tuberculosis	800	5	2	negative	1	2	Neomycin	-5.636364
Mycobacterium tuberculosis	800	5	2	negative	1	2	Neomycin	-5.515151
Mycobacterium tuberculosis	800	5	2	negative	1	2	Neomycin	-5.393939

#ggplot
final_df <- final_df %>% mutate(Sigmoid = 1/(1 + exp(-T)))

okay so now we have the final dataset, we can get in the ggplot2 land.

p <- ggplot(data = final_df , aes(x = T , y = Sigmoid ))
p + geom_point()

#Making best slope
#Different slop will separate our curves
final_df<-final_df %>% mutate(bestBacSlope = case_when(
  Best =="Streptomycin" ~ 4 - ID,
  Best =="Neomycin" ~ 9 - ID,
  Best =="Penicillin" ~ 14 - ID
))

final_df<-final_df %>% mutate(curveBest = ID + bestBacSlope * Sigmoid)
#Figuring out ID and labels

label_df<-final_df %>% dplyr::select(c(ID, Bacteria))%>% group_by(Bacteria,ID) %>% summarise(count = n()) %>% dplyr::select(Bacteria,ID) %>% arrange(ID)

Below are the label we will use in y-axis

label_y= c("Mycobacterium tuberculosis" ,  "Salmonella schottmuelleri"  ,    
           "Proteus vulgaris"        ,        "Klebsiella pneumoniae"  ,        
           "Brucella abortus"      ,          "Pseudomonas aeruginosa"    ,     
           "Escherichia coli"    ,            "Salmonella (Eberthella) typhosa",
           "Aerobacter aerogenes"     ,       "Brucella antracis"    ,          
           "Streptococcus fecalis"    ,       "Staphylococcus aureus"      ,    
           "Staphylococcus albus"    ,        "Streptococcus hemolyticus"      ,
           "Streptococcus viridans"    ,      "Diplococcus pneumoniae")

Now it’s a plotting time !

#Plotting the sigmoid plots
library(ggthemes)

## Warning: package 'ggthemes' was built under R version 3.5.2

sankey <- ggplot(data = final_df, aes(x = T , y = curveBest, color =Gram,size = Min,alpha = 0.9,group = Bacteria)) + geom_line() +scale_fill_manual(values=c("green","red")) + 
    scale_y_continuous(breaks = seq(1:16) , labels = label_y)   + theme(axis.title.y = element_blank() , axis.line.x  = element_blank() , axis.ticks.x = element_blank(), axis.title.x =element_blank() , axis.text.x.bottom = element_blank() ) + 
  annotate("text", x = 6, y = 14, label = "Penicillin") +
  annotate("text", x = 6, y = 9, label = "Neomycin") +
  annotate("text", x = 6, y = 4, label = "Streptomycin") +
  annotate("text",x = 5.5,y = 15,label = "Best Antibiotics" ,size = 5, colour = 'blue')+
  theme_minimal()

sankey

Figure 1: Classification of Bacteria

Testing Alcohol level

Wed, 23 May 2018 21:13:14 -0500

Is there really 5.4% alcohol in that beer brand?

We all see that a lot of brand publish on their wrapper that the alcohol level is 5.4%. Let’s say we collected the percent level of volume for those brand. We sampled randomly and measured the alcohol level ourselves

So we believe that the actual beer percent should be 5.4% but as a beer consumer, we feel sometime it’s not.

if we measure one, and found out that beer has 6.7 we would immediately complain that the brand is telling us lie that there is 5.4% . They may argue that our measuring apparatus or technique is not 100% accurate. There is no way of finding our inaccurate our measurement without measuring it multiple times or taking measurement of multiple beers. It might be the case that our measurement is 100% accurate and the beer has more alcohol than the company is saying. We don’t really know. Also, we can’t measure every single beer they ever manufactured. This is the perfect timing to test this with our statistics sense, Below we have a list of measurements from different beer randomly bought, some from midtown, some from walmart. Let’s do a t-test.

level = c(5.1,5.2,6,7,5.01,5.0,6.5,5.6,5.2,6.1,6.2,5.0)
t.test(level, mu = 5.4)

## 
##  One Sample t-test
## 
## data:  level
## t = 1.3139, df = 11, p-value = 0.2156
## alternative hypothesis: true mean is not equal to 5.4
## 95 percent confidence interval:
##  5.225010 6.093323
## sample estimates:
## mean of x 
##  5.659167

The p-value is greater than 0.05 and confidence interval [5.17 to 6.17]. Which means if 100 people have done this random sampling of beer and have calculated the confidence interval , then the mean[5.4] would have always fall in the confidence interval.

Enough with the statistical jargon? Okay let’s enjoy the beer

Why do you have to wait more for the buses?

Sun, 23 Jul 2017 21:13:14 -0500

Average for group vs Individual

Inspection Paradox

Buses and trains are supposed to arrive at constant intervals, but in practice some intervals are longer than others. This means the buses do not follow schedule exactly. There is always some randomness..With your luck, you might think you are more likely to arrive during a long interval. It turns out you are right: a random arrival is more likely to fall in a long interval because, well, it’s longer..!

Let’s think of a scenario…

Suppose a Bus service in your city says they pass a station every 10 minutes. This means you will assume that when you go to station randomly you would think that the average time is 5 minutes but more often you will be waiting longer than five minutes actually 10 minutes on average.

Another example of this paradoxes is: Most of the school report there average class size. But if you, as a student that average is not accurate. Say, there are 4 classes of size 75,13,12,10. Then, the average colleges will report is \((75 + 13 +12 +10)/4 = 27.5\) but you as a prospective student, the average is different.

You are more likely to be in room with 75 students \(((75*75) + (13*13)+(12*12)+(10*10))/110 = 54.89\). Hence, the average reporting is not for you. This kind of paradoxes happen everywhere.

To generalize it in a more abstract way,

This is one case where the perspective of the individual and the group differs.For group, the average is what happens but as a individual the average will not make any sense.

Verifying empirical rule and Chebyshev's theorem

Mon, 01 Jan 0001 00:00:00 +0000

Empirical rule and Chebyshev’s theorem

Let’s talk about this really simple concept but powerful one. Data Distributions. A data distribution is an abstract concept(a function) that gives the the possible values of data and also how often that data is generated. When you want to talk about the all the data of your experiments at once, then talk about data distribution. A data distribution gives us the probability of how often that data will be an output if we keep repeating the experiment.

We rarely have the complete dataset from the experiment.So, it is powerful to have the an idea of how data is distributed and which data occurs more often than others. We can intuitively understand some distributions like the height of the populations. We know there will be few people with really short height while few have more height. But we are sure that most of the people will be in between.This is really convienient for us to know in advance the spread and frequency of the data.

Interesting thing is that there are more than one kinds of distributions in the world. So the convienience if knowing in advance the spread of the data will be helpful. There is a famous theorem that givrs us an idea of how our data is distributed. It’s called Chebyshev’s theorem.

image credit: libretext

It says that most(3/4th) of our data will be at max two standard deviations from the mean.

library(tidyverse)
library(knitr)
library(kableExtra)
stock<- read.csv("~/OneDrive - MNSCU/FALL 2019/MathStat/Data/Stock Trade.csv",stringsAsFactors = FALSE)

Now let’s clean the name,

stock<- stock %>% select(percentStock = X..of.Shares.Outstanding)

The empirical rule says that 68% of the data will be within two standard deviation.

This function below: 1. standardizes the data
2. counts data within z standard deviations
3. outputs the proportion

data_within<- function(df, z){
  func_normalize<-function(x){(x-mean(x))/sd(x)}
  #>11 after removing a data point 
  df<-df %>% filter(percentStock<11)
  df_scaled<- df %>% mutate(percentStock_normal = func_normalize(percentStock)) %>% filter(abs(percentStock_normal)<z)
  proportion = dim(df_scaled)[1]/dim(df)[1]
  return (round(proportion,2))
}

Let’s collect the output in a small tibble.

tb<- tibble(
  first_std_dev = data_within(stock,1),
  second_std_dev = data_within(stock,2),
  third_std_dev = data_within(stock,3),
)

kable(tb)

first_std_dev	second_std_dev	third_std_dev
0.64	0.92	1

We can also test if our function is working correctly,

library(testthat)

## Warning: package 'testthat' was built under R version 3.5.2

normal_generated = tibble(percentStock = rnorm(10,mean = 6.2,sd = 1.2))

#Testing our function
tb_test<- tibble(
  first_std_dev = data_within(normal_generated,1),
  second_std_dev = data_within(normal_generated,2),
  third_std_dev = data_within(normal_generated,3),
)


kable(tb_test)

first_std_dev	second_std_dev	third_std_dev
0.7	1	1

testthat::expect_gt(tb_test$first_std_dev, 0.68,label = "data proportion within first deivation")

Hence, our function is working correctly.Note that the data is randomly generated every time the code is run.