Welcome to our comprehensive guide on harnessing the power of the purrr package in R for functional programming. If you’re keen on elevating your R skills, you’re in for a treat. Today, we’ll be delving into the wonders of the purrr package — a lifesaver for functional programming. With the avalanche of data we encounter nowadays, having the right tools for efficient data wrangling is paramount. If you’ve dabbled in R, you might’ve felt certain built-in functions lacking, especially when grappling with intricate operations.

This is where purrr strides in, offering a plethora of robust tools to fine-tune your code, making it not only clearer but also more sustainable.

Throughout this article, we’ll journey through the intricacies of the purrr package, elucidate its fundamental functions, and showcase its real-world applicability. We’ll also touch on how it can enrich your experience with R, making it more fruitful. By the time you reach the end, you’ll be well-versed in the magic of purrr and ready to wield its power in your data endeavors. Let’s embark on this insightful voyage into the realm of R and unravel the capabilities of the purrr package.

What is functional programming and why is it useful?

Functional programming isn’t merely a way to write code; it’s a philosophical shift that guides how we approach computation. By treating computation as the evaluation of mathematical functions, it foregoes changes to the state and avoids mutable data. Instead, it thrives on pure functions that take given inputs and produce predictable outputs, devoid of side effects. The outcome? Code that’s more modular, predictable, and test-friendly.

Now, if you’re working with R, particularly for data manipulation and analysis, functional programming can be a game-changer. It lets you create more coherent and succinct code, and here’s how:

1. Enhanced Readability and Sustainability: Decomposing complex procedures into smaller, more digestible functions improves the understandability of your code. Plus, it’s easier to tweak as needed.
2. Boosted Productivity: By steering clear of traps like global variables, which may lead to unforeseen behaviors and debugging headaches, functional programming saves time and frustration.
3. Optimized Performance: Embracing functional programming could also enhance the efficiency of your code. It prompts the use of vectorized operations and cuts down on the necessity for explicit loops.

Eager to tap into these benefits? Read on! We’ll dive into the purrr package, an invaluable asset for adopting functional programming in R. Through its power, you can not only elevate your data manipulation and analysis routines but also bring more enjoyment and effectiveness to your programming journey.

Exploring the purrr package

Belonging to the tidyverse collection, the purrrpackage serves as R’s gateway to functional programming. It’s packed with dynamic functions crafted to ease tasks when working with lists and a variety of data structures. Adopting purrrensures that your data transformation, summarization, and manipulation processes benefit from a unified and logical syntax.

Let’s delve into what sets purrrapart:

1. Uniformity in Function Naming: One of purrrs’ strengths is its organized naming structure, simplifying the task of recalling and employing its functions.
2. Proficiency with Complex Data Structures: Be it nested lists, data frames, or any layered data structure, purrrstands out in its management capabilities.
3. Robust Error Management: Real-world data can be messy. purrrlends a hand by equipping you with functions that elegantly tackle errors and unexpected scenarios.
4. Harmony with tidyverse Companions: A significant advantage is purrrcompatibility with renowned tidyverse allies such as dplyr, tidyr, and ggplot2. This cohesion allows for a smoother integration of functional programming into your prevailing data routines.

Keen to commence your purrrjourney? Simply fetch it from CRAN and initialize it in your R workspace.

install.packages("purrr")
library(purrr)

In the next section, we will dive into the core functions provided by the purrr package and demonstrate their usage with practical examples.

Core functions in purrr

In this section, we will cover some of the most important and widely used functions in the purrr package, along with examples to demonstrate their usage.

A. The map() family

The map() family of functions is the heart of the purrr package. These functions allow you to apply a function to each element of a list or a vector and return the results in a specified format.

• map(): Returns a list.
• map_lgl(): Returns a logical vector.
• map_int(): Returns an integer vector.
• map_dbl(): Returns a double vector.
• map_chr(): Returns a character vector.
• map_df(): Returns a data frame.

Example:

# Define a list of numbers
number_list <- list(1, 2, 3, 4)

# Square each number using map()
squared_numbers <- map(number_list, ~ .x^2)
print(squared_numbers)

B. pmap()

The pmap() function is used to apply a function to elements of multiple lists simultaneously.

Example:

# Define two lists
list1 <- list(1, 2, 3)
list2 <- list(4, 5, 6)

# Add corresponding elements of the two lists using pmap()
sum_list <- pmap(list(list1, list2), ~ ..1 + ..2)
print(sum_list)

C. safely(), quietly(), and possibly()

These functions are used to handle errors and exceptions gracefully while applying a function.

• safely(): Returns a list containing the result and any error encountered.
• quietly(): Returns a list containing the result, any warnings, and any messages.
• possibly(): Returns a default value if an error is encountered.

Example:

# Define a list with numbers and a character
mixed_list <- list(1, 2, "a", 3)

# Define a safely wrapped square function
safe_square <- safely(~ .x^2)

# Apply the safe_square function to the mixed_list
results <- map(mixed_list, safe_square)
print(results)

D. compact() and compose()

compact() is used to remove NULL elements from a list, while compose() allows you to combine multiple functions into a single function.

Example:

# Define a list with NULL elements
null_list <- list(1, NULL, 2, NULL, 3)

# Remove NULL elements using compact()
clean_list <- compact(null_list)
print(clean_list)

# Compose two functions: square and increment
square <- function(x) x^2
increment <- function(x) x + 1
square_and_increment <- compose(increment, square)

# Apply the composed function to a number
result <- square_and_increment(3)
print(result)

These core functions are just the beginning of what purrr has to offer. In the next section, we will demonstrate how to use these functions to solve real-world problems through practical examples.

Practical examples with purrr

In this section, we will explore two practical examples that demonstrate how the purrr package can be used to solve real-world problems efficiently.

A. Example 1: Calculating summary statistics for multiple variables

Suppose you have a data frame with multiple numerical variables, and you want to calculate summary statistics (mean, median, and standard deviation) for each of these variables.

# Load required packages
library(dplyr)
library(purrr)

# Create a sample data frame
data <- data.frame(
  var1 = rnorm(100, mean = 10, sd = 2),
  var2 = rnorm(100, mean = 20, sd = 5),
  var3 = rnorm(100, mean = 30, sd = 3),
  stringsAsFactors = FALSE
)

# Define a list of summary functions
summary_functions <- list(mean = mean, median = median, sd = sd)

# Calculate summary statistics for each variable using nested map functions
summary_stats <- map_dfr(summary_functions, ~ map_dfc(data, .x), .id = "Statistic")
print(summary_stats)

B. Example 2: Fitting multiple linear models for different subsets of data

In this example, we will fit linear models for different subsets of the mtcars dataset based on the number of cylinders. We will use purrr functions to apply the linear model function to each subset and extract the model coefficients.

# Load required packages
library(dplyr)
library(purrr)
library(broom)

# Split the mtcars dataset by the number of cylinders
mtcars_split <- mtcars %>% group_split(cyl)

# Define a function to fit a linear model and extract coefficients
fit_lm <- function(data) {
  model <- lm(mpg ~ wt, data = data)
  coef <- data.frame(tidy(model)) %>%
    select(term, estimate) %>%
    mutate(cyl = unique(data$cyl))
  return(coef)
}

# Apply the fit_lm function to each subset using map_dfr()
model_coefs <- map_dfr(mtcars_split, fit_lm)
print(model_coefs)

C. Reading Multiple CSV files with purrr

Suppose you have multiple CSV files in a directory and you want to read them all into a single data frame using purrr. Here’s an example of how you can achieve this:

# Define the directory containing the CSV files
csv_directory <- "path/to/your/csv/files"

# List all CSV files in the directory
csv_files <- list.files(csv_directory, pattern = "*.csv", full.names = TRUE)

# Define a function to read a CSV file and add a column with the filename
read_csv_with_filename <- function(file) {
  data <- read_csv(file)
  data <- data %>% mutate(filename = basename(file))
  return(data)
}

# Read all CSV files using map_dfr() and bind the results into a single data frame
combined_data <- map_dfr(csv_files, read_csv_with_filename)

In this example, we first list all the CSV files in the specified directory. Then, we define a custom function read_csv_with_filename() to read each CSV file and add a column with the filename. Finally, we use purrr‘s map_dfr() function to apply the custom function to each file in the list and bind the results into a single data frame.

D. purrr and ggplot2

In this example, we’ll demonstrate how to use purrr to create multiple ggplots for different subsets of data within a single data frame. We’ll use the mtcars dataset and create separate ggplots for each unique number of cylinders.

# Load required packages
library(purrr)
library(ggplot2)
library(dplyr)
library(cowplot)

# Create a list of data frames, one for each unique number of cylinders in the mtcars dataset
data_list <- mtcars %>%
  split(.$cyl)

# Define a function to create a ggplot for a given data frame
create_ggplot <- function(data) {
  ggplot(data, aes(x = mpg, y = hp)) +
    geom_point(aes(color = factor(gear)), size = 3) +
    labs(title = paste("Number of Cylinders:", unique(data$cyl)),
         x = "Miles per Gallon",
         y = "Horsepower") +
    theme_minimal() +
    theme(legend.title = element_blank()) +
    scale_color_discrete(name = "Gears")
}

# Create a list of ggplots using map()
ggplot_list <- data_list %>% 
  map(create_ggplot)

# Combine the ggplots into a single plot using cowplot's plot_grid()
combined_plot <- plot_grid(plotlist = ggplot_list, ncol = 1, align = "v", rel_heights = c(1, 1, 1))

# Display the combined plot
print(combined_plot)

In this example, we first create a list of data frames, one for each unique number of cylinders in the mtcars dataset. Then, we define a custom function create_ggplot() to create a ggplot for a given data frame. The function creates a scatterplot of miles per gallon (mpg) versus horsepower (hp), with a title that reflects the number of cylinders.

Finally, we use purrr‘s map() function to apply the custom function to each data frame in the list, resulting in a list of ggplots. We use a for loop to display each ggplot.

The plot we get can be seen below:

In this example, we’ve made some changes to the create_ggplot() function to improve the aesthetics of the plots:

1. We use geom_point(aes(color = factor(gear)), size = 3) to color the points by the number of gears and increase their size.
2. We apply theme_minimal() to use a minimalistic theme for the plots.
3. We remove the legend title using theme(legend.title = element_blank()).
4. We rename the color scale to “Gears” using scale_color_discrete(name = "Gears").

Finally, we use the plot_grid() function from the cowplot package to combine the ggplots in the ggplot_list into a single plot with one column and display the combined plot.

These examples showcase how the purrr package can help you write more efficient and readable code, making your data analysis workflows more robust and maintainable. By incorporating purrr into your R projects, you can take full advantage of functional programming techniques and harness their power to solve complex problems.

Tips and Best Practices for Using purrr

In this final section, we will share some tips and best practices for using the purrr package in your R projects. These recommendations will help you write more efficient, readable, and maintainable code.

1. Use anonymous functions when appropriate

When using map() functions, you can create anonymous functions using the ~ notation, which allows for concise and readable code. However, if the function becomes too complex or is used multiple times, consider defining it as a separate named function for better code organization and readability.

2. Leverage the power of function composition

The compose() function allows you to create new functions by combining existing ones. This technique promotes code reusability and makes it easier to build complex functionality by breaking it down into simpler, more manageable parts.

3. Handle errors gracefully

When applying a function to a list or vector, use functions like safely(), quietly(), and possibly() to handle errors gracefully without stopping the execution of your code. This approach ensures that your code remains robust and can handle unexpected input values.

4. Know when to use purrr vs. base R or dplyr

While purrr provides a powerful and flexible way to manipulate data, there are cases where base R or dplyr functions may be more appropriate or efficient. For example, if you need to perform simple operations on a data frame, consider using dplyr functions like mutate() or summarize(). Evaluate the needs of your specific task and choose the best tool for the job.

5. Familiarize yourself with the purrr documentation

The purrr package has a wealth of functions and features that can help you streamline your code and solve complex problems. Make sure to consult the official documentation (https://purrr.tidyverse.org/) to explore its full capabilities and discover new techniques.

By following these tips and best practices, you can fully leverage the power of the purrr package in your R projects, making your code more efficient, readable, and maintainable. Embrace the functional programming paradigm and use purrr to solve real-world data analysis challenges with ease.

Wrapping up

Throughout this article, we’ve delved into the capabilities and adaptability of R’s purrr package in the realm of functional programming and data handling. Spanning from foundational functional programming principles to the pivotal role of the map() function suite, all the way to intricate subjects like engaging nested data sets and adept error management.

Using real-world scenarios, we’ve showcased how purrr can be instrumental in de-complicating daunting tasks, optimizing your scripts, and enhancing its legibility and sustainability. Incorporating purrr into your R utilities ensures a smoother journey through data manipulation and analytical hurdles.

As you venture further into the depths of the purrr package, bear in mind that mastery comes with repetition. Embrace exploration, and endeavor to ingeniously apply purrr functionalities in your endeavors. With perseverance, you’ll cultivate a profound grasp of its intricacies, propelling you towards proficient data management in R.

Happy coding!

Further Reading and Exploration:

For those eager to expand their expertise on purrr and R’s functional programming, consider the following treasure trove of resources:

1. purrr’s Official Guide: As a logical first step, the purrr package’s official documentation provides a thorough overview of all it offers. Dive into the nuances at purrr’s official site.
2. R for Data Science: A masterpiece penned by Hadley Wickham and Garrett Grolemund, this digital tome offers an exhaustive look into R’s role in data science. Notably, it features a segment dedicated to purrr’s prowess in functional programming. Grab your copy here.
3. Advanced R: A deeper dive by Hadley Wickham, “Advanced R” ventures into the more intricate aspects of R, shedding light on advanced functional programming paradigms. Embark on this advanced journey here.
4. RStudio’s Vibrant Community: Seeking advice, hoping to discuss new findings, or simply aiming to network? The RStudio community is a hub of enthusiasts, experts, and curious minds. Engage with like-minded individuals right here.

Harnessing these resources and proactively mingling with the wider R circle will undoubtedly refine your prowess with both the purrr package and R’s functional programming realm. Continue your journey of discovery, trial, and collaborative learning to blossom as an adept data scientist and R aficionado.

The post Unleash the Power of Functional Programming in R with the purrr Package appeared first on Analytica Data Science Solutions.

Introduction

Probability and statistics are essential disciplines for data scientists, analysts, and researchers. They provide a solid foundation for understanding, interpreting, and drawing meaningful conclusions from data. As the demand for data-driven insights and decision-making grows across various industries, a strong grasp of these concepts is crucial for anyone seeking a career in data science or looking to enhance their analytical skills.

In this blog post, we will introduce probability and statistics by exploring their basic concepts and terminology. We will explain the core principles and ideas that underpin these disciplines, using real-world examples and R code to help you visualize and understand these concepts in action. By incorporating visuals, we aim to make the material more engaging and easier to comprehend, allowing you to build a strong foundation for future learning.

Our journey will begin with probability fundamentals, including definitions, types of probability, and essential rules. We will then move on to descriptive statistics, discussing measures of central tendency, dispersion, and shape. Throughout the blog post, we will use the tidyverse package in R, focusing on ggplot2 for data visualization. This popular package offers a powerful and flexible way to create high-quality graphics, aiding in data exploration and communication.

By the end of this post, you will have a solid understanding of basic probability and descriptive statistics, supported by clear visualizations. This foundational knowledge will prepare you for more advanced topics and techniques in data analysis and machine learning, setting you on a path to success in the ever-evolving world of data science.

Stay tuned as we dive into the fascinating realm of probability and statistics, providing you with practical examples and insights to enhance your understanding and skills.

Probability Fundamentals

Probability is the study of randomness and uncertainty. It provides a way to quantify the likelihood of specific outcomes or events occurring in various situations. Understanding probability fundamentals is crucial for data science, as it underlies many statistical techniques and machine learning algorithms. In this section, we will elaborate on the basic concepts, principles, and rules of probability, using real-world data when possible.

I. Definitions:

• Experiment: An action or procedure that results in one of several possible outcomes. For example, rolling a die is an experiment with six possible outcomes (1, 2, 3, 4, 5, or 6).
• Outcome: The result of an experiment. In the die-rolling example, if the die lands on 3, then the outcome is 3.
• Event: A set of one or more outcomes. In the die-rolling example, an event could be the die showing an even number, which includes the outcomes {2, 4, 6}.
• Sample Space: The set of all possible outcomes of an experiment. For the die-rolling example, the sample space is {1, 2, 3, 4, 5, 6}.

II. Types of Probability:

• Classical: Based on the assumption that all outcomes are equally likely. In the die-rolling example, the classical probability of getting an even number is 1/2 (3 even numbers out of 6 possible outcomes).
• Relative Frequency: Based on the observed frequencies of outcomes in a sample. For instance, suppose we roll a die 100 times and observe 40 even numbers. The relative frequency of getting an even number is 40/100 = 0.4.
• Subjective: Based on an individual’s personal judgment or belief. A person may believe that it is more likely to rain tomorrow based on their interpretation of weather patterns and past experiences, even if objective data suggests otherwise.

Let’s use a real-world dataset to illustrate the relative frequency approach. We will analyze the number of cylinders in vehicles from the mtcars dataset, which is included in R. We will calculate the relative frequency of vehicles with 4, 6, and 8 cylinders.

library(knitr)
data(mtcars)

cylinder_counts <- mtcars %>% count(cyl)
total_cars <- sum(cylinder_counts$n)
relative_frequencies <- cylinder_counts %>% 
mutate(relative_frequency = n / total_cars)

kable(relative_frequencies, caption = "Relative Frequencies of Cylinder Counts in Vehicles")

In the table above, the “cyl” column represents the number of cylinders in a vehicle, the “n” column shows the count of vehicles with the corresponding number of cylinders, and the “relative_frequency” column displays the relative frequency of each cylinder count in the dataset.

Now, let’s visualize the relative frequencies using ggplot2.

library(tidyverse)

ggplot(relative_frequencies, aes(x = factor(cyl), y = relative_frequency)) +
  geom_col(fill = "steelblue") +
  labs(title = "Relative Frequency of Cylinder Counts in Vehicles",
       x = "Number of Cylinders",
       y = "Relative Frequency") +
  theme_minimal()

Another way to demonstrate the relative frequency approach is to simulate a die experiment, we will roll a fair six-sided die 1000 times and calculate the relative frequency of each outcome (1, 2, 3, 4, 5, and 6). Additionally, we will visualize the results using ggplot2.

set.seed(42) # Set seed for reproducibility
n_rolls <- 1000
die_rolls <- sample(1:6, size = n_rolls, replace = TRUE)

die_rolls_df <- data.frame(outcome = die_rolls) %>%
  count(outcome) %>%
  mutate(relative_frequency = n / n_rolls)

die_rolls_df

This will generate a data frame with the outcome, count, and relative frequency of each die roll:

Now, let’s create a bar chart to visualize the relative frequencies:

ggplot(die_rolls_df, aes(x = factor(outcome), y = relative_frequency)) +
  geom_col(fill = "steelblue") +
  labs(title = "Die Roll Simulation",
       x = "Outcome",
       y = "Relative Frequency") +
  theme_minimal()

The resulting bar chart displays the relative frequencies of each outcome from the die roll simulation. The chart illustrates the concept of relative frequency by showing the proportion of each outcome observed in the experiment.

III. Probability Rules:

Addition Rule: The addition rule helps us calculate the probability of either event A or event B (or both) occurring. The rule is defined as:

P(A ∪ B) = P(A) + P(B) — P(A ∩ B)

Here, P(A ∪ B) represents the probability of event A or event B occurring, while P(A ∩ B) denotes the probability of both events A and B happening together.

Example: Suppose we have a deck of 52 playing cards. What is the probability of drawing either a red card (hearts or diamonds) or a queen?

There are 26 red cards and 4 queens in the deck, but 2 of the queens are also red cards (queen of hearts and queen of diamonds). So, applying the addition rule:

P(Red ∪ Queen) = P(Red) + P(Queen) — P(Red ∩ Queen)
P(Red ∪ Queen) = (26/52) + (4/52) — (2/52) = 28/52 ≈ 0.5385

Multiplication Rule: The multiplication rule helps us determine the probability of both events A and B occurring simultaneously. The rule is defined as:

P(A ∩ B) = P(A|B) * P(B)

Here, P(A|B) represents the probability of event A occurring given that event B has occurred.

Example: Consider a bag containing 5 blue and 3 red balls. We draw two balls from the bag without replacement. What is the probability of drawing a blue ball first, followed by a red ball?

To apply the multiplication rule, we first calculate the probability of each event:

P(Blue1) = 5/8 P(Red2|Blue1) = 3/7

Now, we can compute the probability of both events happening together:

P(Blue1 ∩ Red2) = P(Blue1) * P(Red2|Blue1) = (5/8) * (3/7) ≈ 0.2679

In the next part of this series, we will cover topics related to Descriptive Statistics.

For more practical tips and insights on AI, data science, and statistics, explore our blog at Analytica Data Science Solutions. Discover engaging content to expand your knowledge and stay up-to-date with the latest developments.

In this blog post, we have covered the basics of probability and statistics. If you wish to further expand your knowledge and understanding, here are some references to help you dive deeper into these topics:

Books:

1. DeGroot, M. H., & Schervish, M. J. (2012). Probability and Statistics (4th ed.). Pearson.
2. Wackerly, D., Mendenhall, W., & Scheaffer, R. L. (2007). Mathematical Statistics with Applications (7th ed.). Cengage Learning.
3. Casella, G., & Berger, R. L. (2001). Statistical Inference (2nd ed.). Duxbury Press.
4. Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd ed.). Springer.
5. Wickham, H., & Grolemund, G. (2016). R for Data Science: Import, Tidy, Transform, Visualize, and Model Data. O’Reilly Media.

Websites:

1. Khan Academy — Probability and Statistics: https://www.khanacademy.org/math/statistics-probability
2. Stat Trek — Teach yourself statistics: https://stattrek.com/
3. Carnegie Mellon University Probability & Statistics: https://oli.cmu.edu/courses/probability-statistics-open-free/

Online Courses:

1. Coursera — Statistics with R Specialization by Duke University: https://www.coursera.org/specializations/statistics
2. Coursera — Introduction to Probability and Data with R by Duke University: https://www.coursera.org/learn/probability-intro
3. edX — Probability and Statistics in Data Science using Python by the University of California, San Diego: https://www.edx.org/course/probability-and-statistics-in-data-science-using-python
4. DataCamp — Introduction to Probability in R: https://www.datacamp.com/courses/introduction-to-probability-in-r
5. DataCamp — Foundations of Probability in R: https://www.datacamp.com/courses/foundations-of-probability-in-r

Read More blogs in AnalyticaDSS Blogs here : BLOGS

Read More blogs in Medium : Medium Blogs

The post Introduction to Probability and Statistics: Basic Concepts and Terminology with Visuals — Part I appeared first on Analytica Data Science Solutions.

The power of R comes from the vast collection of software libraries, i.e. packages, that can be easily installed and loaded in R. Today we will cover two of the most powerful packages in R, the tidyverse and data.table packages.

The tidyverse and data.table are two popular packages in R that provide functions for working with data. They both have their own strengths and are suitable for different types of tasks.

The tidyverse is a collection of packages designed for data manipulation, visualization, and modeling. It is based on the principles of tidy data, which suggests that data should be structured in a way that makes it easy to work with. The tidyverse includes packages such as dplyr, tidyr, and ggplot2, which provides functions for data manipulation, cleaning, and visualization.

One of the main advantages of the tidyverse is its simplicity. The functions in the tidyverse are easy to learn and use, and they often require fewer lines of code compared to other packages. They also have a consistent syntax, which makes it easier to learn and use multiple functions.

Examples: Tidyverse Examples

Here are some examples of how to use the tidyverse:

To select specific columns from a dataset:

# Load the tidyverse package
library(tidyverse)

# Load the mpg dataset from the ggplot2 package
data(mpg)

# Select the "manufacturer" and "model" columns
mpg %>% select(manufacturer, model)

And to group and summarize a dataset:

# Load the tidyverse package
library(tidyverse)

# Load the mpg dataset from the ggplot2 package
data(mpg)

# Group the dataset by "class" and compute the mean of the "hwy" column
mpg %>% group_by(class) %>% summarize(mean_hwy = mean(hwy))

To join two datasets:

# Load the tidyverse package
library(tidyverse)

# Load the mpg and cylinders datasets from the ggplot2 package
data(mpg)
data(cylinders)

# Join the mpg and cylinders datasets on the "manufacturer" column
mpg %>% left_join(cylinders, by = "manufacturer")

To perform a linear regression using the lm function from the stats package:

# Load the tidyverse and stats packages
library(tidyverse)
library(stats)

# Load the mtcars dataset
data(mtcars)

# Perform a linear regression to predict mpg (miles per gallon) using wt (weight) as the predictor variable
fit <- mtcars %>% 
  lm(mpg ~ wt, data = .)

# Summarize the model results
summary(fit)

Create a scatterplot matrix using the scatterplotMatrix function from the car package:

# Load the tidyverse and car packages
library(tidyverse)
library(car)

# Load the iris dataset
data(iris)

# Create a scatterplot matrix of the iris dataset
scatterplotMatrix(iris, smooth = FALSE)

Create a faceted bar plot using ggplot2:

# Load the tidyverse package
library(tidyverse)

# Load the mpg dataset from the ggplot2 package
data(mpg)

# Create a faceted bar plot showing the distribution of hwy (highway miles per gallon) by class and drv (drive type)
ggplot(mpg, aes(x = hwy)) +
  geom_histogram(binwidth = 2) +
  facet_wrap(~ class + drv, nrow = 2)

Examples: data.table Examples

The data.table package, on the other hand, is a high-performance package for working with large datasets. It provides functions for manipulating and querying data efficiently. The data.table package is particularly useful when working with datasets that are too large to fit in memory or when you need to perform complex operations on large datasets.

One of the main advantages of the data.table package

One of the main advantages of the data.table package is its speed. The functions in the data.table package are generally faster than their counterparts in the tidyverse, especially when working with large datasets.

Here are some more examples of how to use the data.table package:

To select specific columns from a dataset:

# Load the data.table package
library(data.table)

# Load the mpg dataset from the ggplot2 package
data(mpg)

# Convert the dataset to a data.table
mpg <- as.data.table(mpg)

# Select the "manufacturer" and "model" columns
mpg[, .(manufacturer, model)]

and to group and summarize a dataset:

# Load the data.table package
library(data.table)

# Load the mpg dataset from the ggplot2 package
data(mpg)

# Convert the dataset to a data.table
mpg <- as.data.table(mpg)

# Group the dataset by "class" and compute the mean of the "hwy" column
mpg[, .(mean_hwy = mean(hwy)), by = class]

To join two datasets:

# Load the data.table package
library(data.table)

# Load the mpg and cylinders datasets from the ggplot2 package
data(mpg)
data(cylinders)

# Convert the datasets to data.tables
mpg <- as.data.table(mpg)
cylinders <- as.data.table(cylinders)

# Join the mpg and cylinders datasets on the "manufacturer" column
mpg[cylinders, on = "manufacturer"]

Perform a linear regression using the lm function from the stats package and the data.table package:

# Load the data.table and stats packages
library(data.table)
library(stats)

# Load the mtcars dataset
data(mtcars)

# Convert the dataset to a data.table
mtcars <- setDT(mtcars)

# Perform a linear regression to predict mpg (miles per gallon) using wt (weight) as the predictor variable
fit <- mtcars[, lm(mpg ~ wt)]

# Summarize the model results
summary(fit)

Create a scatterplot matrix using the scatterplotMatrix function from the car package and the data.table package:

# Load the data.table and car packages
library(data.table)
library(car)

# Load the iris dataset
data(iris)

# Convert the dataset to a data.table
iris <- as.data.table(iris)

# Create a scatterplot matrix of the iris dataset
scatterplotMatrix(iris, smooth = FALSE)

Create a faceted bar plot using ggplot2 and the data.table package:

# Load the data.table and ggplot2 packages
library(data.table)
library(ggplot2)

# Load the mpg dataset from the ggplot2 package
data(mpg)

# Convert the dataset to a data.table
mpg <- as.data.table(mpg)

# Create a faceted bar plot showing the distribution of hwy (highway miles per gallon) by class and drv (drive type)
ggplot(mpg, aes(x = hwy)) +
  geom_histogram(binwidth = 2) +
  facet_wrap(~ class + drv, nrow = 2)

In terms of implementation, both the tidyverse and data.table packages are written in R, but some of the functions in the data.table package are implemented in C for improved performance.

In summary

the tidyverse and data.table are two popular packages in R that provide functions for working with data. The tidyverse is a collection of packages designed for data manipulation, visualization, and modeling, and it is particularly suitable for tasks that require simplicity and ease of use. The tidyverse functions are easy to learn and use, and they often require fewer lines of code compared to other packages.

The data.table package is a high-performance package for working with large datasets, and it is particularly useful when working with large datasets or when you need to perform complex operations on large datasets. The functions in the data.table package are generally faster than their counterparts in the, especially when working with large datasets.

In general, it is a good idea to use the tidyverse for most tasks, unless you are working with very large datasets or need the extra performance provided by the data.table package.

At Analytica

and since we deal with larger datasets, GB to TB of data, our preferred tool for data wrangling in R is in fact data.table.

I hope this article helps the reader understand the differences between the tidyverse and data.table in R, and how to choose the right package for their tasks. Let me know if you have any questions.

Read More blogs in AnalyticaDSS Blogs here : BLOGS

Read More blogs in Medium : Medium Blogs

Read More blogs in R-bloggers : https://www.r-bloggers.com

The post The Tidyverse and data.table R Packages appeared first on Analytica Data Science Solutions.

Analyzing crypto market

Aous Abdo, WWW.ANALYTICADSS.COM
An interactive version of this post can be found on here.

In my previous post I explored bitcoin data from different exchanges, we also covered some arbitrage-related data. In part 2 of this series I will explore alt coin related data.

R Libraries

Below is a list of R libraries we will be using to help us with our analysis. Not all of them are necessary but they all will make our life easier.

library(PoloniexR)
library(data.table)
library(lubridate)
library(Quandl)
library(plyr)
library(stringr)
library(ggplot2)
library(plotly)
library(janitor)
library(quantmod)
library(pryr)
library(corrplot)
library(PerformanceAnalytics)
library(tidyr)
library(MLmetrics)
library(tidyquant)
library(corrr)
library(cowplot)

Data

The best source I know off to get alt-coin data is through PoloniexR. I have written an R function to help download data.

get_alt_data <- function(tz = "UTC"
                         , coin = c("ETH", "LTC")
                         , add_bitcoin = TRUE
                         , return_in_USDT = TRUE
                         , from = "2017-01-01"
                         , to = "2018-04-09"
                         , period = "D"
                         , verbose = FALSE){
  
  # We will be using the public API
  poloniex.public <- PoloniexPublicAPI()
  
  # set the time zone to utc
  Sys.setenv(tz = tz)
  
  # convert from and to into time obj
  from  <- as.POSIXct(paste(from, tz, sep = ""))
  to    <- as.POSIXct(paste(to, tz, sep = ""))
  
  # lists to store data.tables and xts objects
  chart_list <- list()
  dt_list    <- list()
  
  # make sure the coin pair is in upper case
  coin       <- toupper(coin)
  coin_pairs <- paste0("BTC_", coin[coin != "BTC"])
  if(add_bitcoin | return_in_USDT) coin_pairs <- c("USDT_BTC", coin_pairs)
  
  # loop over the coins to get the data
  for(i in coin_pairs){
    if(verbose)
      invisible(cat('\tGetting data for ', i, ' pair\n'))
    
    # this is a list that will contain the chart data for each coin pair
    try(chart_list[[i]] <- ReturnChartData(theObject = poloniex.public
                                       , pair      = i
                                       , from      = from
                                       , to        = to
                                       , period    = period)
        , silent = TRUE)
    
    # list to contain data.tables 
    try(dt_list[[i]] <- as.data.table(chart_list[[i]]), silent = TRUE)
  }
  
  # convert to data.table and make sure to add a column containing the pairs
  coin_dt <- rbindlist(l = dt_list, use.names = TRUE, idcol = "pair")
  
  # return data in usdt prices
  if(return_in_USDT){
    # to get the price of the alt coin in usdt is not that simple but we'll do it
    # get a DT of the btc_usdt pair
    btc_usd <- coin_dt[pair == "USDT_BTC"]
    btc_usd <- btc_usd[, .(index, pair, weightedaverage)]
    setnames(btc_usd, c("Date", "USDT_BTC_pair", "USDT_BTC_price"))
    
    # get DT with only alt coins
    alt_coins <- copy(coin_dt)#[pair != "USDT_BTC"]
    
    # now we need to add an index to the alt_coins table, but first we have to rename the index column
    alt_coins[, Date := index]
    alt_coins[, index := 1:.N]
    setkey(alt_coins, index)
    
    # now merge the data tables
    coin_dt_usdt <- merge(x = alt_coins, y = btc_usd, by = "Date")
    
    # now calcualte the price in usdt
    coin_dt_usdt[, price_usdt := ifelse(pair == "USDT_BTC", USDT_BTC_price, weightedaverage * USDT_BTC_price)]
    
    # now get rid of the extra columns
    coin_dt_usdt[, c("USDT_BTC_price", "USDT_BTC_pair") := NULL]
    
    # we need to change some column names
    col_names_to_change <- c("pair", "high", "low", "open", "close", "volume", "quotevolume", "weightedaverage")
    col_names <- names(coin_dt_usdt)
    col_names[col_names %in% col_names_to_change] <- paste0(col_names_to_change, '_btc')
    
    setnames(coin_dt_usdt, col_names)
    
    # add a column for the usdt pair
    coin_dt_usdt[, pair_usdt := gsub("BTC_", "USDT_", pair_btc)]
    
    # adjust col order
    setcolorder(coin_dt_usdt, c(1:10, 12, 11))
    
    # set key again
    setkey(coin_dt_usdt, index)
    
    # now get rid of the index column since it is not needed anymore
    coin_dt_usdt[, index := NULL]
    
    # now put together the return list  
    return_list <- list(alt_chart_list = chart_list, alt_dt = coin_dt, alt_usdt_dt = coin_dt_usdt)
  }else{
    return_list <- list(alt_chart_list = chart_list, alt_dt = coin_dt)
  }
  
  return(return_list)
}

The function above can be used to download data for multiple coin at the same time. The function returns a data.table object with data for all coins in the function call. Even if the user doesn’t add bitcoin to the list of coins, the function adds bitcoin by default. This can be deactivated with the add_bitcoin argument. Here is an example

# get alt data for some coins
alt_data <- get_alt_data(return_in_USDT = T
                         , from = "2015-01-01"
                         , coin = c('ETH','XRP', 'BCH', 'LTC', 'NEO', 'XMR', 'DASH', 'XEM'))[['alt_usdt_dt']]

Let’s look at the data we just downloaded

head(alt_data)

The table shows the date, OHLC, Volume, and weightedaverage price in BTC. It also shows the pair and we added the price in USD.

Bitcoin-Altcoins Correlations

Wheneven I look at the prices of the coins available on my coinbase app I always get struck by the similarity of the price trends between the four coins available on coinbase: BTC, ETH, BCH, and LTC, see Figure below. So I thought it will be a good idea to explore the correlation in price trends between altcoins and bitcoin.

Let’s look at price trends of the coins we just downloaded. To better see potential correlations I am going to only zoom in on 2018.

p <- ggplot(alt_data[year(Date) == 2018], aes(x = Date, y =  price_usdt, col = pair_usdt)) + geom_line()
p <- p + facet_wrap(~pair_usdt, scales = "free", ncol = 3) + theme_minimal() + theme(legend.position="none") + ylab("Price (USD)")
p

The figure above shows that some coins seems to be more correlated with Bitcoin than others. The figure also shows that this variablity between Bitcoin and another coin varies over time. More on this below.

Tyring to find correlations bewteen time series data using Pearson correlation coefficient or other metrics used with stationary data, time series is not a form of stationary data, can give misleading results. Similar trends in time series data can also be very misleading, a nice article on this topic can be found here. And always remember that Correlation doesn’t guarantee Causation

Bottom line is the following, one has to be careful when cross-correlating time serice. In order to perform proper correlation analysis we need to add some new variables to our table.

Percentage Daily Change

Percentage daily change calculates the price change of a coin over a period of a day. Let’s add that to the table. Notice that we are calcualting this variable using the USD price, and not the price in Bitcoin.

# add daily price change
alt_data[, pct_change := Delt(price_usdt), by = pair_usdt]

Normalized Price in USD

Since the prices vary a lot, both overtime for the same coin and between coins, we will add a variable of the normalized price in USD. This variable will make it easy to plot prices of coins on the same figure.

# add normalized prices in udst
alt_data[, price_usdt_norm := price_usdt/max(price_usdt), by = pair_usdt]

Now that we have the normalized prices in USD, let’s look at the prices of bitcoin and litcoin on the same figure. We’ll do that for 2018 so we can better see any possible correlations.

p <- ggplot(alt_data[year(Date) == 2018 & pair_usdt %like% "BTC|LTC"], aes(x = Date, y =  price_usdt_norm, col = pair_usdt)) + geom_line()
p <- p + theme_minimal() + ylab("Price (USD)")
ggplotly(p)

The trends in the prices of BTC and LTC are very similar, Let’s look at price trends for 2017

p <- ggplot(alt_data[year(Date) == 2017 & pair_usdt %like% "BTC|LTC"], aes(x = Date, y =  price_usdt_norm, col = pair_usdt)) + geom_line()
p <- p + theme_minimal() + ylab("Price (USD)")
ggplotly(p)

Seems like we need to zoon in on the last quarter of 2017, let’s do that

p <- ggplot(alt_data[Date >= "2017-10-01"  & Date < "2018-01-01" &  pair_usdt %like% "BTC|LTC"], aes(x = Date, y =  price_usdt_norm, col = pair_usdt)) + geom_line()
p <- p + theme_minimal() + ylab("Price (USD)")
ggplotly(p)

It is clear from the above figure that the correlation in the prices of bitcoin and LTC vary over time. Note how the highest price for bitcoin on December 17 2017, preceded that of LTC by two days, which occurred on December 19 2017. This wasn’t the case for the ATH which occurred on January 6th 2018 for both coins.

Static Correlations

(and why you shouldn’t use them with crypto!)

Up until now I haven’t calculated any correltaions between the price of different coins. You might ask why should we even care about correlations in time series. Well, in the case of financial time series data, if one can show that a correlation exists between two time series then one can use this correlation to model/predict the price movement of one coin/stock given the price trends of another coin/stock.

However, as we mentioned earlier, correlation for time series data is not static, it changes over time. Actually let’s show that. To do that I am going to be calculating the Pearson correlation coefficient. In simple words, Pearson correlation coefficient for two vectors of data is a measure that shows how correlated these two vectors of data are. The value of this coefficient varies from -1, perfectly anti-correlated, to 1, perfectly correlated. So the correlation coefficient for a series of numbers on itself is 1. A value of zero means these is no correlation. Remember, this only works for static data.

In order to perform correlation on our data I am going to need to do some data transformation:

# subset data, only keep the date, the pair, and the price
alt_data_sub <- alt_data[, .(Date, pair_usdt, price_usdt)]
# convert to wide format 
alt_data_sub <- spread(data = alt_data_sub, key = "pair_usdt", value = "price_usdt")
# clean column names
setnames(alt_data_sub, gsub("USDT_", "", colnames(alt_data_sub)))

The new table we created contains the date along with the prices in USDT for each coin we have in our table.

tail(alt_data_sub)

Again, what I am doing here is not correct, I am just trying to show you why we shouldn’t be doing static correlations on crypto data. Now we’ll calculate the Pearson correlation coefficient between the coins we have, then we are going to make a nice plot of these coefficients.

# calculate the correlation matrix
M <- cor(alt_data_sub[, -1], use = "complete.obs") # notice how we are ignoring missing data with the last argument
# plot the correlation matrix
corrplot.mixed(corr = M, upper = "ellipse", lower = "number", order = "AOE", tl.col = "black")

The figure above shows the correlation coefficients between the different coins. It is easy to read, visually, the darker the color of the ellipse, and the more diagonal the ellipse, the higher the correlation coefficient. Of course you can also just look at the numbers on the bottom left part of the figure to get the value of the coefficient between two coins :). The figure shows how highly correlated the prices of crypto currencies can be. For example XRP and XEM have a correlation coefficient of 0.93. The highest correlation seems to be between BCH and DASH at 0.97 correlation coefficient.

All of the correlation coefficient we see in the above figure are significant, the question is, do these correlations vary over time. To answer this question I will calculate the correlation coefficient between Bitcoin and DASH on a monthly basis, you can do that for any time period, and will show that this coefficient varies greatly over time. Let’s do that

# subset the data
btc_dash <- alt_data_sub[, .(Date, BTC, DASH)]
# add a year_month column
btc_dash[, year_month := as.yearmon(Date)]
# calculate the correlation coefficient on montly basis
btc_dash_2 <- btc_dash[, cor(BTC, DASH), by = year_month]
# now plot the correlation coefficient as a function of month and year
plot(btc_dash_2$year_month, btc_dash_2$V1, xlab = "Year-Month", main = "Correlation Coeff. Between BTC and DASH Over time"
     , ylab = "Correlation Coefficient", type = "b", pch = 19, col = ifelse(btc_dash_2$V1 > 0, "blue", "red")
     , ylim = c(-1, 1))

This is interesting, the value of the monthly correlation coefficient between bitcoin and DASH varies between -0.91, highly anti-correlated, to 0.98, highly correlated. And this is why you should never use static correlation metrics with crypto data!

A good blog post on this same topic is written by Tom Fawcett from Silicon Valley Data Science and can be found here. In his post Tom shows, with simple simulations, why static correlations should never be used with time series.

Correlation Networks

There is one more plot I would like to make, which is a network plot of the correlations between the different coins. The correlation network plot helps show strengths of correlation between the different coins. Agian, these correlations are time dependent and the figure we will be making will change over time, but I still think it is a good figure to make. Here it is:

# we will be using the great corrr package for this work
# get the correlation matrix, just like we did before
# build the correlation matrix 
# the code snippets below are taken from, that is a great blog BTW 
# http://www.business-science.io/timeseries-analysis/2017/07/30/tidy-timeseries-analysis-pt-3.html
corr_2 <- correlate(alt_data_sub[, -1])
# make the network plot
# Network plot
corr_net <- corr_2 %>%
  network_plot(colours = c(palette_light()[[2]], "white", palette_light()[[4]]), legend = TRUE) +
  labs(
    title = "Static Correlations of some Crypto Currencies",
    subtitle = "2014 through 2018"
  ) +
  theme_tq() +
  theme(legend.position = "bottom")
corr_net

The figure above shows a network which measures how strongly correlated the prices of the coins under stugy are. The darker the color of the edge, line, connecting two coins and the closer they are in the network the stronger the correlation between these two coins.

From the figure, it seems like XMR, LTC, and BTC are in the heart of this network, while BCH seems to be the least correlated with the rest of the coins. Let’s see how the network plot changes between 2017 and 2018:

# subset the data and get correlation matrices
corr_2017 <- correlate(alt_data_sub[year(Date) == 2017][, -1])
corr_2018 <- correlate(alt_data_sub[year(Date) == 2018][, -1])
# build Network plots
corr_net_2017 <- corr_2017 %>%
  network_plot(colours = c(palette_light()[[2]], "white", palette_light()[[4]]), legend = TRUE) +
  labs(
    title = "Static Correlations of some Crypto Currencies",
    subtitle = "2017"
  ) +
  theme_tq() +
  theme(legend.position = "bottom")
corr_net_2018 <- corr_2018 %>%
  network_plot(colours = c(palette_light()[[2]], "white", palette_light()[[4]]), legend = TRUE) +
  labs(
    title = "Static Correlations of some Crypto Currencies",
    subtitle = "2018"
  ) +
  theme_tq() +
  theme(legend.position = "bottom")
# combine network plots
cow_net_plots <-plot_grid(corr_net_2017, corr_net_2018, ncol = 2)
title <- ggdraw() + 
    draw_label(label = 'Crypto Correlation Networks',
               fontface = 'bold', size = 18)
cow_out <- plot_grid(title, cow_net_plots, ncol=1, rel_heights=c(0.1, 1))
cow_out

As can be seen, the correlation networks do change overtime. This is not news since we already saw in the previous section that the value of the correlation varies overtime (I know we showed this to be true for the BTC-DASH air but we’ll show that this is true for the rest of the coins in the next section.)

Daily Returns Correlations

Let’s look at the percentage daily changes of the altcoins between 2015 and today.

# plot the percent changes
p <- ggplot(alt_data[Date > ymd("2015-01-01")], aes(x = Date, y =  (100*pct_change), col = pair_usdt)) + geom_line()
p <- p + ggtitle("% Daily Returns over time") + ylab("Daily Return (%)") 
p <- p + theme_bw() + guides(col=guide_legend(title="Coin Pair"))
ggplotly(p)

Although the above figure is very cluttered, one thing is certain, percentage daily returns vary greatly for crypto. Let’s try to make this figure a bit easier to read

p <- ggplot(alt_data[Date > ymd("2015-01-01")], aes(x = Date, y =  (100*pct_change), col = pair_usdt)) + geom_line() + facet_wrap(~ pair_usdt)
p <- p + ggtitle("Percentage Daily Returns over time") + ylab("Daily Return (%)") 
p <- p + theme_bw() + theme(legend.position="none") 
ggplotly(p)

It is kind of surprising that Bitcoin has the least variability in daily returns. The nice big spike around April 2nd 2017 shows a percentage daily return of ~88% for XRP, this is the highest daily return I have seen!

Let’s look at the percentage daily returns for Bitcoin and Litecoin since they seem to be highly correlated. I am going to zoom in on the time period 2016–02–01 and 2016–05–01.

start_date <- ymd("2016-02-01")
end_date <- ymd("2016-05-01")
p <- ggplot(alt_data[pair_usdt %like% "BTC|LTC" & Date > start_date & Date < end_date], aes(x = Date, y =  (100*pct_change), col = pair_usdt)) + geom_line() + theme_bw() + ylab("Price (USD)")
p

The figure shows clear correlation between the daily returns of Bitcoin and litcoin. It also shows that these correlations can vary overtime. In fact, let’s look at how these correlations vary overtime.

# these steps are similar to the ones in the previous section, the only differnect is that now we are looking at the percentage change in price difference on daily basis instead of the actual price
# subset data, only keep the date, the pair, and the price
alt_data_sub_pct <- alt_data[, .(Date, pair_usdt, pct_change)]
# convert to wide format 
alt_data_sub_pct <- spread(data = alt_data_sub_pct, key = "pair_usdt", value = "pct_change")
# clean column names
setnames(alt_data_sub_pct, gsub("USDT_", "", colnames(alt_data_sub)))
# subset the data
btc_ltc <- alt_data_sub_pct[, .(Date, BTC, LTC)]
# add a year_month column
btc_ltc[, year_month := as.yearmon(Date)]
# calculate the correlation coefficient on montly basis
btc_ltc_2 <- btc_ltc[, cor(BTC, LTC), by = year_month]
# now plot the correlation coefficient as a function of month and year
plot(btc_ltc_2$year_month, btc_ltc_2$V1, xlab = "Year-Month", main = "Correlation Coeff. Between Daily Returns of BTC and LTC Over time"
     , ylab = "Correlation Coefficient", type = "b", pch = 19, col = ifelse(btc_ltc_2$V1 > 0, "blue", "red")
     , ylim = c(-1, 1))

Interesting, the correlation between the percentage daily change of the prices for bitcoin and litecoin is much more on the positive side, we only have one month in which this correlation is negtive, barely negative. This is a lot different than what we saw between Bitcoin and DASH, but that was for the actual prices and not the daily returns. Let’s redo this plot but his time for the actual prices for bitcoin and litecoin, just like we did with DASH.

# subset the data
btc_ltc_price <- alt_data_sub[, .(Date, BTC, LTC)]
# add a year_month column
btc_ltc_price[, year_month := as.yearmon(Date)]
# calculate the correlation coefficient on montly basis
btc_ltc_price_2 <- btc_ltc_price[, cor(BTC, LTC), by = year_month]
# now plot the correlation coefficient as a function of month and year
plot(btc_ltc_price_2$year_month, btc_ltc_price_2$V1, xlab = "Year-Month", main = "Correlation Coeff. Between BTC and Litecoin Over time"
     , ylab = "Correlation Coefficient", type = "b", pch = 19, col = ifelse(btc_ltc_price_2$V1 > 0, "blue", "red")
     , ylim = c(-1, 1))

Trends in the correlations of the daily return of bitcoin and litecoin on mothly basis, boy this is a mouth full, are very similar to those for the prices as we saw in the previous figure.

In the next post we’ll do something more statistically sound, rolling correlations.

Read More blogs in AnalyticaDSS Blogs here : BLOGS

Read More blogs in Medium : Medium Blogs

Read More blogs in R-bloggers : https://www.r-bloggers.com

The post Analyzing Crypto Market using R — Part 2 appeared first on Analytica Data Science Solutions.

Analyzing crypto market

Aous Abdo, WWW.ANALYTICADSS.COM
An interactive version of this post can be found here.

No doubt that crypto currencies with all the promises they bring, both financially and otherwise, are only here to stay. As a data scientist interested in data and numbers, I thought it would be nice to take a look at some crypto currencies with my favorite tool, R.

R Libraries

Below is a list of R libraries we will be using to help us with our analysis. Not all of them are necessary but they all will make our life easier.

library(PoloniexR)
library(data.table)
library(lubridate)
library(Quandl)
library(plyr)
library(stringr)
library(ggplot2)
library(plotly)
library(janitor)
library(quantmod)
library(pryr)
library(corrplot)
library(PerformanceAnalytics)
library(tidyr)
library(MLmetrics)
library(readr)

Getting the Data

1. PoloniexR Package

The easiest way to get current and historical data for cyrpto currencies is by using the PoloniexR developed by Vermeir Jellen. Vermeir Jellen gives a good tutorial on how to start with his package here. The Poloniex exchange includes many coins but not all. For missiong coins on Poloniex, one can scrape the coinmarketcap page, an example is given here.

2. Quandl

Quandl is my go to place for any financial data. Their free-tier API has lots of good data one can use. Quandl offers data from multiple exchanges. Locating crypto data on Quandl is not straight forward. After spending few hours on their site I found out that most of the crypto data can be found here

First, let’s take a look at different exchange data for Bitcoin using Quandl. We will download and plot historical bitcoin data from the following exchanges Kraken, Coinbase, Bitstamp, and ITBIT

# enable your Quandl API key
my_quandl_api_key <- read_file("../../quandl_api_key.txt")
Quandl.api_key(my_quandl_api_key)
# function to download quandl data
get_quandl_data <- function(data_source = "BITFINEX"
                            , pair = 'btcusd'
                            , ...){
  
  # make sure the user supplied the correct data_source
  if(toupper(data_source) != "BITFINEX") stop("data source supplied is wrong...")
  # quandl is case sensitive, all codes have to be upper case
  pair <- toupper(pair)
  tmp <- NA
  try(tmp <- Quandl(code = toupper(paste(data_source, pair, sep = "/")), ...), silent = TRUE)
  return(tmp)
}
# get btc data from different exchanges
  exchange_data <- list()
  
  exchanges <- c('KRAKENUSD','COINBASEUSD','BITSTAMPUSD','ITBITUSD')
  
  for (i in exchanges){
    exchange_data[[i]] <- Quandl(paste0('BCHARTS/', i))
  }

So We need to convert this list of BTC prices from different exchanges into a dataframe and put them all in one data frame so we can plot them.

# put them all in one dataframe to plot in ggplot2
btc_usd <- do.call("rbind", exchange_data)
btc_usd$exchange <- row.names(btc_usd)
btc_usd <- as.data.table(btc_usd)

We also need to do some minor cleaning, so let’s do that. We also need to get rid of rows of data with 0 weighted price.

# some data cleaning
btc_usd[, exchange := as.factor(str_extract(exchange, "[A-Z]+"))]
btc_usd <- clean_names(btc_usd)
btc_usd <- btc_usd[weighted_price > 0]
# set datatable key to be the date column
setkey(btc_usd, date)

Let’s take a look at the data table we just made.

head(btc_usd)

The data includes 10 columns, the date, OCHL prices, volumes in USD and BTC, the weighted price, and the exchange. I wish I had bought me some bitcoine back in 2011!!!

Now we’ll look at the price of bitcoin and color code it by exchange.

ggplot(btc_usd, aes(x = date, y = weighted_price, col = exchange)) + geom_line() + theme_bw()

Arbitrage

It appears the prices of btc on different exchanges are fairly consisant. But this is an artifact in the figure since we are covering several orders of magnitudes during the timeline we selected. To better see any price differenes we need to zoon in on the figure. Let’s zoom in on, say the first month of 2018, were we had the ATH for all coins. This will enable us to better see any differences in prices.

ggplot(btc_usd[date >= ymd("2018-01-01") & date <= ymd("2018-01-31")], aes(x = date, y = weighted_price, col = exchange)) + geom_line() + theme_bw()

There are obvious differences in prices between the exchanges. Differences seem to vary over time as well. Actually it will be interesting to look at the maxiumum price differences as a function of time, let’s do that

# first let's find the minimum price by date
btc_usd[, min_price := min(weighted_price), by = date]
# now we need to find the price difference between the price for each day and the minimum price for that day
# but since the price of bitcoin varies a lot for the time period under study, we need to normalize the price difference
# to do that we will just divide by the median price for each day
btc_usd[, price_diff := 100*(weighted_price - min_price)/median(weighted_price), by = (date)]

Now we have a new column which gives us the percentage of price differences for each day normalized to the median price for each day. Let’s take a look at the new table.

tail(btc_usd)

The reason I looked at the newer dates is that prior to 2014 we only have data for one exchange, so all the price differences were 0. Let’s take a look at the price differences as a function of time.

# first we need to create a new data table with only the maxiumum prices per day
tmp <- btc_usd[, price_diff := max(price_diff), by = date]
# This will help us visualize overlapping points
MyGray <- rgb(t(col2rgb("black")), alpha=50, maxColorValue=255)
tmp[, plot(date, price_diff, pch=20, col = MyGray, xlab = "Date", ylab = "Maximum of Price Difference (%)")]

Let’s show the plot with log scale on y axis. Let’s also discard dates with zero price differences.

tmp[price_diff > 0, plot(date, price_diff, pch=20, col = MyGray, log = "y" , xlab = "Date", ylab = "Maximum of Price Difference (%)")]

As one can see from the figure above, the bulk of the maximum difference in bitcoin prices between the different exchanges is in the 0.5–2.0% range. It is also interesting to see that the differences in prices seem to have come down between 2014 and 2016, but they seem to go up starting in 2017. Let’s fit a gam model to see what we get.

# we'll use ggplot and fit a gam smooth line
ggplot(tmp[price_diff > 0 ], aes(x = date, y = price_diff)) + geom_point(alpha = 0.2, shape = 16, size = 3, show.legend = FALSE) + scale_y_continuous(trans='log10') + geom_smooth(method = "gam", formula = y ~ s(x, bs = "cs")) + theme_bw() + xlab("Date") + ylab("Maximum of Price Difference (%)")

The regression line shows a hint of an over all downtrend from 2014 to mid 2016, except for an uptrend for few months in late 2015. The trend seems to have gone up in mid to late 2017, and again we see a downword movement in price differences starting in December of 2017. This can be seen better in the box-plot figure below.

tmp[, month_year := format(as.Date(date), "%Y-%m")]
ggplot(tmp[price_diff > 0], aes(x = month_year, y = price_diff)) + geom_boxplot() + scale_y_continuous(trans='log10') + xlab("Date (Year-Month)") + ylab("Maximum of Price Difference (%)") + theme_bw() + theme(axis.text.x = element_text(angle = 90, hjust = 1))

The box-plot figure above shows the variation of maximum differences in prices as a function of time. On the x-axis I grouped dates by month since anything less than a one-month period will result in congested figure.

Okay, now let’s find out which of the exchanges contribute the most to these price difference. That is, we are trying to determine which exchanges are constantly selling bitcoin higher, or lower, than the rest of the exchanges. We need to pull some numbers as below, and then we’ll make a bar plot to show the leading exchanges in each category.

# add two columns to our data table which will contain the minimum and maximum prices
tmp[, `:=`(day_max = max(weighted_price), day_min = min(weighted_price)), by = date]
# now put only the columns we care about in a new data.table
tmp2 <- tmp[price_diff > 0 , .(date, exchange, weighted_price, day_min, day_max)]
# notice how we excluded days with no price difference
# now we only want to keep the rows with the maximum and minimum daily prices
tmp2 <- tmp2[weighted_price == day_min | weighted_price == day_max]
# now we'll add a new column designating the price as being the minimum or maximum
tmp2[, max_min := ifelse(weighted_price == day_min, "min", "max")]
# clean the name of the exchange
tmp2[, exchange := gsub("USD", "", exchange)]
# now we'll add a new column containing the exchange name and the min_max column
tmp2[, max_min_exchange := paste(max_min, exchange, sep = "-")]

In the above chunk of code we created a new table which contains the maximum and minimum prices for each day. The table also contains a categorical column showing to which exchange this max/min price belong, and if the price was a maxima or a minima. Before we plot the table above, let’s have a quick look at it.

head(tmp2)

The max_min_exchange column contains all the data we need, so let’s make a barplot of this variable, we’ll color the barplot by the max_min criteria shown in max_min column

# now make a barplot 
ggplot(tmp2, aes(x = max_min_exchange, fill = max_min)) + geom_bar() + theme_bw() + theme(axis.text.x = element_text(angle = 90, hjust = 1)) + ggtitle("Exchanges with the highest and lowest price differences in Bitcoin") + ylab("Frequency") + xlab("Exchange") + scale_fill_discrete(name = "BTC Price Diff. Type")

This is interesting, Kraken seems to be the exchange with the most frequent highest prices for bitcoin. On the other hand, Bitstamp seems to be the one with the most frequent lowest prices among exchanges. So if you want to do arbitrage your best bit is to buy on Bitstamp and sell on Kraken.

Read More blogs in AnalyticaDSS Blogs here : BLOGS

Read More blogs in Medium : Medium Blogs

Read More blogs in R-bloggers : https://www.r-bloggers.com

The post Analyzing Crypto Markets using R — Part 1 appeared first on Analytica Data Science Solutions.