Welcome to our comprehensive guide on harnessing the power of the purrr package in R for functional programming. If you’re keen on elevating your R skills, you’re in for a treat. Today, we’ll be delving into the wonders of the purrr package — a lifesaver for functional programming. With the avalanche of data we encounter nowadays, having the right tools for efficient data wrangling is paramount. If you’ve dabbled in R, you might’ve felt certain built-in functions lacking, especially when grappling with intricate operations.

This is where purrr strides in, offering a plethora of robust tools to fine-tune your code, making it not only clearer but also more sustainable.

Throughout this article, we’ll journey through the intricacies of the purrr package, elucidate its fundamental functions, and showcase its real-world applicability. We’ll also touch on how it can enrich your experience with R, making it more fruitful. By the time you reach the end, you’ll be well-versed in the magic of purrr and ready to wield its power in your data endeavors. Let’s embark on this insightful voyage into the realm of R and unravel the capabilities of the purrr package.

What is functional programming and why is it useful?

Functional programming isn’t merely a way to write code; it’s a philosophical shift that guides how we approach computation. By treating computation as the evaluation of mathematical functions, it foregoes changes to the state and avoids mutable data. Instead, it thrives on pure functions that take given inputs and produce predictable outputs, devoid of side effects. The outcome? Code that’s more modular, predictable, and test-friendly.

Now, if you’re working with R, particularly for data manipulation and analysis, functional programming can be a game-changer. It lets you create more coherent and succinct code, and here’s how:

1. Enhanced Readability and Sustainability: Decomposing complex procedures into smaller, more digestible functions improves the understandability of your code. Plus, it’s easier to tweak as needed.
2. Boosted Productivity: By steering clear of traps like global variables, which may lead to unforeseen behaviors and debugging headaches, functional programming saves time and frustration.
3. Optimized Performance: Embracing functional programming could also enhance the efficiency of your code. It prompts the use of vectorized operations and cuts down on the necessity for explicit loops.

Eager to tap into these benefits? Read on! We’ll dive into the purrr package, an invaluable asset for adopting functional programming in R. Through its power, you can not only elevate your data manipulation and analysis routines but also bring more enjoyment and effectiveness to your programming journey.

Exploring the purrr package

Belonging to the tidyverse collection, the purrrpackage serves as R’s gateway to functional programming. It’s packed with dynamic functions crafted to ease tasks when working with lists and a variety of data structures. Adopting purrrensures that your data transformation, summarization, and manipulation processes benefit from a unified and logical syntax.

Let’s delve into what sets purrrapart:

1. Uniformity in Function Naming: One of purrrs’ strengths is its organized naming structure, simplifying the task of recalling and employing its functions.
2. Proficiency with Complex Data Structures: Be it nested lists, data frames, or any layered data structure, purrrstands out in its management capabilities.
3. Robust Error Management: Real-world data can be messy. purrrlends a hand by equipping you with functions that elegantly tackle errors and unexpected scenarios.
4. Harmony with tidyverse Companions: A significant advantage is purrrcompatibility with renowned tidyverse allies such as dplyr, tidyr, and ggplot2. This cohesion allows for a smoother integration of functional programming into your prevailing data routines.

Keen to commence your purrrjourney? Simply fetch it from CRAN and initialize it in your R workspace.

install.packages("purrr")
library(purrr)

In the next section, we will dive into the core functions provided by the purrr package and demonstrate their usage with practical examples.

Core functions in purrr

In this section, we will cover some of the most important and widely used functions in the purrr package, along with examples to demonstrate their usage.

A. The map() family

The map() family of functions is the heart of the purrr package. These functions allow you to apply a function to each element of a list or a vector and return the results in a specified format.

• map(): Returns a list.
• map_lgl(): Returns a logical vector.
• map_int(): Returns an integer vector.
• map_dbl(): Returns a double vector.
• map_chr(): Returns a character vector.
• map_df(): Returns a data frame.

Example:

# Define a list of numbers
number_list <- list(1, 2, 3, 4)

# Square each number using map()
squared_numbers <- map(number_list, ~ .x^2)
print(squared_numbers)

B. pmap()

The pmap() function is used to apply a function to elements of multiple lists simultaneously.

Example:

# Define two lists
list1 <- list(1, 2, 3)
list2 <- list(4, 5, 6)

# Add corresponding elements of the two lists using pmap()
sum_list <- pmap(list(list1, list2), ~ ..1 + ..2)
print(sum_list)

C. safely(), quietly(), and possibly()

These functions are used to handle errors and exceptions gracefully while applying a function.

• safely(): Returns a list containing the result and any error encountered.
• quietly(): Returns a list containing the result, any warnings, and any messages.
• possibly(): Returns a default value if an error is encountered.

Example:

# Define a list with numbers and a character
mixed_list <- list(1, 2, "a", 3)

# Define a safely wrapped square function
safe_square <- safely(~ .x^2)

# Apply the safe_square function to the mixed_list
results <- map(mixed_list, safe_square)
print(results)

D. compact() and compose()

compact() is used to remove NULL elements from a list, while compose() allows you to combine multiple functions into a single function.

Example:

# Define a list with NULL elements
null_list <- list(1, NULL, 2, NULL, 3)

# Remove NULL elements using compact()
clean_list <- compact(null_list)
print(clean_list)

# Compose two functions: square and increment
square <- function(x) x^2
increment <- function(x) x + 1
square_and_increment <- compose(increment, square)

# Apply the composed function to a number
result <- square_and_increment(3)
print(result)

These core functions are just the beginning of what purrr has to offer. In the next section, we will demonstrate how to use these functions to solve real-world problems through practical examples.

Practical examples with purrr

In this section, we will explore two practical examples that demonstrate how the purrr package can be used to solve real-world problems efficiently.

A. Example 1: Calculating summary statistics for multiple variables

Suppose you have a data frame with multiple numerical variables, and you want to calculate summary statistics (mean, median, and standard deviation) for each of these variables.

# Load required packages
library(dplyr)
library(purrr)

# Create a sample data frame
data <- data.frame(
  var1 = rnorm(100, mean = 10, sd = 2),
  var2 = rnorm(100, mean = 20, sd = 5),
  var3 = rnorm(100, mean = 30, sd = 3),
  stringsAsFactors = FALSE
)

# Define a list of summary functions
summary_functions <- list(mean = mean, median = median, sd = sd)

# Calculate summary statistics for each variable using nested map functions
summary_stats <- map_dfr(summary_functions, ~ map_dfc(data, .x), .id = "Statistic")
print(summary_stats)

B. Example 2: Fitting multiple linear models for different subsets of data

In this example, we will fit linear models for different subsets of the mtcars dataset based on the number of cylinders. We will use purrr functions to apply the linear model function to each subset and extract the model coefficients.

# Load required packages
library(dplyr)
library(purrr)
library(broom)

# Split the mtcars dataset by the number of cylinders
mtcars_split <- mtcars %>% group_split(cyl)

# Define a function to fit a linear model and extract coefficients
fit_lm <- function(data) {
  model <- lm(mpg ~ wt, data = data)
  coef <- data.frame(tidy(model)) %>%
    select(term, estimate) %>%
    mutate(cyl = unique(data$cyl))
  return(coef)
}

# Apply the fit_lm function to each subset using map_dfr()
model_coefs <- map_dfr(mtcars_split, fit_lm)
print(model_coefs)

C. Reading Multiple CSV files with purrr

Suppose you have multiple CSV files in a directory and you want to read them all into a single data frame using purrr. Here’s an example of how you can achieve this:

# Define the directory containing the CSV files
csv_directory <- "path/to/your/csv/files"

# List all CSV files in the directory
csv_files <- list.files(csv_directory, pattern = "*.csv", full.names = TRUE)

# Define a function to read a CSV file and add a column with the filename
read_csv_with_filename <- function(file) {
  data <- read_csv(file)
  data <- data %>% mutate(filename = basename(file))
  return(data)
}

# Read all CSV files using map_dfr() and bind the results into a single data frame
combined_data <- map_dfr(csv_files, read_csv_with_filename)

In this example, we first list all the CSV files in the specified directory. Then, we define a custom function read_csv_with_filename() to read each CSV file and add a column with the filename. Finally, we use purrr‘s map_dfr() function to apply the custom function to each file in the list and bind the results into a single data frame.

D. purrr and ggplot2

In this example, we’ll demonstrate how to use purrr to create multiple ggplots for different subsets of data within a single data frame. We’ll use the mtcars dataset and create separate ggplots for each unique number of cylinders.

# Load required packages
library(purrr)
library(ggplot2)
library(dplyr)
library(cowplot)

# Create a list of data frames, one for each unique number of cylinders in the mtcars dataset
data_list <- mtcars %>%
  split(.$cyl)

# Define a function to create a ggplot for a given data frame
create_ggplot <- function(data) {
  ggplot(data, aes(x = mpg, y = hp)) +
    geom_point(aes(color = factor(gear)), size = 3) +
    labs(title = paste("Number of Cylinders:", unique(data$cyl)),
         x = "Miles per Gallon",
         y = "Horsepower") +
    theme_minimal() +
    theme(legend.title = element_blank()) +
    scale_color_discrete(name = "Gears")
}

# Create a list of ggplots using map()
ggplot_list <- data_list %>% 
  map(create_ggplot)

# Combine the ggplots into a single plot using cowplot's plot_grid()
combined_plot <- plot_grid(plotlist = ggplot_list, ncol = 1, align = "v", rel_heights = c(1, 1, 1))

# Display the combined plot
print(combined_plot)

In this example, we first create a list of data frames, one for each unique number of cylinders in the mtcars dataset. Then, we define a custom function create_ggplot() to create a ggplot for a given data frame. The function creates a scatterplot of miles per gallon (mpg) versus horsepower (hp), with a title that reflects the number of cylinders.

Finally, we use purrr‘s map() function to apply the custom function to each data frame in the list, resulting in a list of ggplots. We use a for loop to display each ggplot.

The plot we get can be seen below:

In this example, we’ve made some changes to the create_ggplot() function to improve the aesthetics of the plots:

1. We use geom_point(aes(color = factor(gear)), size = 3) to color the points by the number of gears and increase their size.
2. We apply theme_minimal() to use a minimalistic theme for the plots.
3. We remove the legend title using theme(legend.title = element_blank()).
4. We rename the color scale to “Gears” using scale_color_discrete(name = "Gears").

Finally, we use the plot_grid() function from the cowplot package to combine the ggplots in the ggplot_list into a single plot with one column and display the combined plot.

These examples showcase how the purrr package can help you write more efficient and readable code, making your data analysis workflows more robust and maintainable. By incorporating purrr into your R projects, you can take full advantage of functional programming techniques and harness their power to solve complex problems.

Tips and Best Practices for Using purrr

In this final section, we will share some tips and best practices for using the purrr package in your R projects. These recommendations will help you write more efficient, readable, and maintainable code.

1. Use anonymous functions when appropriate

When using map() functions, you can create anonymous functions using the ~ notation, which allows for concise and readable code. However, if the function becomes too complex or is used multiple times, consider defining it as a separate named function for better code organization and readability.

2. Leverage the power of function composition

The compose() function allows you to create new functions by combining existing ones. This technique promotes code reusability and makes it easier to build complex functionality by breaking it down into simpler, more manageable parts.

3. Handle errors gracefully

When applying a function to a list or vector, use functions like safely(), quietly(), and possibly() to handle errors gracefully without stopping the execution of your code. This approach ensures that your code remains robust and can handle unexpected input values.

4. Know when to use purrr vs. base R or dplyr

While purrr provides a powerful and flexible way to manipulate data, there are cases where base R or dplyr functions may be more appropriate or efficient. For example, if you need to perform simple operations on a data frame, consider using dplyr functions like mutate() or summarize(). Evaluate the needs of your specific task and choose the best tool for the job.

5. Familiarize yourself with the purrr documentation

The purrr package has a wealth of functions and features that can help you streamline your code and solve complex problems. Make sure to consult the official documentation (https://purrr.tidyverse.org/) to explore its full capabilities and discover new techniques.

By following these tips and best practices, you can fully leverage the power of the purrr package in your R projects, making your code more efficient, readable, and maintainable. Embrace the functional programming paradigm and use purrr to solve real-world data analysis challenges with ease.

Wrapping up

Throughout this article, we’ve delved into the capabilities and adaptability of R’s purrr package in the realm of functional programming and data handling. Spanning from foundational functional programming principles to the pivotal role of the map() function suite, all the way to intricate subjects like engaging nested data sets and adept error management.

Using real-world scenarios, we’ve showcased how purrr can be instrumental in de-complicating daunting tasks, optimizing your scripts, and enhancing its legibility and sustainability. Incorporating purrr into your R utilities ensures a smoother journey through data manipulation and analytical hurdles.

As you venture further into the depths of the purrr package, bear in mind that mastery comes with repetition. Embrace exploration, and endeavor to ingeniously apply purrr functionalities in your endeavors. With perseverance, you’ll cultivate a profound grasp of its intricacies, propelling you towards proficient data management in R.

Happy coding!

Further Reading and Exploration:

For those eager to expand their expertise on purrr and R’s functional programming, consider the following treasure trove of resources:

1. purrr’s Official Guide: As a logical first step, the purrr package’s official documentation provides a thorough overview of all it offers. Dive into the nuances at purrr’s official site.
2. R for Data Science: A masterpiece penned by Hadley Wickham and Garrett Grolemund, this digital tome offers an exhaustive look into R’s role in data science. Notably, it features a segment dedicated to purrr’s prowess in functional programming. Grab your copy here.
3. Advanced R: A deeper dive by Hadley Wickham, “Advanced R” ventures into the more intricate aspects of R, shedding light on advanced functional programming paradigms. Embark on this advanced journey here.
4. RStudio’s Vibrant Community: Seeking advice, hoping to discuss new findings, or simply aiming to network? The RStudio community is a hub of enthusiasts, experts, and curious minds. Engage with like-minded individuals right here.

Harnessing these resources and proactively mingling with the wider R circle will undoubtedly refine your prowess with both the purrr package and R’s functional programming realm. Continue your journey of discovery, trial, and collaborative learning to blossom as an adept data scientist and R aficionado.

The post Unleash the Power of Functional Programming in R with the purrr Package appeared first on Analytica Data Science Solutions.

Introduction

Probability and statistics are essential disciplines for data scientists, analysts, and researchers. They provide a solid foundation for understanding, interpreting, and drawing meaningful conclusions from data. As the demand for data-driven insights and decision-making grows across various industries, a strong grasp of these concepts is crucial for anyone seeking a career in data science or looking to enhance their analytical skills.

In this blog post, we will introduce probability and statistics by exploring their basic concepts and terminology. We will explain the core principles and ideas that underpin these disciplines, using real-world examples and R code to help you visualize and understand these concepts in action. By incorporating visuals, we aim to make the material more engaging and easier to comprehend, allowing you to build a strong foundation for future learning.

Our journey will begin with probability fundamentals, including definitions, types of probability, and essential rules. We will then move on to descriptive statistics, discussing measures of central tendency, dispersion, and shape. Throughout the blog post, we will use the tidyverse package in R, focusing on ggplot2 for data visualization. This popular package offers a powerful and flexible way to create high-quality graphics, aiding in data exploration and communication.

By the end of this post, you will have a solid understanding of basic probability and descriptive statistics, supported by clear visualizations. This foundational knowledge will prepare you for more advanced topics and techniques in data analysis and machine learning, setting you on a path to success in the ever-evolving world of data science.

Stay tuned as we dive into the fascinating realm of probability and statistics, providing you with practical examples and insights to enhance your understanding and skills.

Probability Fundamentals

Probability is the study of randomness and uncertainty. It provides a way to quantify the likelihood of specific outcomes or events occurring in various situations. Understanding probability fundamentals is crucial for data science, as it underlies many statistical techniques and machine learning algorithms. In this section, we will elaborate on the basic concepts, principles, and rules of probability, using real-world data when possible.

I. Definitions:

• Experiment: An action or procedure that results in one of several possible outcomes. For example, rolling a die is an experiment with six possible outcomes (1, 2, 3, 4, 5, or 6).
• Outcome: The result of an experiment. In the die-rolling example, if the die lands on 3, then the outcome is 3.
• Event: A set of one or more outcomes. In the die-rolling example, an event could be the die showing an even number, which includes the outcomes {2, 4, 6}.
• Sample Space: The set of all possible outcomes of an experiment. For the die-rolling example, the sample space is {1, 2, 3, 4, 5, 6}.

II. Types of Probability:

• Classical: Based on the assumption that all outcomes are equally likely. In the die-rolling example, the classical probability of getting an even number is 1/2 (3 even numbers out of 6 possible outcomes).
• Relative Frequency: Based on the observed frequencies of outcomes in a sample. For instance, suppose we roll a die 100 times and observe 40 even numbers. The relative frequency of getting an even number is 40/100 = 0.4.
• Subjective: Based on an individual’s personal judgment or belief. A person may believe that it is more likely to rain tomorrow based on their interpretation of weather patterns and past experiences, even if objective data suggests otherwise.

Let’s use a real-world dataset to illustrate the relative frequency approach. We will analyze the number of cylinders in vehicles from the mtcars dataset, which is included in R. We will calculate the relative frequency of vehicles with 4, 6, and 8 cylinders.

library(knitr)
data(mtcars)

cylinder_counts <- mtcars %>% count(cyl)
total_cars <- sum(cylinder_counts$n)
relative_frequencies <- cylinder_counts %>% 
mutate(relative_frequency = n / total_cars)

kable(relative_frequencies, caption = "Relative Frequencies of Cylinder Counts in Vehicles")

In the table above, the “cyl” column represents the number of cylinders in a vehicle, the “n” column shows the count of vehicles with the corresponding number of cylinders, and the “relative_frequency” column displays the relative frequency of each cylinder count in the dataset.

Now, let’s visualize the relative frequencies using ggplot2.

library(tidyverse)

ggplot(relative_frequencies, aes(x = factor(cyl), y = relative_frequency)) +
  geom_col(fill = "steelblue") +
  labs(title = "Relative Frequency of Cylinder Counts in Vehicles",
       x = "Number of Cylinders",
       y = "Relative Frequency") +
  theme_minimal()

Another way to demonstrate the relative frequency approach is to simulate a die experiment, we will roll a fair six-sided die 1000 times and calculate the relative frequency of each outcome (1, 2, 3, 4, 5, and 6). Additionally, we will visualize the results using ggplot2.

set.seed(42) # Set seed for reproducibility
n_rolls <- 1000
die_rolls <- sample(1:6, size = n_rolls, replace = TRUE)

die_rolls_df <- data.frame(outcome = die_rolls) %>%
  count(outcome) %>%
  mutate(relative_frequency = n / n_rolls)

die_rolls_df

This will generate a data frame with the outcome, count, and relative frequency of each die roll:

Now, let’s create a bar chart to visualize the relative frequencies:

ggplot(die_rolls_df, aes(x = factor(outcome), y = relative_frequency)) +
  geom_col(fill = "steelblue") +
  labs(title = "Die Roll Simulation",
       x = "Outcome",
       y = "Relative Frequency") +
  theme_minimal()

The resulting bar chart displays the relative frequencies of each outcome from the die roll simulation. The chart illustrates the concept of relative frequency by showing the proportion of each outcome observed in the experiment.

III. Probability Rules:

Addition Rule: The addition rule helps us calculate the probability of either event A or event B (or both) occurring. The rule is defined as:

P(A ∪ B) = P(A) + P(B) — P(A ∩ B)

Here, P(A ∪ B) represents the probability of event A or event B occurring, while P(A ∩ B) denotes the probability of both events A and B happening together.

Example: Suppose we have a deck of 52 playing cards. What is the probability of drawing either a red card (hearts or diamonds) or a queen?

There are 26 red cards and 4 queens in the deck, but 2 of the queens are also red cards (queen of hearts and queen of diamonds). So, applying the addition rule:

P(Red ∪ Queen) = P(Red) + P(Queen) — P(Red ∩ Queen)
P(Red ∪ Queen) = (26/52) + (4/52) — (2/52) = 28/52 ≈ 0.5385

Multiplication Rule: The multiplication rule helps us determine the probability of both events A and B occurring simultaneously. The rule is defined as:

P(A ∩ B) = P(A|B) * P(B)

Here, P(A|B) represents the probability of event A occurring given that event B has occurred.

Example: Consider a bag containing 5 blue and 3 red balls. We draw two balls from the bag without replacement. What is the probability of drawing a blue ball first, followed by a red ball?

To apply the multiplication rule, we first calculate the probability of each event:

P(Blue1) = 5/8 P(Red2|Blue1) = 3/7

Now, we can compute the probability of both events happening together:

P(Blue1 ∩ Red2) = P(Blue1) * P(Red2|Blue1) = (5/8) * (3/7) ≈ 0.2679

In the next part of this series, we will cover topics related to Descriptive Statistics.

For more practical tips and insights on AI, data science, and statistics, explore our blog at Analytica Data Science Solutions. Discover engaging content to expand your knowledge and stay up-to-date with the latest developments.

In this blog post, we have covered the basics of probability and statistics. If you wish to further expand your knowledge and understanding, here are some references to help you dive deeper into these topics:

Books:

1. DeGroot, M. H., & Schervish, M. J. (2012). Probability and Statistics (4th ed.). Pearson.
2. Wackerly, D., Mendenhall, W., & Scheaffer, R. L. (2007). Mathematical Statistics with Applications (7th ed.). Cengage Learning.
3. Casella, G., & Berger, R. L. (2001). Statistical Inference (2nd ed.). Duxbury Press.
4. Hastie, T., Tibshirani, R., & Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd ed.). Springer.
5. Wickham, H., & Grolemund, G. (2016). R for Data Science: Import, Tidy, Transform, Visualize, and Model Data. O’Reilly Media.

Websites:

1. Khan Academy — Probability and Statistics: https://www.khanacademy.org/math/statistics-probability
2. Stat Trek — Teach yourself statistics: https://stattrek.com/
3. Carnegie Mellon University Probability & Statistics: https://oli.cmu.edu/courses/probability-statistics-open-free/

Online Courses:

1. Coursera — Statistics with R Specialization by Duke University: https://www.coursera.org/specializations/statistics
2. Coursera — Introduction to Probability and Data with R by Duke University: https://www.coursera.org/learn/probability-intro
3. edX — Probability and Statistics in Data Science using Python by the University of California, San Diego: https://www.edx.org/course/probability-and-statistics-in-data-science-using-python
4. DataCamp — Introduction to Probability in R: https://www.datacamp.com/courses/introduction-to-probability-in-r
5. DataCamp — Foundations of Probability in R: https://www.datacamp.com/courses/foundations-of-probability-in-r

Read More blogs in AnalyticaDSS Blogs here : BLOGS

Read More blogs in Medium : Medium Blogs

The post Introduction to Probability and Statistics: Basic Concepts and Terminology with Visuals — Part I appeared first on Analytica Data Science Solutions.

R is a popular programming language for data analysis and statistical computing. It is widely used in a variety of fields, including finance, healthcare, and research, and is known for its powerful tools for data manipulation, visualization, and statistical analysis.

Despite its many strengths, R can sometimes be slow, especially when performing computationally intensive tasks or working with large datasets. This can be a problem for users who need to analyze data quickly or who are working on time-sensitive projects. To address this issue, there is a need for tools and techniques that can help to speed up R code.

One way to speed up R code is to use Rcpp, a package for R that allows you to easily integrate C++ code into R. By using Rcpp, you can take advantage of the speed and efficiency of C++ to make your R code run faster. In this article, we will explore the benefits of using Rcpp and how it can help you to speed up your R code.

Why Rcpp is faster than R

One of the main reasons why Rcpp can be faster than R is that it allows you to write code in C++, which is a compiled language. This means that the code is transformed into machine code before it is executed, which can be much faster than interpreted languages like R.

In contrast, interpreted languages like R are executed directly by the interpreter, without the need for pre-compilation. While this can make them easier to use and more flexible, it can also make them slower, as the interpreter has to parse and execute the code on the fly.

Rcpp makes it easy to write C++ code that can be called from R. When you use Rcpp, your C++ code is compiled into a shared library, which can then be loaded and called from R. This allows you to take advantage of the speed and efficiency of C++ while still using R for your overall workflow.

There are many types of tasks that can benefit from using Rcpp. In general, tasks that involve heavy computation or looping can be particularly well-suited for Rcpp, as these types of tasks can be very slow in R. Examples of tasks that might benefit from using Rcpp include machine learning algorithms, simulations, and data manipulation.

Getting started with Rcpp

To use Rcpp, you will first need to install it. You can do this by running the following command in your R session:

install.packages("Rcpp")

Once Rcpp is installed, you can load it into your R session using the library function:

library(Rcpp)

An Rcpp function is a C++ function that can be called from R. It has a specific structure that includes a list of input arguments and a return value. Here is an example of a simple Rcpp function that takes two integers as input and returns their sum:

#include 
using namespace Rcpp;

// [[Rcpp::export]]
int sum(int x, int y) {
  return x + y;
}

The #include  line includes the Rcpp header file, which provides access to various Rcpp functions and types. The using namespace Rcpp; line allows you to use Rcpp functions and types without having to prefix them with Rcpp::. The [[Rcpp::export]] attribute tells Rcpp to make the function available to R.

Here is an example of a more complete Rcpp function that demonstrates how to pass variables between R and C++ and return a result to R:

#include 
using namespace Rcpp;

// [[Rcpp::export]]
NumericVector matrixMultiply(NumericMatrix A, NumericVector x) {
  int nrow = A.nrow(), ncol = A.ncol();

  // Check that the dimensions of A and x are compatible
  if (ncol != x.size()) {
    stop("Incompatible dimensions: cannot multiply matrix and vector.");
  }

  // Create the result vector
  NumericVector y(nrow);

  // Perform the matrix-vector multiplication
  for (int i = 0; i < nrow; i++) {
    double sum = 0;
    for (int j = 0; j < ncol; j++) {
      sum += A(i, j) * x[j];
    }
    y[i] = sum;
  }

  return y;
}

This function takes in a matrix A and a vector x, and returns their matrix-vector product as a new vector y. It first checks that the dimensions of A and x are compatible, and then performs the matrix-vector multiplication by looping over the rows of A and summing the products of the corresponding entries. Finally, it returns the result vector y to R.

Tips for optimizing Rcpp code

One of the first steps in optimizing Rcpp code is to identify the bottlenecks in your code, i.e., the parts of the code that are taking the most time to execute. There are a number of tools available for profiling R code, such as the profvis package and the Rprof function. By using these tools, you can get a sense of which parts of your code are taking the most time, and focus your optimization efforts on those areas.

There are a number of ways to optimize Rcpp code, depending on the specific needs of your project. Here are a few tips:

• Avoid unnecessary copies: When passing data between R and C++, it is often more efficient to pass pointers to the data rather than making copies of the data. Rcpp provides special types and functions for this purpose, such as the NumericMatrix and NumericVector types and the as and wrap functions.
• Use Rcpp’s special types and functions: Rcpp provides a number of special types and functions that can make it easier to work with R data from C++. For example, the NumericMatrix and NumericVector types provide convenient ways to access and manipulate matrix and vector data, while the Rcout and Rcerr streams allow you to print to the R console from C++.
• Consider using parallelization: If your code can be parallelized, using RcppParallel can be a powerful way to speed up your code. RcppParallel provides a number of tools for writing concurrent C++ code that can be called from R

Real-world examples of using Rcpp

To give you a sense of the types of tasks that can benefit from using Rcpp, here are a few examples of real-world projects that have used Rcpp to speed up their code:

• Machine learning: Rcpp has been used to speed up various machine learning algorithms, such as gradient boosting and k-means clustering. For example, the xgboost package uses Rcpp to provide a fast implementation of the XGBoost algorithm.
• Simulations: Rcpp can be very useful for performing complex simulations, as it allows you to take advantage of C++’s speed and efficiency to run many simulations in a short amount of time. For example, the simstudy package uses Rcpp to perform simulations for statistical power calculations.
• Data manipulation: Rcpp can be used to perform complex data manipulation tasks, such as reshaping or aggregating data. For example, the data.table package uses Rcpp to provide fast and efficient..

Conclusion

In this article, we have explored the benefits of using Rcpp to speed up R code. We have seen that Rcpp allows you to write C++ code that is compiled and called from R, taking advantage of the speed and efficiency of C++ to make your R code run faster. We have also looked at some tips for optimizing Rcpp code and a few examples of real-world projects that have used Rcpp to speed up their code.

If you are working on a project that involves heavy computation or looping, or if you simply need to speed up your R code, you may want to consider using Rcpp. While Rcpp can be somewhat more complex to use than pure R code, it can be a powerful tool for improving the performance of your R code.

To learn more about Rcpp, you may want to check out the following resources:

• The Rcpp documentation: https://cran.r-project.org/web/packages/Rcpp/index.html
• The Rcpp gallery: https://rcpp.org/gallery/
• Hadley Wickham’s “Advanced R” book: https://adv-r.hadley.nz/

Read More blogs in AnalyticaDSS Blogs here : BLOGS

Read More blogs in Medium : Medium Blogs

The post R on Steroids with Rcpp Library! appeared first on Analytica Data Science Solutions.

There are still many unsolved problems in Mathematics, despite countless research trying to solve these problems. Our Math problem for today is about the Goldbach Conjecture. In a previous post, I talked about the Collatz Conjecture, which is one of my favorite unsolved problems in Mathematics.

The Goldbach conjecture is a famous open problem in mathematics that states that every even integer greater than 2 can be expressed as the sum of two prime numbers. This conjecture has been verified for very large numbers, but a complete proof or disproof has eluded mathematicians for over three centuries.

The conjecture was proposed by ?

The conjecture was first proposed by Christian Goldbach, a Prussian mathematician, in a letter to his colleague Leonhard Euler in 1742. Goldbach’s conjecture states that every even integer greater than 2 can be written as the sum of two prime numbers. For example, 4 can be written as the sum of 2 and 2, 6 can be written as the sum of 3 and 3, and 8 can be written as the sum of 5 and 3.

Despite its simplicity, the Goldbach conjecture has proven to be a difficult problem to solve. Over the years, many mathematicians have attempted to prove or disprove the conjecture, but to date, no one has been able to come up with a complete proof.

One of the reasons the Goldbach conjecture is so difficult to prove is that it involves the concept of prime numbers, which are numbers that are divisible only by themselves and 1. Prime numbers play a crucial role in mathematics and are often used to prove other mathematical results, but they are also notoriously difficult to work with.

Another reason:

Another reason the Goldbach conjecture is difficult to prove is that it involves the concept of infinity. The conjecture states that every even integer greater than 2 can be written as the sum of two prime numbers, which means that the conjecture applies to an infinite number of even integers. Proving an infinite number of statements can be challenging, as it requires a different approach than proving a finite number of statements.

Despite the difficulty of the problem, many mathematicians have attempted to prove the Goldbach conjecture over the years. In the 19th century, mathematician Bernhard Riemann made significant progress towards a proof by developing a new mathematical tool called the zeta function. This function allows mathematicians to study the distribution of prime numbers and has been used to make significant progress on many other open problems in mathematics.

Over the years

Over the years, mathematicians have used computers to verify the Goldbach conjecture for very large numbers, and the largest number for which the conjecture has been verified is currently around 4 x 10¹⁸, or 40 quintillion. This verification was performed by a team of mathematicians led by Michael O. Rabin in the 1980s.

However, it is important to note that verifying the conjecture for a very large number does not constitute a complete proof of the conjecture. In order to prove the conjecture, it would be necessary to find a general proof that applies to all even integers, not just a specific set of very large numbers.

Despite the challenges, many mathematicians continue to work on the Goldbach conjecture, as it remains one of the most famous open problems in mathematics. The conjecture has inspired much research and has led to the development of new mathematical techniques and tools, which have in turn been used to make progress on other open problems in mathematics.

In Conclusion,

the Goldbach conjecture is a famous open problem in mathematics that has eluded a complete proof or disproof for over three centuries. Despite the difficulty of the problem, many mathematicians continue to work on it, as it remains an important and fascinating area of study.

Read More blogs in AnalyticaDSS Blogs here : BLOGS

Read More blogs in Medium : Medium Blogs

Read More blogs in R-bloggers : https://www.r-bloggers.com

The post Unsolved Math Problems: The Goldbach Conjecture! appeared first on Analytica Data Science Solutions.

The power of R comes from the vast collection of software libraries, i.e. packages, that can be easily installed and loaded in R. Today we will cover two of the most powerful packages in R, the tidyverse and data.table packages.

The tidyverse and data.table are two popular packages in R that provide functions for working with data. They both have their own strengths and are suitable for different types of tasks.

The tidyverse is a collection of packages designed for data manipulation, visualization, and modeling. It is based on the principles of tidy data, which suggests that data should be structured in a way that makes it easy to work with. The tidyverse includes packages such as dplyr, tidyr, and ggplot2, which provides functions for data manipulation, cleaning, and visualization.

One of the main advantages of the tidyverse is its simplicity. The functions in the tidyverse are easy to learn and use, and they often require fewer lines of code compared to other packages. They also have a consistent syntax, which makes it easier to learn and use multiple functions.

Examples: Tidyverse Examples

Here are some examples of how to use the tidyverse:

To select specific columns from a dataset:

# Load the tidyverse package
library(tidyverse)

# Load the mpg dataset from the ggplot2 package
data(mpg)

# Select the "manufacturer" and "model" columns
mpg %>% select(manufacturer, model)

And to group and summarize a dataset:

# Load the tidyverse package
library(tidyverse)

# Load the mpg dataset from the ggplot2 package
data(mpg)

# Group the dataset by "class" and compute the mean of the "hwy" column
mpg %>% group_by(class) %>% summarize(mean_hwy = mean(hwy))

To join two datasets:

# Load the tidyverse package
library(tidyverse)

# Load the mpg and cylinders datasets from the ggplot2 package
data(mpg)
data(cylinders)

# Join the mpg and cylinders datasets on the "manufacturer" column
mpg %>% left_join(cylinders, by = "manufacturer")

To perform a linear regression using the lm function from the stats package:

# Load the tidyverse and stats packages
library(tidyverse)
library(stats)

# Load the mtcars dataset
data(mtcars)

# Perform a linear regression to predict mpg (miles per gallon) using wt (weight) as the predictor variable
fit <- mtcars %>% 
  lm(mpg ~ wt, data = .)

# Summarize the model results
summary(fit)

Create a scatterplot matrix using the scatterplotMatrix function from the car package:

# Load the tidyverse and car packages
library(tidyverse)
library(car)

# Load the iris dataset
data(iris)

# Create a scatterplot matrix of the iris dataset
scatterplotMatrix(iris, smooth = FALSE)

Create a faceted bar plot using ggplot2:

# Load the tidyverse package
library(tidyverse)

# Load the mpg dataset from the ggplot2 package
data(mpg)

# Create a faceted bar plot showing the distribution of hwy (highway miles per gallon) by class and drv (drive type)
ggplot(mpg, aes(x = hwy)) +
  geom_histogram(binwidth = 2) +
  facet_wrap(~ class + drv, nrow = 2)

Examples: data.table Examples

The data.table package, on the other hand, is a high-performance package for working with large datasets. It provides functions for manipulating and querying data efficiently. The data.table package is particularly useful when working with datasets that are too large to fit in memory or when you need to perform complex operations on large datasets.

One of the main advantages of the data.table package

One of the main advantages of the data.table package is its speed. The functions in the data.table package are generally faster than their counterparts in the tidyverse, especially when working with large datasets.

Here are some more examples of how to use the data.table package:

To select specific columns from a dataset:

# Load the data.table package
library(data.table)

# Load the mpg dataset from the ggplot2 package
data(mpg)

# Convert the dataset to a data.table
mpg <- as.data.table(mpg)

# Select the "manufacturer" and "model" columns
mpg[, .(manufacturer, model)]

and to group and summarize a dataset:

# Load the data.table package
library(data.table)

# Load the mpg dataset from the ggplot2 package
data(mpg)

# Convert the dataset to a data.table
mpg <- as.data.table(mpg)

# Group the dataset by "class" and compute the mean of the "hwy" column
mpg[, .(mean_hwy = mean(hwy)), by = class]

To join two datasets:

# Load the data.table package
library(data.table)

# Load the mpg and cylinders datasets from the ggplot2 package
data(mpg)
data(cylinders)

# Convert the datasets to data.tables
mpg <- as.data.table(mpg)
cylinders <- as.data.table(cylinders)

# Join the mpg and cylinders datasets on the "manufacturer" column
mpg[cylinders, on = "manufacturer"]

Perform a linear regression using the lm function from the stats package and the data.table package:

# Load the data.table and stats packages
library(data.table)
library(stats)

# Load the mtcars dataset
data(mtcars)

# Convert the dataset to a data.table
mtcars <- setDT(mtcars)

# Perform a linear regression to predict mpg (miles per gallon) using wt (weight) as the predictor variable
fit <- mtcars[, lm(mpg ~ wt)]

# Summarize the model results
summary(fit)

Create a scatterplot matrix using the scatterplotMatrix function from the car package and the data.table package:

# Load the data.table and car packages
library(data.table)
library(car)

# Load the iris dataset
data(iris)

# Convert the dataset to a data.table
iris <- as.data.table(iris)

# Create a scatterplot matrix of the iris dataset
scatterplotMatrix(iris, smooth = FALSE)

Create a faceted bar plot using ggplot2 and the data.table package:

# Load the data.table and ggplot2 packages
library(data.table)
library(ggplot2)

# Load the mpg dataset from the ggplot2 package
data(mpg)

# Convert the dataset to a data.table
mpg <- as.data.table(mpg)

# Create a faceted bar plot showing the distribution of hwy (highway miles per gallon) by class and drv (drive type)
ggplot(mpg, aes(x = hwy)) +
  geom_histogram(binwidth = 2) +
  facet_wrap(~ class + drv, nrow = 2)

In terms of implementation, both the tidyverse and data.table packages are written in R, but some of the functions in the data.table package are implemented in C for improved performance.

In summary

the tidyverse and data.table are two popular packages in R that provide functions for working with data. The tidyverse is a collection of packages designed for data manipulation, visualization, and modeling, and it is particularly suitable for tasks that require simplicity and ease of use. The tidyverse functions are easy to learn and use, and they often require fewer lines of code compared to other packages.

The data.table package is a high-performance package for working with large datasets, and it is particularly useful when working with large datasets or when you need to perform complex operations on large datasets. The functions in the data.table package are generally faster than their counterparts in the, especially when working with large datasets.

In general, it is a good idea to use the tidyverse for most tasks, unless you are working with very large datasets or need the extra performance provided by the data.table package.

At Analytica

and since we deal with larger datasets, GB to TB of data, our preferred tool for data wrangling in R is in fact data.table.

I hope this article helps the reader understand the differences between the tidyverse and data.table in R, and how to choose the right package for their tasks. Let me know if you have any questions.

Read More blogs in AnalyticaDSS Blogs here : BLOGS

Read More blogs in Medium : Medium Blogs

Read More blogs in R-bloggers : https://www.r-bloggers.com

The post The Tidyverse and data.table R Packages appeared first on Analytica Data Science Solutions.

Table of Content:

I. Introduction to loops and vectorization in programming languages

• Definition of loops
• Types of loops (for, while, repeat)
• Definition of vectorization
• Advantages of vectorization over loops

II. Loops in programming languages

• How loops work
• Examples of loop usage
• Common pitfalls of using loops

III. Vectorization in programming languages

• How vectorization works
• Examples of vectorized operations
• Advantages of vectorization (speed, efficiency)

IV. When to use loops vs. vectorization

• Situations where loops are necessary
• Situations where vectorization is preferred
• Trade-offs between loops and vectorization

V. Best practices for using loops and vectorization

• Tips for optimizing loop performance
• Tips for choosing between loops and vectorization

VI. Conclusion

• Summary of key points
• Importance of understanding loops and vectorization in programming languages

I. Introduction to loops and vectorization in programming languages

Loops and vectorization are two important concepts in programming languages that refer to different ways of performing the same task. They are used to manipulate data, perform calculations, and achieve the desired outcome. Understanding how to use loops and vectorization effectively can have a significant impact on the efficiency and performance of your code.

Definition of loops

A loop is a way to repeat a set of instructions multiple times. In programming languages, there are several types of loops, including for loops, while loops, and repeat loops.

A for loop is used to iterate over a sequence of objects, such as a list or an array. It has the following syntax:

for (variable in sequence) {
  statements
}

A while loop, on the other hand, continues to execute as long as a certain condition is true. It has the following syntax:

while (condition) {
  statements
}

Finally, a repeat loop is similar to a while loop, except that it always executes at least once before checking the condition. It has the following syntax:

repeat {
  statements
} while (condition)

Definition of vectorization

Vectorization is a way to perform operations on multiple elements of a vector simultaneously, rather than using a loop to iterate over each element individually. Vectorized operations are generally faster and more efficient than looping, because they take advantage of the underlying structure of vectors and the optimized routines in the programming language’s base package.

Here is an example of vectorization in the R statistical programming language:

# Create a vector of numbers
numbers <- c(1, 2, 3, 4, 5)

# Add 1 to each element of the vector using vectorization
numbers <- numbers + 1

# Print the resulting vector
print(numbers)

The output of this code will be:

2 3 4 5 6

In this example, the + 1 operation is applied to the entire vector numbers, rather than to each element individually. This is an example of vectorization because it takes advantage of the underlying structure of vectors and the optimized routines in R’s base package.

Advantages of vectorization over loops

Vectorization has several advantages over loops. First, vectorized operations are generally faster than looping, because they take advantage of optimized routines and the underlying structure of vectors. Second, vectorization is often easier to read and understand than looping, because it uses concise and expressive syntax. Finally, vectorization can improve the maintainability of your code, because it is easier to modify and debug than looping.

II. Loops in programming languages

Loops are a fundamental concept in programming languages and are used to repeat a set of instructions multiple times. In this section, we will explore how loops work, provide examples of their usage, and discuss the common pitfalls of using loops.

How loops work

Loops work by iterating over a sequence of objects and executing a set of instructions for each iteration. The number of iterations is determined by the length of the sequence or by a specified condition.

In most programming languages, loops are controlled by a looping construct, such as a for loop or a while loop. The looping construct specifies the sequence to be iterated over and the statements to be executed for each iteration.

Examples of loop usage

Here are some examples of loop usage in different programming languages:

R:

# Create a vector of numbers
numbers <- c(1, 2, 3, 4, 5)

# Use a for loop to iterate over the vector and print each number
for (i in numbers) {
  print(i)
}

Python:

# Create a list of numbers
numbers = [1, 2, 3, 4, 5]

# Use a for loop to iterate over the list and print each number
for i in numbers:
  print(i)

Java:

// Create an array of numbers
int[] numbers = {1, 2, 3, 4, 5};

// Use a for loop to iterate over the array and print each number
for (int i = 0; i < numbers.length; i++) {
  System.out.println(numbers[i]);
}

Common pitfalls of using loops

There are several common pitfalls to be aware of when using loops. One pitfall is the risk of infinite loops, which occur when the looping condition is always true or the loop counter is not updated properly. Infinite loops can cause your program to run indefinitely and can be difficult to debug.

Another pitfall is the risk of off-by-one errors, which occur when the loop counter is not properly initialized or the loop condition is not properly defined. Off-by-one errors can cause your loop to either iterate too few or too many times, resulting in incorrect output or unintended behavior.

Finally, loops can be slower and less efficient than vectorized operations, particularly for large datasets. This can be a problem if performance is critical for your application.

In general, it is important to carefully consider the performance and readability of your code when using loops and to choose the appropriate looping construct for your specific needs.

III. Vectorization in programming languages

Vectorization is a way to perform operations on multiple elements of a vector simultaneously, rather than using a loop to iterate over each element individually. Vectorized operations are generally faster and more efficient than looping because they take advantage of the underlying structure of vectors and the optimized routines in the programming language’s base package. In this section, we will explore how vectorization works, provide examples of vectorized operations, and discuss the advantages of vectorization.

How vectorization works

Vectorization works by applying an operation to an entire vector at once, rather than to each element individually. Most programming languages have built-in functions or operators that support vectorization, such as element-wise arithmetic operators and functions in R, NumPy, and Python, or the apply family of functions in R.

Vectorization is typically faster and more efficient than looping because it avoids the overhead of calling a looping construct and iterating over each element individually. It also often results in more readable and expressive code.

Examples of vectorized operations

Here are some examples of vectorized operations in different programming languages:

R:

# Create a vector of numbers
numbers <- c(1, 2, 3, 4, 5)

# Add 1 to each element of the vector using vectorization
numbers <- numbers + 1

# Print the resulting vector
print(numbers)

Python:

# Import the NumPy library
import numpy as np

# Create a NumPy array of numbers
numbers = np.array([1, 2, 3, 4, 5])

# Add 1 to each element of the array using vectorization
numbers = numbers + 1

# Print the resulting array
print(numbers)

Java:

// Import the Apache Commons Math library
import org.apache.commons.math3.util.FastMath;

// Create a Java array of numbers
double[] numbers = {1, 2, 3, 4, 5};

// Use the mapToDouble function from the Apache Commons Math library to apply a vectorized operation to the array
double[] squared = Arrays.stream(numbers).mapToDouble(x -> FastMath.pow(x, 2)).toArray();

// Print the resulting array
System.out.println(Arrays.toString(squared));

Advantages of vectorization

Vectorization has several advantages over looping. First, vectorized operations are generally faster than looping, because they take advantage of optimized routines and the underlying structure of vectors. Second, vectorization is often easier to read and understand than looping, because it uses concise and expressive syntax. Finally, vectorization can improve the maintainability of your code, because it is easier to modify and debug than looping.

However, it’s worth noting that vectorization is not always possible or appropriate. In some cases, you may need to use a loop to perform an operation that is not vectorizable, or to perform an operation that depends on the previous iteration. In these cases, looping may be necessary or more appropriate.

IV. When to use loops vs. vectorization

Loops and vectorization are two different approaches to performing the same task in programming. In general, vectorization is preferred because it is faster and more efficient than looping, but there are situations where loops may be necessary or more appropriate. In this section, we will explore when to use loops vs. vectorization.

Situations where loops are necessary

There are several situations where loops may be necessary or more appropriate than vectorization. One such situation is when you need to perform an operation that is not vectorizable, such as reading a file line by line or interacting with a user through the console. In these cases, a loop is the only way to achieve the desired behavior.

Another situation where loops may be necessary is when you need to perform an operation that depends on the previous iteration. For example, consider the following code, which uses a loop to calculate the factorial of a number:

factorial <- function(n) {
  result <- 1
  for (i in 1:n) {
    result <- result * i
  }
  return(result)
}

In this case, the factorial of a number is calculated by multiplying the current number by the result of the previous iteration. This operation cannot be vectorized because it depends on the previous iteration.

Situations where vectorization is preferred

In general, vectorization is preferred over looping because it is faster and more efficient. This is particularly true for large datasets, where the overhead of calling a looping construct and iterating over each element individually can significantly impact performance.

Vectorization is also often easier to read and understand than looping because it uses concise and expressive syntax. This can improve the maintainability of your code because it is easier to modify and debug than looping.

Trade-offs between loops and vectorization

There are trade-offs to consider when choosing between loops and vectorization. Loops may be slower and less efficient than vectorized operations, particularly for large datasets. However, loops can be more flexible and easier to modify than vectorized operations, particularly when the operation depends on the previous iteration.

In general, it is important to carefully consider the performance and readability of your code when choosing between loops and vectorization and to choose the appropriate approach for your specific needs.

V. Best practices for using loops and vectorization

Using loops and vectorization effectively can have a significant impact on the efficiency and performance of your code. In this section, we will discuss some best practices for using loops and vectorization in programming languages.

Tips for optimizing loop performance

There are several ways to optimize the performance of loops:

1. Use the appropriate looping construct: Choose the looping construct that is most appropriate for your specific needs. For example, use a for loop to iterate over a sequence of objects, use a while loop to continue executing as long as a certain condition is true, or use a repeat loop to execute at least once before checking the condition.
2. Avoid unnecessary calculations: Only perform calculations that are necessary for the current iteration. Avoid performing unnecessary calculations or creating unnecessary variables, as this can slow down your loop.
3. Pre-allocate memory: If you are creating an object within the loop, such as a list or an array, pre-allocate memory for it before the loop starts. This can improve the performance of your loop by avoiding the overhead of repeatedly reallocating memory.
4. Use optimized functions: Use optimized functions and libraries, such as the apply family of functions in R or the NumPy library in Python, to perform common operations. These functions are generally faster and more efficient than looping.

Tips for choosing between loops and vectorization

When choosing between loops and vectorization, consider the following factors:

1. Performance: Vectorization is generally faster and more efficient than looping, particularly for large datasets. However, there are situations where loops may be faster, such as when the vector is very small.
2. Readability: Vectorization is often easier to read and understand than looping because it uses concise and expressive syntax. This can improve the maintainability of your code because it is easier to modify and debug than looping.
3. Flexibility: Loops can be more flexible than vectorized operations, particularly when the operation depends on the previous iteration. However, vectorization can be more flexible in some cases, because it allows you to perform operations on multiple elements simultaneously.

In general, it is important to carefully consider the performance and readability of your code when choosing between loops and vectorization and to choose the appropriate approach for your specific needs.

VI. Conclusion

In conclusion, loops and vectorization are two important concepts in programming languages that refer to different ways of performing the same task. Loops are used to iterate over a sequence of objects and execute a set of instructions for each iteration, while vectorization is a way to perform operations on multiple elements of a vector simultaneously.

Vectorization is generally preferred over looping because it is faster and more efficient, and because it often results in more readable and expressive code. However, there are situations where loops may be necessary or more appropriate, such as when the operation is not vectorizable or depends on the previous iteration.

It is important to understand when to use loops and when to use vectorization, and to choose the appropriate approach for your specific needs. Some best practices for using loops and vectorization include optimizing loop performance, choosing the appropriate looping construct, and using optimized functions and libraries.

Overall, understanding loops and vectorization is crucial for writing efficient and effective code in programming languages.

Read More blogs in AnalyticaDSS Blogs here : BLOGS

Read More blogs in Medium : Medium Blogs

Read More blogs in R-bloggers : https://www.r-bloggers.com

The post Maximizing Efficiency with Loops and Vectorization in Programming Languages appeared first on Analytica Data Science Solutions.

Analyzing crypto market

Aous Abdo, WWW.ANALYTICADSS.COM
An interactive version of this post can be found on here.

In my previous post I explored bitcoin data from different exchanges, we also covered some arbitrage-related data. In part 2 of this series I will explore alt coin related data.

R Libraries

Below is a list of R libraries we will be using to help us with our analysis. Not all of them are necessary but they all will make our life easier.

library(PoloniexR)
library(data.table)
library(lubridate)
library(Quandl)
library(plyr)
library(stringr)
library(ggplot2)
library(plotly)
library(janitor)
library(quantmod)
library(pryr)
library(corrplot)
library(PerformanceAnalytics)
library(tidyr)
library(MLmetrics)
library(tidyquant)
library(corrr)
library(cowplot)

Data

The best source I know off to get alt-coin data is through PoloniexR. I have written an R function to help download data.

get_alt_data <- function(tz = "UTC"
                         , coin = c("ETH", "LTC")
                         , add_bitcoin = TRUE
                         , return_in_USDT = TRUE
                         , from = "2017-01-01"
                         , to = "2018-04-09"
                         , period = "D"
                         , verbose = FALSE){
  
  # We will be using the public API
  poloniex.public <- PoloniexPublicAPI()
  
  # set the time zone to utc
  Sys.setenv(tz = tz)
  
  # convert from and to into time obj
  from  <- as.POSIXct(paste(from, tz, sep = ""))
  to    <- as.POSIXct(paste(to, tz, sep = ""))
  
  # lists to store data.tables and xts objects
  chart_list <- list()
  dt_list    <- list()
  
  # make sure the coin pair is in upper case
  coin       <- toupper(coin)
  coin_pairs <- paste0("BTC_", coin[coin != "BTC"])
  if(add_bitcoin | return_in_USDT) coin_pairs <- c("USDT_BTC", coin_pairs)
  
  # loop over the coins to get the data
  for(i in coin_pairs){
    if(verbose)
      invisible(cat('\tGetting data for ', i, ' pair\n'))
    
    # this is a list that will contain the chart data for each coin pair
    try(chart_list[[i]] <- ReturnChartData(theObject = poloniex.public
                                       , pair      = i
                                       , from      = from
                                       , to        = to
                                       , period    = period)
        , silent = TRUE)
    
    # list to contain data.tables 
    try(dt_list[[i]] <- as.data.table(chart_list[[i]]), silent = TRUE)
  }
  
  # convert to data.table and make sure to add a column containing the pairs
  coin_dt <- rbindlist(l = dt_list, use.names = TRUE, idcol = "pair")
  
  # return data in usdt prices
  if(return_in_USDT){
    # to get the price of the alt coin in usdt is not that simple but we'll do it
    # get a DT of the btc_usdt pair
    btc_usd <- coin_dt[pair == "USDT_BTC"]
    btc_usd <- btc_usd[, .(index, pair, weightedaverage)]
    setnames(btc_usd, c("Date", "USDT_BTC_pair", "USDT_BTC_price"))
    
    # get DT with only alt coins
    alt_coins <- copy(coin_dt)#[pair != "USDT_BTC"]
    
    # now we need to add an index to the alt_coins table, but first we have to rename the index column
    alt_coins[, Date := index]
    alt_coins[, index := 1:.N]
    setkey(alt_coins, index)
    
    # now merge the data tables
    coin_dt_usdt <- merge(x = alt_coins, y = btc_usd, by = "Date")
    
    # now calcualte the price in usdt
    coin_dt_usdt[, price_usdt := ifelse(pair == "USDT_BTC", USDT_BTC_price, weightedaverage * USDT_BTC_price)]
    
    # now get rid of the extra columns
    coin_dt_usdt[, c("USDT_BTC_price", "USDT_BTC_pair") := NULL]
    
    # we need to change some column names
    col_names_to_change <- c("pair", "high", "low", "open", "close", "volume", "quotevolume", "weightedaverage")
    col_names <- names(coin_dt_usdt)
    col_names[col_names %in% col_names_to_change] <- paste0(col_names_to_change, '_btc')
    
    setnames(coin_dt_usdt, col_names)
    
    # add a column for the usdt pair
    coin_dt_usdt[, pair_usdt := gsub("BTC_", "USDT_", pair_btc)]
    
    # adjust col order
    setcolorder(coin_dt_usdt, c(1:10, 12, 11))
    
    # set key again
    setkey(coin_dt_usdt, index)
    
    # now get rid of the index column since it is not needed anymore
    coin_dt_usdt[, index := NULL]
    
    # now put together the return list  
    return_list <- list(alt_chart_list = chart_list, alt_dt = coin_dt, alt_usdt_dt = coin_dt_usdt)
  }else{
    return_list <- list(alt_chart_list = chart_list, alt_dt = coin_dt)
  }
  
  return(return_list)
}

The function above can be used to download data for multiple coin at the same time. The function returns a data.table object with data for all coins in the function call. Even if the user doesn’t add bitcoin to the list of coins, the function adds bitcoin by default. This can be deactivated with the add_bitcoin argument. Here is an example

# get alt data for some coins
alt_data <- get_alt_data(return_in_USDT = T
                         , from = "2015-01-01"
                         , coin = c('ETH','XRP', 'BCH', 'LTC', 'NEO', 'XMR', 'DASH', 'XEM'))[['alt_usdt_dt']]

Let’s look at the data we just downloaded

head(alt_data)

The table shows the date, OHLC, Volume, and weightedaverage price in BTC. It also shows the pair and we added the price in USD.

Bitcoin-Altcoins Correlations

Wheneven I look at the prices of the coins available on my coinbase app I always get struck by the similarity of the price trends between the four coins available on coinbase: BTC, ETH, BCH, and LTC, see Figure below. So I thought it will be a good idea to explore the correlation in price trends between altcoins and bitcoin.

Let’s look at price trends of the coins we just downloaded. To better see potential correlations I am going to only zoom in on 2018.

p <- ggplot(alt_data[year(Date) == 2018], aes(x = Date, y =  price_usdt, col = pair_usdt)) + geom_line()
p <- p + facet_wrap(~pair_usdt, scales = "free", ncol = 3) + theme_minimal() + theme(legend.position="none") + ylab("Price (USD)")
p

The figure above shows that some coins seems to be more correlated with Bitcoin than others. The figure also shows that this variablity between Bitcoin and another coin varies over time. More on this below.

Tyring to find correlations bewteen time series data using Pearson correlation coefficient or other metrics used with stationary data, time series is not a form of stationary data, can give misleading results. Similar trends in time series data can also be very misleading, a nice article on this topic can be found here. And always remember that Correlation doesn’t guarantee Causation

Bottom line is the following, one has to be careful when cross-correlating time serice. In order to perform proper correlation analysis we need to add some new variables to our table.

Percentage Daily Change

Percentage daily change calculates the price change of a coin over a period of a day. Let’s add that to the table. Notice that we are calcualting this variable using the USD price, and not the price in Bitcoin.

# add daily price change
alt_data[, pct_change := Delt(price_usdt), by = pair_usdt]

Normalized Price in USD

Since the prices vary a lot, both overtime for the same coin and between coins, we will add a variable of the normalized price in USD. This variable will make it easy to plot prices of coins on the same figure.

# add normalized prices in udst
alt_data[, price_usdt_norm := price_usdt/max(price_usdt), by = pair_usdt]

Now that we have the normalized prices in USD, let’s look at the prices of bitcoin and litcoin on the same figure. We’ll do that for 2018 so we can better see any possible correlations.

p <- ggplot(alt_data[year(Date) == 2018 & pair_usdt %like% "BTC|LTC"], aes(x = Date, y =  price_usdt_norm, col = pair_usdt)) + geom_line()
p <- p + theme_minimal() + ylab("Price (USD)")
ggplotly(p)

The trends in the prices of BTC and LTC are very similar, Let’s look at price trends for 2017

p <- ggplot(alt_data[year(Date) == 2017 & pair_usdt %like% "BTC|LTC"], aes(x = Date, y =  price_usdt_norm, col = pair_usdt)) + geom_line()
p <- p + theme_minimal() + ylab("Price (USD)")
ggplotly(p)

Seems like we need to zoon in on the last quarter of 2017, let’s do that

p <- ggplot(alt_data[Date >= "2017-10-01"  & Date < "2018-01-01" &  pair_usdt %like% "BTC|LTC"], aes(x = Date, y =  price_usdt_norm, col = pair_usdt)) + geom_line()
p <- p + theme_minimal() + ylab("Price (USD)")
ggplotly(p)

It is clear from the above figure that the correlation in the prices of bitcoin and LTC vary over time. Note how the highest price for bitcoin on December 17 2017, preceded that of LTC by two days, which occurred on December 19 2017. This wasn’t the case for the ATH which occurred on January 6th 2018 for both coins.

Static Correlations

(and why you shouldn’t use them with crypto!)

Up until now I haven’t calculated any correltaions between the price of different coins. You might ask why should we even care about correlations in time series. Well, in the case of financial time series data, if one can show that a correlation exists between two time series then one can use this correlation to model/predict the price movement of one coin/stock given the price trends of another coin/stock.

However, as we mentioned earlier, correlation for time series data is not static, it changes over time. Actually let’s show that. To do that I am going to be calculating the Pearson correlation coefficient. In simple words, Pearson correlation coefficient for two vectors of data is a measure that shows how correlated these two vectors of data are. The value of this coefficient varies from -1, perfectly anti-correlated, to 1, perfectly correlated. So the correlation coefficient for a series of numbers on itself is 1. A value of zero means these is no correlation. Remember, this only works for static data.

In order to perform correlation on our data I am going to need to do some data transformation:

# subset data, only keep the date, the pair, and the price
alt_data_sub <- alt_data[, .(Date, pair_usdt, price_usdt)]
# convert to wide format 
alt_data_sub <- spread(data = alt_data_sub, key = "pair_usdt", value = "price_usdt")
# clean column names
setnames(alt_data_sub, gsub("USDT_", "", colnames(alt_data_sub)))

The new table we created contains the date along with the prices in USDT for each coin we have in our table.

tail(alt_data_sub)

Again, what I am doing here is not correct, I am just trying to show you why we shouldn’t be doing static correlations on crypto data. Now we’ll calculate the Pearson correlation coefficient between the coins we have, then we are going to make a nice plot of these coefficients.

# calculate the correlation matrix
M <- cor(alt_data_sub[, -1], use = "complete.obs") # notice how we are ignoring missing data with the last argument
# plot the correlation matrix
corrplot.mixed(corr = M, upper = "ellipse", lower = "number", order = "AOE", tl.col = "black")

The figure above shows the correlation coefficients between the different coins. It is easy to read, visually, the darker the color of the ellipse, and the more diagonal the ellipse, the higher the correlation coefficient. Of course you can also just look at the numbers on the bottom left part of the figure to get the value of the coefficient between two coins :). The figure shows how highly correlated the prices of crypto currencies can be. For example XRP and XEM have a correlation coefficient of 0.93. The highest correlation seems to be between BCH and DASH at 0.97 correlation coefficient.

All of the correlation coefficient we see in the above figure are significant, the question is, do these correlations vary over time. To answer this question I will calculate the correlation coefficient between Bitcoin and DASH on a monthly basis, you can do that for any time period, and will show that this coefficient varies greatly over time. Let’s do that

# subset the data
btc_dash <- alt_data_sub[, .(Date, BTC, DASH)]
# add a year_month column
btc_dash[, year_month := as.yearmon(Date)]
# calculate the correlation coefficient on montly basis
btc_dash_2 <- btc_dash[, cor(BTC, DASH), by = year_month]
# now plot the correlation coefficient as a function of month and year
plot(btc_dash_2$year_month, btc_dash_2$V1, xlab = "Year-Month", main = "Correlation Coeff. Between BTC and DASH Over time"
     , ylab = "Correlation Coefficient", type = "b", pch = 19, col = ifelse(btc_dash_2$V1 > 0, "blue", "red")
     , ylim = c(-1, 1))

This is interesting, the value of the monthly correlation coefficient between bitcoin and DASH varies between -0.91, highly anti-correlated, to 0.98, highly correlated. And this is why you should never use static correlation metrics with crypto data!

A good blog post on this same topic is written by Tom Fawcett from Silicon Valley Data Science and can be found here. In his post Tom shows, with simple simulations, why static correlations should never be used with time series.

Correlation Networks

There is one more plot I would like to make, which is a network plot of the correlations between the different coins. The correlation network plot helps show strengths of correlation between the different coins. Agian, these correlations are time dependent and the figure we will be making will change over time, but I still think it is a good figure to make. Here it is:

# we will be using the great corrr package for this work
# get the correlation matrix, just like we did before
# build the correlation matrix 
# the code snippets below are taken from, that is a great blog BTW 
# http://www.business-science.io/timeseries-analysis/2017/07/30/tidy-timeseries-analysis-pt-3.html
corr_2 <- correlate(alt_data_sub[, -1])
# make the network plot
# Network plot
corr_net <- corr_2 %>%
  network_plot(colours = c(palette_light()[[2]], "white", palette_light()[[4]]), legend = TRUE) +
  labs(
    title = "Static Correlations of some Crypto Currencies",
    subtitle = "2014 through 2018"
  ) +
  theme_tq() +
  theme(legend.position = "bottom")
corr_net

The figure above shows a network which measures how strongly correlated the prices of the coins under stugy are. The darker the color of the edge, line, connecting two coins and the closer they are in the network the stronger the correlation between these two coins.

From the figure, it seems like XMR, LTC, and BTC are in the heart of this network, while BCH seems to be the least correlated with the rest of the coins. Let’s see how the network plot changes between 2017 and 2018:

# subset the data and get correlation matrices
corr_2017 <- correlate(alt_data_sub[year(Date) == 2017][, -1])
corr_2018 <- correlate(alt_data_sub[year(Date) == 2018][, -1])
# build Network plots
corr_net_2017 <- corr_2017 %>%
  network_plot(colours = c(palette_light()[[2]], "white", palette_light()[[4]]), legend = TRUE) +
  labs(
    title = "Static Correlations of some Crypto Currencies",
    subtitle = "2017"
  ) +
  theme_tq() +
  theme(legend.position = "bottom")
corr_net_2018 <- corr_2018 %>%
  network_plot(colours = c(palette_light()[[2]], "white", palette_light()[[4]]), legend = TRUE) +
  labs(
    title = "Static Correlations of some Crypto Currencies",
    subtitle = "2018"
  ) +
  theme_tq() +
  theme(legend.position = "bottom")
# combine network plots
cow_net_plots <-plot_grid(corr_net_2017, corr_net_2018, ncol = 2)
title <- ggdraw() + 
    draw_label(label = 'Crypto Correlation Networks',
               fontface = 'bold', size = 18)
cow_out <- plot_grid(title, cow_net_plots, ncol=1, rel_heights=c(0.1, 1))
cow_out

As can be seen, the correlation networks do change overtime. This is not news since we already saw in the previous section that the value of the correlation varies overtime (I know we showed this to be true for the BTC-DASH air but we’ll show that this is true for the rest of the coins in the next section.)

Daily Returns Correlations

Let’s look at the percentage daily changes of the altcoins between 2015 and today.

# plot the percent changes
p <- ggplot(alt_data[Date > ymd("2015-01-01")], aes(x = Date, y =  (100*pct_change), col = pair_usdt)) + geom_line()
p <- p + ggtitle("% Daily Returns over time") + ylab("Daily Return (%)") 
p <- p + theme_bw() + guides(col=guide_legend(title="Coin Pair"))
ggplotly(p)

Although the above figure is very cluttered, one thing is certain, percentage daily returns vary greatly for crypto. Let’s try to make this figure a bit easier to read

p <- ggplot(alt_data[Date > ymd("2015-01-01")], aes(x = Date, y =  (100*pct_change), col = pair_usdt)) + geom_line() + facet_wrap(~ pair_usdt)
p <- p + ggtitle("Percentage Daily Returns over time") + ylab("Daily Return (%)") 
p <- p + theme_bw() + theme(legend.position="none") 
ggplotly(p)

It is kind of surprising that Bitcoin has the least variability in daily returns. The nice big spike around April 2nd 2017 shows a percentage daily return of ~88% for XRP, this is the highest daily return I have seen!

Let’s look at the percentage daily returns for Bitcoin and Litecoin since they seem to be highly correlated. I am going to zoom in on the time period 2016–02–01 and 2016–05–01.

start_date <- ymd("2016-02-01")
end_date <- ymd("2016-05-01")
p <- ggplot(alt_data[pair_usdt %like% "BTC|LTC" & Date > start_date & Date < end_date], aes(x = Date, y =  (100*pct_change), col = pair_usdt)) + geom_line() + theme_bw() + ylab("Price (USD)")
p

The figure shows clear correlation between the daily returns of Bitcoin and litcoin. It also shows that these correlations can vary overtime. In fact, let’s look at how these correlations vary overtime.

# these steps are similar to the ones in the previous section, the only differnect is that now we are looking at the percentage change in price difference on daily basis instead of the actual price
# subset data, only keep the date, the pair, and the price
alt_data_sub_pct <- alt_data[, .(Date, pair_usdt, pct_change)]
# convert to wide format 
alt_data_sub_pct <- spread(data = alt_data_sub_pct, key = "pair_usdt", value = "pct_change")
# clean column names
setnames(alt_data_sub_pct, gsub("USDT_", "", colnames(alt_data_sub)))
# subset the data
btc_ltc <- alt_data_sub_pct[, .(Date, BTC, LTC)]
# add a year_month column
btc_ltc[, year_month := as.yearmon(Date)]
# calculate the correlation coefficient on montly basis
btc_ltc_2 <- btc_ltc[, cor(BTC, LTC), by = year_month]
# now plot the correlation coefficient as a function of month and year
plot(btc_ltc_2$year_month, btc_ltc_2$V1, xlab = "Year-Month", main = "Correlation Coeff. Between Daily Returns of BTC and LTC Over time"
     , ylab = "Correlation Coefficient", type = "b", pch = 19, col = ifelse(btc_ltc_2$V1 > 0, "blue", "red")
     , ylim = c(-1, 1))

Interesting, the correlation between the percentage daily change of the prices for bitcoin and litecoin is much more on the positive side, we only have one month in which this correlation is negtive, barely negative. This is a lot different than what we saw between Bitcoin and DASH, but that was for the actual prices and not the daily returns. Let’s redo this plot but his time for the actual prices for bitcoin and litecoin, just like we did with DASH.

# subset the data
btc_ltc_price <- alt_data_sub[, .(Date, BTC, LTC)]
# add a year_month column
btc_ltc_price[, year_month := as.yearmon(Date)]
# calculate the correlation coefficient on montly basis
btc_ltc_price_2 <- btc_ltc_price[, cor(BTC, LTC), by = year_month]
# now plot the correlation coefficient as a function of month and year
plot(btc_ltc_price_2$year_month, btc_ltc_price_2$V1, xlab = "Year-Month", main = "Correlation Coeff. Between BTC and Litecoin Over time"
     , ylab = "Correlation Coefficient", type = "b", pch = 19, col = ifelse(btc_ltc_price_2$V1 > 0, "blue", "red")
     , ylim = c(-1, 1))

Trends in the correlations of the daily return of bitcoin and litecoin on mothly basis, boy this is a mouth full, are very similar to those for the prices as we saw in the previous figure.

In the next post we’ll do something more statistically sound, rolling correlations.

Read More blogs in AnalyticaDSS Blogs here : BLOGS

Read More blogs in Medium : Medium Blogs

Read More blogs in R-bloggers : https://www.r-bloggers.com

The post Analyzing Crypto Market using R — Part 2 appeared first on Analytica Data Science Solutions.

Analyzing crypto market

Aous Abdo, WWW.ANALYTICADSS.COM
An interactive version of this post can be found here.

No doubt that crypto currencies with all the promises they bring, both financially and otherwise, are only here to stay. As a data scientist interested in data and numbers, I thought it would be nice to take a look at some crypto currencies with my favorite tool, R.

R Libraries

Below is a list of R libraries we will be using to help us with our analysis. Not all of them are necessary but they all will make our life easier.

library(PoloniexR)
library(data.table)
library(lubridate)
library(Quandl)
library(plyr)
library(stringr)
library(ggplot2)
library(plotly)
library(janitor)
library(quantmod)
library(pryr)
library(corrplot)
library(PerformanceAnalytics)
library(tidyr)
library(MLmetrics)
library(readr)

Getting the Data

1. PoloniexR Package

The easiest way to get current and historical data for cyrpto currencies is by using the PoloniexR developed by Vermeir Jellen. Vermeir Jellen gives a good tutorial on how to start with his package here. The Poloniex exchange includes many coins but not all. For missiong coins on Poloniex, one can scrape the coinmarketcap page, an example is given here.

2. Quandl

Quandl is my go to place for any financial data. Their free-tier API has lots of good data one can use. Quandl offers data from multiple exchanges. Locating crypto data on Quandl is not straight forward. After spending few hours on their site I found out that most of the crypto data can be found here

First, let’s take a look at different exchange data for Bitcoin using Quandl. We will download and plot historical bitcoin data from the following exchanges Kraken, Coinbase, Bitstamp, and ITBIT

# enable your Quandl API key
my_quandl_api_key <- read_file("../../quandl_api_key.txt")
Quandl.api_key(my_quandl_api_key)
# function to download quandl data
get_quandl_data <- function(data_source = "BITFINEX"
                            , pair = 'btcusd'
                            , ...){
  
  # make sure the user supplied the correct data_source
  if(toupper(data_source) != "BITFINEX") stop("data source supplied is wrong...")
  # quandl is case sensitive, all codes have to be upper case
  pair <- toupper(pair)
  tmp <- NA
  try(tmp <- Quandl(code = toupper(paste(data_source, pair, sep = "/")), ...), silent = TRUE)
  return(tmp)
}
# get btc data from different exchanges
  exchange_data <- list()
  
  exchanges <- c('KRAKENUSD','COINBASEUSD','BITSTAMPUSD','ITBITUSD')
  
  for (i in exchanges){
    exchange_data[[i]] <- Quandl(paste0('BCHARTS/', i))
  }

So We need to convert this list of BTC prices from different exchanges into a dataframe and put them all in one data frame so we can plot them.

# put them all in one dataframe to plot in ggplot2
btc_usd <- do.call("rbind", exchange_data)
btc_usd$exchange <- row.names(btc_usd)
btc_usd <- as.data.table(btc_usd)

We also need to do some minor cleaning, so let’s do that. We also need to get rid of rows of data with 0 weighted price.

# some data cleaning
btc_usd[, exchange := as.factor(str_extract(exchange, "[A-Z]+"))]
btc_usd <- clean_names(btc_usd)
btc_usd <- btc_usd[weighted_price > 0]
# set datatable key to be the date column
setkey(btc_usd, date)

Let’s take a look at the data table we just made.

head(btc_usd)

The data includes 10 columns, the date, OCHL prices, volumes in USD and BTC, the weighted price, and the exchange. I wish I had bought me some bitcoine back in 2011!!!

Now we’ll look at the price of bitcoin and color code it by exchange.

ggplot(btc_usd, aes(x = date, y = weighted_price, col = exchange)) + geom_line() + theme_bw()

Arbitrage

It appears the prices of btc on different exchanges are fairly consisant. But this is an artifact in the figure since we are covering several orders of magnitudes during the timeline we selected. To better see any price differenes we need to zoon in on the figure. Let’s zoom in on, say the first month of 2018, were we had the ATH for all coins. This will enable us to better see any differences in prices.

ggplot(btc_usd[date >= ymd("2018-01-01") & date <= ymd("2018-01-31")], aes(x = date, y = weighted_price, col = exchange)) + geom_line() + theme_bw()

There are obvious differences in prices between the exchanges. Differences seem to vary over time as well. Actually it will be interesting to look at the maxiumum price differences as a function of time, let’s do that

# first let's find the minimum price by date
btc_usd[, min_price := min(weighted_price), by = date]
# now we need to find the price difference between the price for each day and the minimum price for that day
# but since the price of bitcoin varies a lot for the time period under study, we need to normalize the price difference
# to do that we will just divide by the median price for each day
btc_usd[, price_diff := 100*(weighted_price - min_price)/median(weighted_price), by = (date)]

Now we have a new column which gives us the percentage of price differences for each day normalized to the median price for each day. Let’s take a look at the new table.

tail(btc_usd)

The reason I looked at the newer dates is that prior to 2014 we only have data for one exchange, so all the price differences were 0. Let’s take a look at the price differences as a function of time.

# first we need to create a new data table with only the maxiumum prices per day
tmp <- btc_usd[, price_diff := max(price_diff), by = date]
# This will help us visualize overlapping points
MyGray <- rgb(t(col2rgb("black")), alpha=50, maxColorValue=255)
tmp[, plot(date, price_diff, pch=20, col = MyGray, xlab = "Date", ylab = "Maximum of Price Difference (%)")]

Let’s show the plot with log scale on y axis. Let’s also discard dates with zero price differences.

tmp[price_diff > 0, plot(date, price_diff, pch=20, col = MyGray, log = "y" , xlab = "Date", ylab = "Maximum of Price Difference (%)")]

As one can see from the figure above, the bulk of the maximum difference in bitcoin prices between the different exchanges is in the 0.5–2.0% range. It is also interesting to see that the differences in prices seem to have come down between 2014 and 2016, but they seem to go up starting in 2017. Let’s fit a gam model to see what we get.

# we'll use ggplot and fit a gam smooth line
ggplot(tmp[price_diff > 0 ], aes(x = date, y = price_diff)) + geom_point(alpha = 0.2, shape = 16, size = 3, show.legend = FALSE) + scale_y_continuous(trans='log10') + geom_smooth(method = "gam", formula = y ~ s(x, bs = "cs")) + theme_bw() + xlab("Date") + ylab("Maximum of Price Difference (%)")

The regression line shows a hint of an over all downtrend from 2014 to mid 2016, except for an uptrend for few months in late 2015. The trend seems to have gone up in mid to late 2017, and again we see a downword movement in price differences starting in December of 2017. This can be seen better in the box-plot figure below.

tmp[, month_year := format(as.Date(date), "%Y-%m")]
ggplot(tmp[price_diff > 0], aes(x = month_year, y = price_diff)) + geom_boxplot() + scale_y_continuous(trans='log10') + xlab("Date (Year-Month)") + ylab("Maximum of Price Difference (%)") + theme_bw() + theme(axis.text.x = element_text(angle = 90, hjust = 1))

The box-plot figure above shows the variation of maximum differences in prices as a function of time. On the x-axis I grouped dates by month since anything less than a one-month period will result in congested figure.

Okay, now let’s find out which of the exchanges contribute the most to these price difference. That is, we are trying to determine which exchanges are constantly selling bitcoin higher, or lower, than the rest of the exchanges. We need to pull some numbers as below, and then we’ll make a bar plot to show the leading exchanges in each category.

# add two columns to our data table which will contain the minimum and maximum prices
tmp[, `:=`(day_max = max(weighted_price), day_min = min(weighted_price)), by = date]
# now put only the columns we care about in a new data.table
tmp2 <- tmp[price_diff > 0 , .(date, exchange, weighted_price, day_min, day_max)]
# notice how we excluded days with no price difference
# now we only want to keep the rows with the maximum and minimum daily prices
tmp2 <- tmp2[weighted_price == day_min | weighted_price == day_max]
# now we'll add a new column designating the price as being the minimum or maximum
tmp2[, max_min := ifelse(weighted_price == day_min, "min", "max")]
# clean the name of the exchange
tmp2[, exchange := gsub("USD", "", exchange)]
# now we'll add a new column containing the exchange name and the min_max column
tmp2[, max_min_exchange := paste(max_min, exchange, sep = "-")]

In the above chunk of code we created a new table which contains the maximum and minimum prices for each day. The table also contains a categorical column showing to which exchange this max/min price belong, and if the price was a maxima or a minima. Before we plot the table above, let’s have a quick look at it.

head(tmp2)

The max_min_exchange column contains all the data we need, so let’s make a barplot of this variable, we’ll color the barplot by the max_min criteria shown in max_min column

# now make a barplot 
ggplot(tmp2, aes(x = max_min_exchange, fill = max_min)) + geom_bar() + theme_bw() + theme(axis.text.x = element_text(angle = 90, hjust = 1)) + ggtitle("Exchanges with the highest and lowest price differences in Bitcoin") + ylab("Frequency") + xlab("Exchange") + scale_fill_discrete(name = "BTC Price Diff. Type")

This is interesting, Kraken seems to be the exchange with the most frequent highest prices for bitcoin. On the other hand, Bitstamp seems to be the one with the most frequent lowest prices among exchanges. So if you want to do arbitrage your best bit is to buy on Bitstamp and sell on Kraken.

Read More blogs in AnalyticaDSS Blogs here : BLOGS

Read More blogs in Medium : Medium Blogs

Read More blogs in R-bloggers : https://www.r-bloggers.com

The post Analyzing Crypto Markets using R — Part 1 appeared first on Analytica Data Science Solutions.

The Game of Life, also known as the “Conway’s Game of Life,” is a cellular automaton invented by mathematician John Horton Conway in 1970. It is a zero-player game, meaning that once the game is set up, it runs on its own, and there is no further input from a player.

The game is played on a grid of cells, each of which can be in one of two states: “alive” or “dead.” The state of each cell in the grid is determined by the state of the cells surrounding it, according to a set of rules. The game proceeds in a series of “generations,” with the state of each cell in the next generation being determined by the state of the cells in the current generation.

The game has been used to study a wide range of topics in mathematics and computer science, including patterns, self-organization, and complexity. It has also been used as a tool for exploring artificial life and artificial intelligence.

Game Rules

The game rules are as follows:

1. Any live cell with fewer than two live neighbors dies, as if by underpopulation.
2. Any live cell with two or three live neighbors lives on to the next generation.
3. Any live cell with more than three live neighbors dies, as if by overpopulation.
4. Any dead cell with exactly three live neighbors becomes a live cell, as if by reproduction.

These rules are applied to each cell in the grid simultaneously, with the state of the cells in the next generation being determined based on the state of the cells in the current generation. The game continues in this way, with the state of the cells being updated in each generation.

The Game of Life can be implemented in many different ways, with the rules being applied to a two-dimensional grid, or to a one-dimensional “tape,” or even to a three-dimensional space. There are also many variations of the rules that have been explored, with different sets of rules leading to different patterns and behaviors in the game.

The Game of Life is capable of producing a wide range of patterns, depending on the initial configuration of the cells and the specific rules that are being used. Some patterns remain stable over time, while others may evolve and change over the course of the game.

Game Patterns

Here are a few examples of patterns that can appear in the Game of Life:

• Still lifes: These are patterns that remain stable over time and do not change from one generation to the next. Examples of still lifes include blocks, beehives, and loafs.
• Oscillators: These are patterns that repeat themselves over time, cycling through a fixed set of states. Examples of oscillators include blinkers, toads, and pulsars.
• Spaceships: These are patterns that move across the grid, leaving behind a “trail” of cells as they go. Spaceships can move in any of the four cardinal directions, and their movement may be periodic or aperiodic.
• Gliders: These are a type of spaceship that move diagonally across the grid, leaving behind a distinctive “trail” of cells. Gliders are one of the simplest and most well-known spaceships in the Game of Life.
• Guns: These are patterns that produce an endless stream of spaceships or other patterns. The first gun was discovered in 1970 by Bill Gosper, and many more have been found since.
• Patterns with complex behavior: Some patterns in the Game of Life exhibit behavior that is difficult to predict or understand. These patterns may exhibit seemingly random behavior, or they may exhibit complex, self-organizing behavior.

These are just a few examples of the patterns that can appear in the Game of Life. The game is capable of producing a wide range of patterns and behaviors, and new patterns are still being discovered today.

Real-Life Examples

The Game of Life has inspired a number of real-world applications and has been used to model a wide range of systems and phenomena in fields such as biology, physics, and computer science. Here are a few examples of how the Game of Life has been used in real life:

• Biology: The Game of Life has been used to model the behavior of cellular automata, including the growth and division of cells. It has also been used to study patterns in the distribution of species in ecosystems and the spread of epidemics.
• Physics: The Game of Life has been used to model the behavior of physical systems, such as the formation of patterns in crystals and the behavior of fluids.
• Computer science: The Game of Life has been used as a test bed for exploring algorithms and data structures, and it has inspired the development of a number of computational models and techniques.
• Art and design: The Game of Life has inspired a number of artistic and design projects, including digital art, installations, and interactive exhibits.
• Education: The Game of Life has been used as a tool for teaching concepts in mathematics, computer science, and other fields, and it has been incorporated into educational software and curricula.

These are just a few examples of how the Game of Life has been used in real life. The game’s simplicity and versatility have made it a popular tool for studying a wide range of systems and phenomena.

Game Simulation in R

The script below simulates the Game of Life and produces a plot of the evolution of the cells over time. The plotly package is used to create an interactive heatmap, with “dead” cells being shown in darker shades and “alive” cells being shown in lighter shades. The x-axis of the plot represents the generation, and the y-axis represents the cell.

# Install the plotly and furrr packages if they are not already installed
# install.packages("plotly")
# install.packages("furrr")

library(plotly)
library(furrr)

# Set up the grid for the game
grid <- matrix(sample(c(0, 1), 100*100, replace = TRUE), nrow = 100)

# Initialize a list to store the state of the cells at each generation
grid_history <- rep(list(grid), generations)

# Simulate the game using the furrr package to parallelize the simulation
plan(multisession)
grid_history <- future_map(1:generations, function(i) {
  # Get the current state of the cells
  current_state <- grid_history[[i]]
  
  # Initialize the next state of the cells
  next_state <- matrix(0, nrow = nrow(current_state), ncol = ncol(current_state))
  
  # Iterate over each cell in the grid
  for (x in 1:nrow(current_state)) {
    for (y in 1:ncol(current_state)) {
      # Get the number of live neighbors for the current cell
      neighbors <- sum(current_state[max(x-1, 1):min(x+1, nrow(current_state)), 
                                     max(y-1, 1):min(y+1, ncol(current_state))]) - current_state[x, y]
      
      # Apply the rules of the Game of Life to determine the next state of the cell
      if (current_state[x, y] == 1) {
        if (neighbors < 2 || neighbors > 3) {
          next_state[x, y] <- 0
        } else {
          next_state[x, y] <- 1
        }
      } else {
        if (neighbors == 3) {
          next_state[x, y] <- 1
        } else {
          next_state[x, y] <- 0
        }
      }
    }
  }
  
  # Update the state of the cells
  grid <- next_state
  
  # Store the state of the cells at the current generation
  grid
})

# Plot the evolution of the cells over time using the plotly package
plot_ly(z = grid_history, colorscale = "Blackbody", type = "heatmapgl") %>%
  layout(xaxis = list(title = "Generation"), yaxis = list(title = "Cell"))

Game Simulation in Python

The script below simulates the Game of Life and produces a plot of the evolution of the cells over time. The matplotlib package is used to create a heatmap, with “dead” cells being shown in darker shades and “alive” cells being shown in lighter shades. The x-axis of the plot represents the generation, and the y-axis represents the cell.

import numpy as np
import matplotlib.pyplot as plt

# Set up the grid for the game
grid = np.random.choice([0, 1], size=(100, 100))

# Initialize a list to store the state of the cells at each generation
grid_history = [grid]

# Simulate the game
for i in range(generations):
    # Get the current state of the cells
    current_state = grid_history[i]

    # Initialize the next state of the cells
    next_state = np.zeros((100, 100))

    # Iterate over each cell in the grid
    for x in range(100):
        for y in range(100):
            # Get the number of live neighbors for the current cell
            neighbors = (current_state[max(x-1, 0):min(x+2, 100), max(y-1, 0):min(y+2, 100)]).sum() - current_state[x, y]

            # Apply the rules of the Game of Life to determine the next state of the cell
            if current_state[x, y] == 1:
                if neighbors < 2 or neighbors > 3:
                    next_state[x, y] = 0
                else:
                    next_state[x, y] = 1
            else:
                if neighbors == 3:
                    next_state[x, y] = 1
                else:
                    next_state[x, y] = 0

    # Update the state of the cells
    grid = next_state

    # Store the state of the cells at the current generation
    grid_history.append(grid)

# Plot the evolution of the cells over time using the matplotlib package
plt.imshow(grid_history, cmap="Greys")
plt.xlabel("Generation")
plt.ylabel("Cell")
plt.show()

In conclusion,

the Game of Life, also known as the cellular automaton, is a classic example of a simple mathematical model that can produce complex and fascinating patterns. In the game, a grid of cells is initialized with a certain number of “alive” cells, and the state of the cells at each generation is determined by a set of rules that depend on the number of live neighbors each cell has. Depending on the initial state of the cells and the rules applied, the game can produce a wide range of patterns, from stable configurations to chaotic patterns.

Read More blogs in AnalyticaDSS Blogs here : BLOGS

Read More blogs in Medium : Medium Blogs

Read More blogs in R-bloggers : https://www.r-bloggers.com

The post Conway’s Game of Life With Examples in R and Python appeared first on Analytica Data Science Solutions.

In a reinforcement learning system, the agent interacts with its environment by taking actions and observing the resulting rewards or punishments. The goal of the agent is to learn the best possible strategy for maximizing the rewards over time. This is done through trial and error, where the agent explores different actions and learns from their consequences.

One of the key features of reinforcement learning is that the agent can learn from experience, without being explicitly programmed with a set of rules or instructions. This allows the agent to adapt and improve its behavior based on the feedback it receives from the environment.

An example of reinforcement learning in action:

An example of reinforcement learning in action is a robot that is trained to navigate a maze. The robot is placed in the maze and must find its way to the goal. As it moves through the maze, it receives rewards for taking actions that bring it closer to the goal and punishments for taking actions that move it away from the goal. Over time, the robot learns the best strategy for navigating the maze and finds the quickest way to the goal.

Another example

Another example of reinforcement learning is a computer program that learns to play a game, such as chess or Go. The program is given the rules of the game and must learn to make the best possible moves based on the rewards and punishments it receives for each action. This requires the program to analyze the current state of the game and consider various possible moves, in order to choose the one that is most likely to lead to a win.

A robotic arm used in a manufacturing setting can be trained using reinforcement learning to perform tasks such as picking up and placing objects. The robotic arm receives rewards for successfully completing the tasks and punishments for making mistakes, and learns to optimize its movements over time.

A virtual personal assistant, such as Apple’s Siri or Amazon’s Alexa, can use reinforcement learning to improve its performance over time. The assistant receives rewards for providing accurate and helpful responses to user requests, and learns to optimize its decision making and natural language processing abilities based on this feedback.

A stock trading algorithm can use reinforcement learning to make decisions about buying and selling stocks. The algorithm receives rewards for making profitable trades and punishments for making unprofitable ones, and learns to optimize its predictions and decision making based on this feedback.

Here is a simple example of reinforcement learning in Python using the OpenAI Gym library:

import gym

# create the environment
env = gym.make('MountainCar-v0')

# initialize the agent
agent = Agent()

# run the simulation for 100 episodes
for episode in range(100):
    # reset the environment
    state = env.reset()
    
    # run the episode until it is done
    while True:
        # choose an action based on the current state
        action = agent.choose_action(state)
        
        # take the action and observe the reward and next state
        next_state, reward, done, _ = env.step(action)
        
        # update the agent based on the reward and next state
        agent.update(state, action, reward, next_state)
        
        # update the current state
        state = next_state
        
        # if the episode is done, break the loop
        if done:
            break

In this example, we create an environment using the gym.make function and initialize the agent using the Agent class. Then, we run the simulation for 100 episodes, where the agent chooses actions based on the current state and receives rewards based on the actions it takes. The agent is updated after each step, and the simulation ends when the episode is done.

And here is a somewhat more sophisticated example:

# install the OpenAI Gym and TensorFlow libraries
!pip install gym tensorflow

# import the required libraries
import gym
import numpy as np
import tensorflow as tf

# create the environment
env = gym.make('CartPole-v0')

# create the model
model = tf.keras.models.Sequential([
    tf.keras.layers.Dense(32, activation='relu', input_shape=(4,)),
    tf.keras.layers.Dense(2, activation='linear')
])

# compile the model
model.compile(
    optimizer='adam',
    loss='mse'
)

# define the agent
agent = {
    'model': model,
    'memory': [],
    'epsilon': 1,
    'epsilon_min': 0.01,
    'epsilon_decay': 0.995
}

# define the choose_action function
def choose_action(state):
    # if a random number is less than epsilon, choose a random action
    if np.random.uniform() < agent['epsilon']:
        action = np.random.randint(0, 2)
    else:
        # otherwise, predict the action using the model
        action = np.argmax(model.predict(np.array([state]))[0])
    
    # return the action
    return action

# define the remember function
def remember(state, action, reward, next_state, done):
    # add the experience to the memory
    agent['memory'].append((state, action, reward, next_state, done))
# define the replay function
def replay(batch_size):
    # sample a random batch of experiences from the memory
    batch = np.random.choice(agent['memory'], batch_size)
    
    # create empty arrays for the states, actions, and targets
    states = np.zeros((batch_size, 4))
    actions = np.zeros((batch_size, 1))
    targets = np.zeros((batch_size, 2))
    
    # loop over the experiences in the batch
    for i in range(batch_size):
        # get the state, action, reward, next_state, and done from the experience
        state = batch[i][0]
        action = batch[i][1]
        reward = batch[i][2]
        next_state = batch[i][3]
        done = batch[i][4]
        
        # if the episode is not done, calculate the target
        if not done:
            target = reward + 0.95 * np.max(model.predict(np.array([next_state]))[0])
        else:
            target = reward
        
        # add the state, action, and target to the arrays
        states[i] = state
        actions[i] = action
        targets[i] = target
    
    # update the model using the states, actions, and targets
    model.fit(states, targets, epochs=1, verbose=0)

Of course, these are simple examples, and a real-world reinforcement learning system would be much more complex. But this gives a general idea of how reinforcement learning works in Python using the OpenAI Gym library.

Reinforcement limitations

While reinforcement learning is a powerful approach to machine learning, it does have some limitations. One of the main challenges with reinforcement learning is that it can be difficult to define the rewards and punishments that the agent will receive for its actions. This can make it difficult to train the agent to optimize its behavior in a way that aligns with the desired outcomes.

Another limitation of reinforcement learning is that it can require a lot of data and computation in order to learn effectively. The agent must explore a wide range of possible actions and receive feedback in order to learn the optimal strategy, which can be time-consuming and resource-intensive.

Additionally, reinforcement learning can struggle with environments that are highly complex or stochastic, where the consequences of actions are difficult to predict. In these cases, it can be challenging for the agent to learn the optimal strategy and adapt its behavior effectively.

Overall, while reinforcement learning is a powerful approach to machine learning, it is not a perfect solution and has some limitations that need to be considered. In order to use reinforcement learning effectively, it is important to carefully define the rewards and punishments, ensure that there is enough data and computation available, and carefully consider the complexity of the environment.

In conclusion, reinforcement learning is a powerful approach to machine learning that allows agents to learn from experience and adapt their behavior based on the feedback they receive. It has many real-world applications, from robotics and gaming to finance and healthcare, and will continue to be an important area of research and development in the future.

Read More blogs in AnalyticaDSS Blogs here : BLOGS

Read More blogs in Medium : Medium Blogs

Read More blogs in R-bloggers : https://www.r-bloggers.com

The post What is Reinforcement Learning? appeared first on Analytica Data Science Solutions.