<?xml version="1.0" encoding="UTF-8"?><rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/"><channel><title>Analytica Insights</title><description>Writing on applied AI, data science, statistics, and technical computing from Analytica Data Science Solutions.</description><link>https://analyticadss.com/insights/</link><language>en-us</language><atom:link href="https://analyticadss.com/insights/rss.xml" rel="self" type="application/rss+xml"/><item><title>AI, ML, data science: three words, one map</title><link>https://analyticadss.com/insights/decoding-the-buzz-ai-ml-and-data-science-unveiled/</link><guid isPermaLink="true">https://analyticadss.com/insights/decoding-the-buzz-ai-ml-and-data-science-unveiled/</guid><description>AI is the goal, machine learning is the method, data science is the discipline. How the three fit together, where they overlap, and where to start.</description><pubDate>Tue, 01 Aug 2023 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;&lt;em&gt;Written in 2023. The specifics are of their time; the argument is what has been kept.&lt;/em&gt;&lt;/p&gt;&lt;p&gt;Three terms carry most of the hype in modern technology, and they get used as if they’re interchangeable. They aren’t, and once you see how they fit together you can’t unsee it.&lt;/p&gt;
&lt;p&gt;Here’s the map. &lt;strong&gt;AI is a goal&lt;/strong&gt;: machines doing things that would take intelligence from us. &lt;strong&gt;ML is a method&lt;/strong&gt;: reaching that goal by learning from data instead of hand-coding rules. &lt;strong&gt;Data science is a discipline&lt;/strong&gt;: extracting understanding and decisions from data, using ML when it helps and plenty of other tools when it doesn’t. I’ve written before about &lt;a href=&quot;https://analyticadss.com/insights/what-is-the-difference-between-ai-and-ml-anyway/&quot;&gt;the AI/ML boundary specifically&lt;/a&gt;; this is the wider frame.&lt;/p&gt;
&lt;p tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Figure&quot;&gt;&lt;img src=&quot;https://analyticadss.com/img/posts/buzz/ai-ml-data-science-map.svg&quot; alt=&quot;A nested diagram. Artificial intelligence is the outer region, labelled as a goal. Machine learning sits inside it as a method, with deep learning inside that. Data science overlaps machine learning but extends well outside the AI boundary, because much of it involves no learning at all.&quot; width=&quot;760&quot; height=&quot;418&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot;/&gt;&lt;/p&gt;
&lt;p&gt;The part of that picture people find surprising is the overlap on the right. Data science is not a subset of AI. Most of the working day — deciding what to measure, repairing what got measured, checking a result is not confidently wrong — has no learning in it anywhere.&lt;/p&gt;
&lt;p&gt;Each has its own history, and the histories explain the shapes. The AI ambition is old. Older than computers, if you count the myths about mechanical minds. As a research field it dates to the 1950s, and it has been through boom and winter more than once. ML grew inside it, starting from the perceptron era, and eventually became the dominant route because learned behaviour scaled where hand-written rules collapsed. Data science is the youngest: it exists because the volume of recorded data exploded past what classical analysis was built for, and someone needed to combine statistics, computation and domain judgement into one working practice.&lt;/p&gt;
&lt;p&gt;You already use all three before breakfast. Your phone sorting photos by face: ML. The predictive text above your keyboard: ML again, of a kind I have &lt;a href=&quot;https://analyticadss.com/insights/what-are-large-language-models-llms-and-how-are-they-being-used/&quot;&gt;an old soft spot for&lt;/a&gt;. Amazon nudging you toward a product, Netflix and Spotify curating your evening: recommendation models, learning from millions of people’s behaviour. A thermostat that’s learned your schedule: a small, humble intelligence in the AI sense, with ML inside. And behind every one of those products sits the data-science work nobody screenshots: deciding what to measure, cleaning what got measured, and checking the model isn’t confidently wrong.&lt;/p&gt;
&lt;p&gt;Why does the distinction earn its keep? Because the words set expectations, and expectations drive budgets. “We need AI” usually decodes to a business problem that might need a model, might need better data plumbing, and occasionally needs a spreadsheet and an honest afternoon. Calling everything AI makes every project sound like the same project. Knowing which of the three things you’re actually asking for is the difference between buying a capability and buying a word.&lt;/p&gt;
&lt;p&gt;If you’re trying to enter the field, the map tells you where to start, and it isn’t where the buzz points. Start with data science fundamentals: statistics, code, and the habit of asking what the data can actually support. ML sits on top of that. The AI framing arrives last, once you’ve earned the judgement about when learning helps at all. Curiosity is the only prerequisite, but order matters: the people who skip the foundations end up fluent in buzzwords and lost in practice.&lt;/p&gt;</content:encoded><dc:creator>Dr. Aous Abdo</dc:creator><category>AI</category><category>Data science</category><category>Machine learning</category></item><item><title>The trade we made for the internet</title><link>https://analyticadss.com/insights/pixels-and-paradoxes-the-double-edged-sword-of-digital-progress/</link><guid isPermaLink="true">https://analyticadss.com/insights/pixels-and-paradoxes-the-double-edged-sword-of-digital-progress/</guid><description>Technology quietly solved a stack of old economic problems and invoiced us in a new currency: attention, comparison and self-doubt.</description><pubDate>Thu, 01 Jun 2023 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;&lt;em&gt;Written in 2023. The specifics are of their time; the argument is what has been kept.&lt;/em&gt;&lt;/p&gt;&lt;p&gt;The internet solved a long list of hard old problems and sent us the bill in a currency we didn’t know we’d be paying.&lt;/p&gt;
&lt;p&gt;The problems it solved were economic, and it’s easy to forget how real they were. Information used to be scarce and expensive; now it’s free and instant. Communication was slow; now distance barely exists. Whole industries reorganised around that abundance, and most of us are richer for it in every way that’s easy to measure.&lt;/p&gt;
&lt;p&gt;The bill arrives in the ways that are hard to measure. A feed is a comparison engine, and it never runs out of people doing better than you — richer, fitter, happier, at least in the thirty seconds they chose to post. Self-worth starts keeping score in likes. And anonymity, which genuinely protects some people, also staffs a permanent volunteer army of cruelty. None of this was the plan. It’s just what happens when three billion people move their social lives into an attention market.&lt;/p&gt;
&lt;p tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Figure&quot;&gt;&lt;img src=&quot;https://analyticadss.com/img/posts/pixels/what-got-cheap.svg&quot; alt=&quot;Two columns. What got cheap: information, communication and reach. What got expensive: attention, self-worth, and safety from strangers.&quot; width=&quot;760&quot; height=&quot;348&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot;/&gt;&lt;/p&gt;
&lt;p&gt;Everything in the left column is easy to count, which is why it dominated every argument about whether this was going well. Nothing in the right column has a unit.&lt;/p&gt;
&lt;p&gt;So what do you actually do about it? I don’t think the answer is dramatic, and I’m suspicious of anyone selling one. A few things seem robustly true, though.&lt;/p&gt;
&lt;p&gt;Log off sometimes, on purpose. Not as a digital detox with a hashtag — just the ordinary practice of being somewhere the scoreboard can’t see you. A walk, a paper book, a conversation that nobody’s performing.&lt;/p&gt;
&lt;p&gt;Teach the mechanics, especially to kids. A person who understands that the feed is optimised to keep them scrolling has a fighting chance against it; a person who thinks it’s a neutral window onto the world doesn’t. That’s a literacy, and like any literacy it has to be taught on purpose.&lt;/p&gt;
&lt;p&gt;And take the harassment problem seriously as a design and policy failure, not a character one. Telling targets to toughen up has been tried; it doesn’t scale. Platforms that want safe spaces can build for them, and regulation has a legitimate role where they won’t.&lt;/p&gt;
&lt;p&gt;The tool isn’t going away, and I wouldn’t want it to — my whole field exists because of it. But a tool this powerful deserves to be handled deliberately. We shape these systems, and then they shape us back. The only real mistake is pretending the second half of that sentence isn’t happening.&lt;/p&gt;</content:encoded><dc:creator>Dr. Aous Abdo</dc:creator><category>Technology</category><category>Society</category></item><item><title>Functional programming in R with purrr</title><link>https://analyticadss.com/insights/unleash-the-power-of-functional-programming-in-r-with-the-purrr-package/</link><guid isPermaLink="true">https://analyticadss.com/insights/unleash-the-power-of-functional-programming-in-r-with-the-purrr-package/</guid><description>A practical guide to purrr in R: map(), map_dbl() and pmap(), safely(), quietly() and possibly(), plus examples for CSVs, models per group and plots.</description><pubDate>Fri, 14 Apr 2023 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;&lt;em&gt;Written some years ago. The ideas hold, but check package versions and API details against current documentation before relying on the code.&lt;/em&gt;&lt;/p&gt;&lt;h2 id=&quot;introduction&quot;&gt;Introduction&lt;/h2&gt;
&lt;p&gt;Most R code I inherit has the same shape: a &lt;code&gt;for&lt;/code&gt; loop, an accumulator initialised to &lt;code&gt;NULL&lt;/code&gt;, and an &lt;code&gt;if&lt;/code&gt; inside the loop that quietly does something different on the third iteration. It works. It’s also nearly impossible to test, because no piece of it is small enough to test on its own.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;purrr&lt;/code&gt; is the tidyverse’s answer to that. It gives you a family of functions that apply one function across a list and hand back something whose type you can predict, which sounds modest until you’ve spent an afternoon working out why &lt;code&gt;sapply()&lt;/code&gt; returned a matrix instead of a vector.&lt;/p&gt;
&lt;p&gt;This post covers the &lt;code&gt;map()&lt;/code&gt; family, the error wrappers I reach for most, and a few examples with real shapes: reading a directory of CSVs, fitting a model per group, generating a plot per subset. If you already write &lt;code&gt;lapply()&lt;/code&gt; everywhere, some of this will look like a rename. The parts that aren’t a rename are the ones worth your time.&lt;/p&gt;
&lt;hr/&gt;
&lt;h2 id=&quot;what-functional-programming-actually-buys-you&quot;&gt;What functional programming actually buys you&lt;/h2&gt;
&lt;p&gt;The one-line version: write functions that take inputs and return outputs, and don’t let them touch anything else.&lt;/p&gt;
&lt;p&gt;That constraint sounds academic. It isn’t. A function with no side effects returns the same answer every time you run it, so you can test it on its own, and the order you call things in stops mattering. Most of the debugging time I’ve lost in R has gone on the opposite: a variable modified three scopes away, or a loop whose result depended on what happened to be left in the environment from the previous run.&lt;/p&gt;
&lt;p&gt;Two things follow from it.&lt;/p&gt;
&lt;p&gt;Small functions are testable functions. Break a hundred-line pipeline into six named steps and you can check each against a handful of inputs. Leave it as one block and your only test is whether the whole thing crashed.&lt;/p&gt;
&lt;p&gt;Avoiding explicit loops also nudges you toward vectorised operations, which in R are the fast path. That’s a side benefit rather than the point, but it’s a real one.&lt;/p&gt;
&lt;hr/&gt;
&lt;h2 id=&quot;what-purrr-is&quot;&gt;What &lt;code&gt;purrr&lt;/code&gt; is&lt;/h2&gt;
&lt;p&gt;&lt;code&gt;purrr&lt;/code&gt; is part of the tidyverse, and its job is applying functions over lists without the type surprises base R hands you.&lt;/p&gt;
&lt;p&gt;The naming is what I’d point at first. &lt;code&gt;map()&lt;/code&gt; returns a list. &lt;code&gt;map_dbl()&lt;/code&gt; returns a double vector. &lt;code&gt;map_chr()&lt;/code&gt; returns a character vector. That suffix is a contract: if your function returns something else, you get an error on that line rather than a confusing one four steps later. &lt;code&gt;sapply()&lt;/code&gt; simplifies to whatever it feels like, and I’ve been caught by that on data where the first group happened to have exactly one row.&lt;/p&gt;
&lt;p&gt;Beyond the naming, it handles nested structures without you flattening them by hand, it has real error handling (which gets its own section below), and it composes with &lt;code&gt;dplyr&lt;/code&gt; and &lt;code&gt;tidyr&lt;/code&gt; because it was built alongside them.&lt;/p&gt;
&lt;p&gt;It’s on CRAN.&lt;/p&gt;
&lt;pre class=&quot;astro-code github-dark&quot; style=&quot;background-color:#24292e;color:#e1e4e8;overflow-x:auto&quot; tabindex=&quot;0&quot; data-language=&quot;r&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;install.packages&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt;&amp;quot;purrr&amp;quot;&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;library&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;purrr&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The rest of this post is the functions themselves, with examples you can paste.&lt;/p&gt;
&lt;hr/&gt;
&lt;h2 id=&quot;the-functions-worth-knowing&quot;&gt;The functions worth knowing&lt;/h2&gt;
&lt;p&gt;Four things cover most of what I use &lt;code&gt;purrr&lt;/code&gt; for. The &lt;code&gt;map()&lt;/code&gt; family, &lt;code&gt;pmap()&lt;/code&gt; for when several arguments vary together, the three error wrappers, and a couple of list utilities.&lt;/p&gt;
&lt;h2 id=&quot;a-the-map-family&quot;&gt;A. The map() family&lt;/h2&gt;
&lt;p&gt;This is the part you’ll use every day. &lt;code&gt;map()&lt;/code&gt; takes a list and a function, applies the function to each element, and returns a list the same length. The variants differ only in what they promise to return, and that promise is the whole value: &lt;code&gt;map_dbl()&lt;/code&gt; will error rather than quietly hand you a list.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;map()&lt;/code&gt;: Returns a list.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;map_lgl()&lt;/code&gt;: Returns a logical vector.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;map_int()&lt;/code&gt;: Returns an integer vector.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;map_dbl()&lt;/code&gt;: Returns a double vector.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;map_chr()&lt;/code&gt;: Returns a character vector.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;map_df()&lt;/code&gt;: Returns a data frame.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Example:&lt;/p&gt;
&lt;pre class=&quot;astro-code github-dark&quot; style=&quot;background-color:#24292e;color:#e1e4e8;overflow-x:auto&quot; tabindex=&quot;0&quot; data-language=&quot;r&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Define a list of numbers&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;number_list&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; list&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 2&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 3&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 4&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Square each number using map()&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;squared_numbers&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; map&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;number_list&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; ~&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; .x&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;^&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;2&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;print&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;squared_numbers&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;h2 id=&quot;b-pmap&quot;&gt;B. pmap()&lt;/h2&gt;
&lt;p&gt;&lt;code&gt;pmap()&lt;/code&gt; is for when more than one thing varies. Give it a list of argument lists and it walks them in parallel, which beats writing a loop with three indices you have to keep aligned by hand.&lt;/p&gt;
&lt;p&gt;Example:&lt;/p&gt;
&lt;pre class=&quot;astro-code github-dark&quot; style=&quot;background-color:#24292e;color:#e1e4e8;overflow-x:auto&quot; tabindex=&quot;0&quot; data-language=&quot;r&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Define two lists&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;list1&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; list&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 2&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 3&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;list2&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; list&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;4&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 5&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 6&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Add corresponding elements of the two lists using pmap()&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;sum_list&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; pmap&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;list&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;list1&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; list2&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; ~&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; ..1&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; +&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; ..2&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;print&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;sum_list&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;h2 id=&quot;c-safely-quietly-and-possibly&quot;&gt;C. safely(), quietly(), and possibly()&lt;/h2&gt;
&lt;p&gt;Is this worth a whole section? If you have ever lost a two-hour batch job to one malformed file at element 400, yes. This is the part I’d read first if you’re processing files or hitting an API. In a loop, one bad element kills the whole run and you lose everything before it. These three wrappers turn a failure into a value you can inspect afterwards, so the run finishes and you find out what broke at the end rather than at element 400 of 2,000.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;code&gt;safely()&lt;/code&gt;: Returns a list containing the result and any error encountered.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;quietly()&lt;/code&gt;: Returns a list containing the result, any warnings, and any messages.&lt;/li&gt;
&lt;li&gt;&lt;code&gt;possibly()&lt;/code&gt;: Returns a default value if an error is encountered.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Example:&lt;/p&gt;
&lt;pre class=&quot;astro-code github-dark&quot; style=&quot;background-color:#24292e;color:#e1e4e8;overflow-x:auto&quot; tabindex=&quot;0&quot; data-language=&quot;r&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Define a list with numbers and a character&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mixed_list&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; list&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 2&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt; &amp;quot;a&amp;quot;&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 3&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Define a safely wrapped square function&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;safe_square&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; safely&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;~&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; .x&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;^&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;2&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Apply the safe_square function to the mixed_list&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;results&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; map&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mixed_list&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; safe_square&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;print&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;results&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;h2 id=&quot;d-compact-and-compose&quot;&gt;D. compact() and compose()&lt;/h2&gt;
&lt;p&gt;&lt;code&gt;compact()&lt;/code&gt; is used to remove &lt;code&gt;NULL&lt;/code&gt; elements from a list, while &lt;code&gt;compose()&lt;/code&gt; allows you to combine multiple functions into a single function.&lt;/p&gt;
&lt;p&gt;Example:&lt;/p&gt;
&lt;pre class=&quot;astro-code github-dark&quot; style=&quot;background-color:#24292e;color:#e1e4e8;overflow-x:auto&quot; tabindex=&quot;0&quot; data-language=&quot;r&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Define a list with NULL elements&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;null_list&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; list&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; NULL&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 2&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; NULL&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 3&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Remove NULL elements using compact()&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;clean_list&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; compact&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;null_list&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;print&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;clean_list&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Compose two functions: square and increment&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;square&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; function&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;x&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;) &lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;x&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;^&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;2&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;increment&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; function&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;x&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;) &lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;x&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; +&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 1&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;square_and_increment&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; compose&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;increment&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; square&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Apply the composed function to a number&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;result&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; square_and_increment&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;3&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;print&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;result&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;That’s the core. What follows is the same functions on data with realistic shapes.&lt;/p&gt;
&lt;hr/&gt;
&lt;h2 id=&quot;practical-examples-with-purrr&quot;&gt;Practical examples with purrr&lt;/h2&gt;
&lt;p&gt;Four examples. Summary statistics across columns, a model per subgroup, a directory of CSVs read in one pass, and a plot per group.&lt;/p&gt;
&lt;h2 id=&quot;a-example-1-calculating-summary-statistics-for-multiple-variables&quot;&gt;A. Example 1: Calculating summary statistics for multiple variables&lt;/h2&gt;
&lt;p&gt;You have a data frame and you want the same set of statistics for every numeric column. The loop version needs a pre-allocated result and a counter. This version doesn’t.&lt;/p&gt;
&lt;pre class=&quot;astro-code github-dark&quot; style=&quot;background-color:#24292e;color:#e1e4e8;overflow-x:auto&quot; tabindex=&quot;0&quot; data-language=&quot;r&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Load required packages&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;library&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;dplyr&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;library&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;purrr&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Create a sample data frame&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;data&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; data.frame&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;  var1&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; rnorm&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;100&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; mean&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 10&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; sd&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 2&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;  var2&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; rnorm&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;100&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; mean&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 20&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; sd&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 5&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;  var3&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; rnorm&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;100&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; mean&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 30&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; sd&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 3&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;  stringsAsFactors&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; FALSE&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Define a list of summary functions&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;summary_functions&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; list&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mean&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; mean&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; median&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; median&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; sd&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; sd&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Calculate summary statistics for each variable using nested map functions&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;summary_stats&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; map_dfr&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;summary_functions&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; ~&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; map_dfc&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;data&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; .x&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; .id&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt; &amp;quot;Statistic&amp;quot;&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;print&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;summary_stats&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;h2 id=&quot;b-example-2-fitting-multiple-linear-models-for-different-subsets-of-data&quot;&gt;B. Example 2: Fitting multiple linear models for different subsets of data&lt;/h2&gt;
&lt;p&gt;Fitting one model per group is where &lt;code&gt;purrr&lt;/code&gt; starts paying for itself. Split, map, and you have a list of models you can then map over again to pull out coefficients or R-squared, without ever writing an index.&lt;/p&gt;
&lt;pre class=&quot;astro-code github-dark&quot; style=&quot;background-color:#24292e;color:#e1e4e8;overflow-x:auto&quot; tabindex=&quot;0&quot; data-language=&quot;r&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Load required packages&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;library&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;dplyr&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;library&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;purrr&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;library&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;broom&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Split the mtcars dataset by the number of cylinders&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mtcars_split&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; mtcars&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; %&amp;gt;%&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; group_split&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;cyl&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Define a function to fit a linear model and extract coefficients&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;fit_lm&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; function&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;data&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;) {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;  model&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; lm&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mpg&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; ~&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; wt&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; data&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; data&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;  coef&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; data.frame&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;tidy&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;model&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)) &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;%&amp;gt;%&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;    select&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;term&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; estimate&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;) &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;%&amp;gt;%&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;    mutate&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;cyl&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; unique&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;data&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;$&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;cyl&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;))&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;  return&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;coef&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Apply the fit_lm function to each subset using map_dfr()&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;model_coefs&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; map_dfr&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mtcars_split&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; fit_lm&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;print&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;model_coefs&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;h2 id=&quot;c-reading-multiple-csv-files-with-purrr&quot;&gt;C. Reading Multiple CSV files with purrr&lt;/h2&gt;
&lt;p&gt;This is the case I use most often, and it’s also where &lt;code&gt;safely()&lt;/code&gt; earns its keep: one malformed file in a directory of two hundred should not cost you the other 199.&lt;/p&gt;
&lt;pre class=&quot;astro-code github-dark&quot; style=&quot;background-color:#24292e;color:#e1e4e8;overflow-x:auto&quot; tabindex=&quot;0&quot; data-language=&quot;r&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Define the directory containing the CSV files&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;csv_directory&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt; &amp;quot;path/to/your/csv/files&amp;quot;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# List all CSV files in the directory&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;csv_files&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; list.files&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;csv_directory&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; pattern&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt; &amp;quot;*.csv&amp;quot;&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; full.names&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; TRUE&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Define a function to read a CSV file and add a column with the filename&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;read_csv_with_filename&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; function&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;file&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;) {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;  data&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; read_csv&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;file&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;  data&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; data&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; %&amp;gt;%&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; mutate&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;filename&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; basename&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;file&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;))&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;  return&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;data&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Read all CSV files using map_dfr() and bind the results into a single data frame&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;combined_data&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; map_dfr&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;csv_files&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; read_csv_with_filename&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;We list the CSVs, define a reader, map it over the paths, then bind the results into one frame. Swap &lt;code&gt;map()&lt;/code&gt; for &lt;code&gt;map(safely(read_one))&lt;/code&gt; and a bad file becomes a row you can inspect rather than an error that ends the run.&lt;/p&gt;
&lt;h2 id=&quot;d-purrr-and-ggplot2&quot;&gt;D. purrr and ggplot2&lt;/h2&gt;
&lt;p&gt;Same idea, different output. Instead of a list of models, you get a list of ggplot objects, one per group, which you can then arrange on a grid.&lt;/p&gt;
&lt;pre class=&quot;astro-code github-dark&quot; style=&quot;background-color:#24292e;color:#e1e4e8;overflow-x:auto&quot; tabindex=&quot;0&quot; data-language=&quot;r&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Load required packages&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;library&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;purrr&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;library&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;ggplot2&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;library&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;dplyr&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;library&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;cowplot&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Create a list of data frames, one for each unique number of cylinders in the mtcars dataset&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;data_list&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; mtcars&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; %&amp;gt;%&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;  split&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;.&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;$&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;cyl&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Define a function to create a ggplot for a given data frame&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;create_ggplot&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; function&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;data&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;) {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;  ggplot&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;data&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; aes&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;x&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; mpg&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; y&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; hp&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)) &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;+&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;    geom_point&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;aes&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;color&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; factor&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;gear&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;))&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; size&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 3&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;) &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;+&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;    labs&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;title&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; paste&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt;&amp;quot;Number of Cylinders:&amp;quot;&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; unique&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;data&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;$&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;cyl&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;))&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;         x&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt; &amp;quot;Miles per Gallon&amp;quot;&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;         y&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt; &amp;quot;Horsepower&amp;quot;&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;) &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;+&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;    theme_minimal&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;() &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;+&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;    theme&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;legend.title&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; element_blank&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;()) &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;+&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;    scale_color_discrete&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;name&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt; &amp;quot;Gears&amp;quot;&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Create a list of ggplots using map()&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;ggplot_list&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; data_list&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; %&amp;gt;%&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt; &lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;  map&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;create_ggplot&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Combine the ggplots into a single plot using cowplot&amp;#39;s plot_grid()&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;combined_plot&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; plot_grid&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;plotlist&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; ggplot_list&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; ncol&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 1&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; align&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt; &amp;quot;v&amp;quot;&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; rel_heights&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; c&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 1&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 1&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;))&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Display the combined plot&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;print&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;combined_plot&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;And this is what that one &lt;code&gt;map&lt;/code&gt; call produces — a plot per cylinder group, drawn by the same function:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://analyticadss.com/img/posts/purrr/mtcars-by-cylinder.webp&quot; alt=&quot;Three scatterplots of horsepower against miles per gallon, one panel per cylinder count, coloured by gear count — each panel generated by the same mapped function.&quot; width=&quot;720&quot; height=&quot;663&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot;/&gt;&lt;/p&gt;
&lt;p&gt;In this example, we first create a list of data frames, one for each unique number of cylinders in the &lt;code&gt;mtcars&lt;/code&gt; dataset. Then, we define a custom function &lt;code&gt;create_ggplot()&lt;/code&gt; to create a ggplot for a given data frame. The function creates a scatterplot of miles per gallon (mpg) versus horsepower (hp), with a title that reflects the number of cylinders.&lt;/p&gt;
&lt;p&gt;Finally, we use &lt;code&gt;purrr&lt;/code&gt;‘s &lt;code&gt;map()&lt;/code&gt; function to apply the custom function to each data frame in the list, resulting in a list of ggplots. We use a for loop to display each ggplot.&lt;/p&gt;
&lt;p&gt;The plot we get can be seen below:&lt;/p&gt;
&lt;p&gt;In this example, we’ve made some changes to the &lt;code&gt;create_ggplot()&lt;/code&gt; function to improve the aesthetics of the plots:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;We use &lt;code&gt;geom_point(aes(color = factor(gear)), size = 3)&lt;/code&gt; to color the points by the number of gears and increase their size.&lt;/li&gt;
&lt;li&gt;We apply &lt;code&gt;theme_minimal()&lt;/code&gt; to use a minimalistic theme for the plots.&lt;/li&gt;
&lt;li&gt;We remove the legend title using &lt;code&gt;theme(legend.title = element_blank())&lt;/code&gt;.&lt;/li&gt;
&lt;li&gt;We rename the color scale to “Gears” using &lt;code&gt;scale_color_discrete(name = &amp;quot;Gears&amp;quot;)&lt;/code&gt;.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Finally, we use the &lt;code&gt;plot_grid()&lt;/code&gt; function from the &lt;code&gt;cowplot&lt;/code&gt; package to combine the ggplots in the &lt;code&gt;ggplot_list&lt;/code&gt; into a single plot with one column and display the combined plot.&lt;/p&gt;
&lt;p&gt;The pattern in all four is the same: name the operation you want to do once, then map it. What changes is what comes back.&lt;/p&gt;
&lt;hr/&gt;
&lt;h2 id=&quot;tips-and-best-practices-for-using-purrr&quot;&gt;Tips and Best Practices for Using purrr&lt;/h2&gt;
&lt;p&gt;A few things I’d tell someone picking this up.&lt;/p&gt;
&lt;h2 id=&quot;1-use-anonymous-functions-when-appropriate&quot;&gt;1. Use anonymous functions when appropriate&lt;/h2&gt;
&lt;p&gt;When using &lt;code&gt;map()&lt;/code&gt; functions, you can create anonymous functions using the &lt;code&gt;~&lt;/code&gt; notation, which allows for concise and readable code. However, if the function becomes too complex or is used multiple times, consider defining it as a separate named function for better code organization and readability.&lt;/p&gt;
&lt;h2 id=&quot;2-compose-functions-instead-of-nesting-them&quot;&gt;2. Compose functions instead of nesting them&lt;/h2&gt;
&lt;p&gt;&lt;code&gt;compose()&lt;/code&gt; builds a new function out of existing ones, right to left. Two small named functions and a &lt;code&gt;compose()&lt;/code&gt; beats one function doing both jobs, mostly because you can test each half.&lt;/p&gt;
&lt;h2 id=&quot;3-wrap-anything-that-touches-the-outside-world&quot;&gt;3. Wrap anything that touches the outside world&lt;/h2&gt;
&lt;p&gt;Files, APIs, database calls: use &lt;code&gt;safely()&lt;/code&gt;, &lt;code&gt;quietly()&lt;/code&gt; or &lt;code&gt;possibly()&lt;/code&gt;. A single bad element should cost you that element, not the run.&lt;/p&gt;
&lt;h2 id=&quot;4-know-when-to-use-purrr-vs-base-r-or-dplyr&quot;&gt;4. Know when to use purrr vs. base R or dplyr&lt;/h2&gt;
&lt;p&gt;While &lt;code&gt;purrr&lt;/code&gt; provides a powerful and flexible way to manipulate data, there are cases where base R or &lt;code&gt;dplyr&lt;/code&gt; functions may be more appropriate or efficient. For example, if you need to perform simple operations on a data frame, consider using &lt;code&gt;dplyr&lt;/code&gt; functions like &lt;code&gt;mutate()&lt;/code&gt; or &lt;code&gt;summarize()&lt;/code&gt;. Evaluate the needs of your specific task and choose the best tool for the job.&lt;/p&gt;
&lt;h2 id=&quot;5-read-the-reference-not-just-the-cheatsheet&quot;&gt;5. Read the reference, not just the cheatsheet&lt;/h2&gt;
&lt;p&gt;There’s more in the package than the &lt;code&gt;map()&lt;/code&gt; family, and the reference is short. &lt;code&gt;imap()&lt;/code&gt;, &lt;code&gt;walk()&lt;/code&gt; and &lt;code&gt;reduce()&lt;/code&gt; are the three I wish I’d found sooner.&lt;/p&gt;
&lt;p&gt;None of this is a rule. But if you write R for a living, the error wrappers alone will save you an afternoon within the month.&lt;/p&gt;
&lt;hr/&gt;
&lt;h2 id=&quot;wrapping-up&quot;&gt;Wrapping up&lt;/h2&gt;
&lt;p&gt;&lt;code&gt;purrr&lt;/code&gt; is a rename of &lt;code&gt;lapply()&lt;/code&gt; right up until it isn’t. The &lt;code&gt;map_*()&lt;/code&gt; suffixes stop type surprises at the line that caused them, &lt;code&gt;pmap()&lt;/code&gt; removes the multi-index loop, and the error wrappers turn a crashed batch job into a result set with some failures in it.&lt;/p&gt;
&lt;p&gt;If you take one thing: wrap anything that touches a file or a network in &lt;code&gt;safely()&lt;/code&gt; before you run it over two thousand elements. I have learned that one the expensive way.&lt;/p&gt;
&lt;p&gt;Happy coding.&lt;/p&gt;
&lt;h2 id=&quot;further-reading&quot;&gt;Further reading&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;The &lt;a href=&quot;https://purrr.tidyverse.org/&quot; rel=&quot;noopener noreferrer&quot; target=&quot;_blank&quot;&gt;purrr reference&lt;/a&gt; — short, and the examples run.&lt;/li&gt;
&lt;li&gt;&lt;em&gt;R for Data Science&lt;/em&gt;, Wickham and Grolemund. The iteration chapter is the gentlest introduction to this material anywhere.&lt;/li&gt;
&lt;li&gt;&lt;em&gt;Advanced R&lt;/em&gt;, Wickham. Read the Functionals chapter when you want to know why &lt;code&gt;map()&lt;/code&gt; is built the way it is.&lt;/li&gt;
&lt;li&gt;The Posit Community forum, if you are stuck on something specific.
Harnessing these resources and proactively mingling with the wider R circle will undoubtedly refine your prowess with both the &lt;code&gt;purrr&lt;/code&gt; package and R’s functional programming realm. Continue your journey of discovery, trial, and collaborative learning to blossom as an adept data scientist and R aficionado.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Artificial Intelligence Data Science functional programming Programming R Rstats&lt;/p&gt;</content:encoded><dc:creator>Dr. Aous Abdo</dc:creator><category>R</category><category>Functional programming</category><category>tidyverse</category></item><item><title>Probability and statistics: the concepts worth being precise about</title><link>https://analyticadss.com/insights/introduction-to-probability-and-statistics-basic-concepts-and-terminology-with-visuals-part-i/</link><guid isPermaLink="true">https://analyticadss.com/insights/introduction-to-probability-and-statistics-basic-concepts-and-terminology-with-visuals-part-i/</guid><description>The vocabulary that causes the most trouble when it is used loosely — with the visuals that make each one concrete.</description><pubDate>Wed, 22 Mar 2023 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;&lt;em&gt;Written some years ago. The ideas hold, but check package versions and API details against current documentation before relying on the code.&lt;/em&gt;&lt;/p&gt;&lt;p&gt;Ask ten people what a 70% chance of rain means and you’ll get at least three answers. That’s not a failure of arithmetic. It’s that probability has more than one definition, and which one you’re using changes what the number is claiming.&lt;/p&gt;
&lt;h2 id=&quot;introduction&quot;&gt;Introduction&lt;/h2&gt;
&lt;p&gt;This is the first part of a short series on the statistics you actually need before any of the modelling makes sense. Not a course. More the set of ideas I find myself re-explaining most often, with the pictures I’d draw on a whiteboard.&lt;/p&gt;
&lt;p&gt;We’ll start with the three ways of defining probability and why they disagree, then work through the rules that let you combine events, then the descriptive statistics that summarise a sample. Every plot is generated by R code you can run, and the code is in the post.&lt;/p&gt;
&lt;h2 id=&quot;probability-fundamentals&quot;&gt;Probability Fundamentals&lt;/h2&gt;
&lt;p&gt;Probability quantifies how likely something is. That much everyone agrees on. Where it gets interesting is what the number is measuring, and there are three different answers.&lt;/p&gt;
&lt;h2 id=&quot;i-definitions&quot;&gt;I. Definitions:&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;Experiment: An action or procedure that results in one of several possible outcomes. For example, rolling a die is an experiment with six possible outcomes (1, 2, 3, 4, 5, or 6).&lt;/li&gt;
&lt;li&gt;Outcome: The result of an experiment. In the die-rolling example, if the die lands on 3, then the outcome is 3.&lt;/li&gt;
&lt;li&gt;Event: A set of one or more outcomes. In the die-rolling example, an event could be the die showing an even number, which includes the outcomes 6.&lt;/li&gt;
&lt;li&gt;Sample Space: The set of all possible outcomes of an experiment. For the die-rolling example, the sample space is 6.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;ii-types-of-probability&quot;&gt;II. Types of Probability:&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Classical&lt;/strong&gt;: Based on the assumption that all outcomes are equally likely. In the die-rolling example, the classical probability of getting an even number is 1/2 (3 even numbers out of 6 possible outcomes).&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Relative Frequency&lt;/strong&gt;: Based on the observed frequencies of outcomes in a sample. For instance, suppose we roll a die 100 times and observe 40 even numbers. The relative frequency of getting an even number is 40/100 = 0.4.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Subjective&lt;/strong&gt;: Based on an individual’s personal judgment or belief. A person may believe that it is more likely to rain tomorrow based on their interpretation of weather patterns and past experiences, even if objective data suggests otherwise.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Let’s use a real-world dataset to illustrate the relative frequency approach. We will analyze the number of cylinders in vehicles from the &lt;code&gt;mtcars&lt;/code&gt; dataset, which is included in R. We will calculate the relative frequency of vehicles with 4, 6, and 8 cylinders.&lt;/p&gt;
&lt;pre class=&quot;astro-code github-dark&quot; style=&quot;background-color:#24292e;color:#e1e4e8;overflow-x:auto&quot; tabindex=&quot;0&quot; data-language=&quot;r&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;library&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;knitr&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;data&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mtcars&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;cylinder_counts&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; mtcars&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; %&amp;gt;%&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; count&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;cyl&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;total_cars&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; sum&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;cylinder_counts&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;$&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;n&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;relative_frequencies&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; cylinder_counts&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; %&amp;gt;%&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt; &lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;mutate&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;relative_frequency&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; n&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; /&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; total_cars&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;kable&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;relative_frequencies&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; caption&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt; &amp;quot;Relative Frequencies of Cylinder Counts in Vehicles&amp;quot;&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Relative Frequencies of Cylinder Counts in Vehicles&lt;/p&gt;
&lt;p&gt;In the table above, the “cyl” column represents the number of cylinders in a vehicle, the “n” column shows the count of vehicles with the corresponding number of cylinders, and the “relative_frequency” column displays the relative frequency of each cylinder count in the dataset.&lt;/p&gt;
&lt;p&gt;Here’s the table rendered as a chart — the original output from this post’s code:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://analyticadss.com/img/posts/probability/cylinder-relative-frequency.webp&quot; alt=&quot;Relative frequency of cylinder counts in the mtcars vehicles: three bars for 4, 6 and 8 cylinders, with 8-cylinder cars the most common.&quot; width=&quot;828&quot; height=&quot;496&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot;/&gt;&lt;/p&gt;
&lt;p&gt;Now, the ggplot2 code that draws it:&lt;/p&gt;
&lt;pre class=&quot;astro-code github-dark&quot; style=&quot;background-color:#24292e;color:#e1e4e8;overflow-x:auto&quot; tabindex=&quot;0&quot; data-language=&quot;r&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;library&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;tidyverse&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;ggplot&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;relative_frequencies&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; aes&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;x&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; factor&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;cyl&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; y&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; relative_frequency&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)) &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;+&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;  geom_col&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;fill&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt; &amp;quot;steelblue&amp;quot;&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;) &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;+&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;  labs&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;title&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt; &amp;quot;Relative Frequency of Cylinder Counts in Vehicles&amp;quot;&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;       x&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt; &amp;quot;Number of Cylinders&amp;quot;&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;       y&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt; &amp;quot;Relative Frequency&amp;quot;&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;) &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;+&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;  theme_minimal&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;()&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Simulation is the quickest way to see the relative-frequency idea work. Roll a fair die a few times and the proportions are all over the place; roll it ten thousand times and they settle near a sixth. Watch how fast that happens.&lt;/p&gt;
&lt;pre class=&quot;astro-code github-dark&quot; style=&quot;background-color:#24292e;color:#e1e4e8;overflow-x:auto&quot; tabindex=&quot;0&quot; data-language=&quot;r&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;set.seed&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;42&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;) &lt;/span&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Set seed for reproducibility&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;n_rolls&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 1000&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;die_rolls&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; sample&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;:&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;6&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; size&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; n_rolls&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; replace&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; TRUE&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;die_rolls_df&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; data.frame&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;outcome&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; die_rolls&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;) &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;%&amp;gt;%&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;  count&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;outcome&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;) &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;%&amp;gt;%&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;  mutate&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;relative_frequency&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; n&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; /&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; n_rolls&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;die_rolls_df&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;This will generate a data frame with the outcome, count, and relative frequency of each die roll:&lt;/p&gt;
&lt;p&gt;Now, let’s create a bar chart to visualize the relative frequencies:&lt;/p&gt;
&lt;pre class=&quot;astro-code github-dark&quot; style=&quot;background-color:#24292e;color:#e1e4e8;overflow-x:auto&quot; tabindex=&quot;0&quot; data-language=&quot;r&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;ggplot&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;die_rolls_df&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; aes&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;x&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; factor&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;outcome&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; y&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; relative_frequency&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)) &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;+&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;  geom_col&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;fill&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt; &amp;quot;steelblue&amp;quot;&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;) &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;+&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;  labs&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;title&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt; &amp;quot;Die Roll Simulation&amp;quot;&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;       x&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt; &amp;quot;Outcome&amp;quot;&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;       y&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt; &amp;quot;Relative Frequency&amp;quot;&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;) &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;+&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;  theme_minimal&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;()&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;And here’s what came out of that run:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://analyticadss.com/img/posts/probability/die-roll-simulation.webp&quot; alt=&quot;Die-roll simulation results: six bars all close to one-sixth, none exactly equal — relative frequency converging on classical probability.&quot; width=&quot;828&quot; height=&quot;488&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot;/&gt;&lt;/p&gt;
&lt;p&gt;Six outcomes, ten thousand rolls, and every bar hovers near one-sixth without any of them landing exactly there. That wobble &lt;em&gt;is&lt;/em&gt; the point: relative frequency converges on the classical answer, and never quite equals it at any finite number of rolls.&lt;/p&gt;
&lt;h2 id=&quot;iii-probability-rules&quot;&gt;III. Probability Rules:&lt;/h2&gt;
&lt;p&gt;&lt;strong&gt;Addition Rule&lt;/strong&gt;: The addition rule helps us calculate the probability of either event A or event B (or both) occurring. The rule is defined as:&lt;/p&gt;
&lt;p&gt;P(A ∪ B) = P(A) + P(B) − P(A ∩ B)&lt;/p&gt;
&lt;p&gt;Here, P(A ∪ B) represents the probability of event A or event B occurring, while P(A ∩ B) denotes the probability of both events A and B happening together.&lt;/p&gt;
&lt;p&gt;Example: Suppose we have a deck of 52 playing cards. What is the probability of drawing either a red card (hearts or diamonds) or a queen?&lt;/p&gt;
&lt;p&gt;There are 26 red cards and 4 queens in the deck, but 2 of the queens are also red cards (queen of hearts and queen of diamonds). So, applying the addition rule:&lt;/p&gt;
&lt;p&gt;P(Red ∪ Queen) = P(Red) + P(Queen) − P(Red ∩ Queen)&lt;br/&gt;
P(Red ∪ Queen) = (26/52) + (4/52) − (2/52) = 28/52 ≈ 0.5385&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Multiplication Rule&lt;/strong&gt;: The multiplication rule helps us determine the probability of both events A and B occurring simultaneously. The rule is defined as:&lt;/p&gt;
&lt;p&gt;P(A ∩ B) = P(A|B) * P(B)&lt;/p&gt;
&lt;p&gt;Here, P(A|B) represents the probability of event A occurring given that event B has occurred.&lt;/p&gt;
&lt;p&gt;Example: Consider a bag containing 5 blue and 3 red balls. We draw two balls from the bag without replacement. What is the probability of drawing a blue ball first, followed by a red ball?&lt;/p&gt;
&lt;p&gt;To apply the multiplication rule, we first calculate the probability of each event:&lt;/p&gt;
&lt;p&gt;P(Blue1) = 5/8 P(Red2|Blue1) = 3/7&lt;/p&gt;
&lt;p&gt;Now, we can compute the probability of both events happening together:&lt;/p&gt;
&lt;p&gt;P(Blue1 ∩ Red2) = P(Blue1) * P(Red2|Blue1) = (5/8) * (3/7) ≈ 0.2679&lt;/p&gt;
&lt;p&gt;In the next part of this series, we will cover topics related to Descriptive Statistics.&lt;/p&gt;
&lt;p&gt;For more practical tips and insights on AI, data science, and statistics, explore our blog at Analytica Data Science Solutions. Discover engaging content to expand your knowledge and stay up-to-date with the latest developments.&lt;/p&gt;
&lt;p&gt;In this blog post, we have covered the basics of probability and statistics. If you wish to further expand your knowledge and understanding, here are some references to help you dive deeper into these topics:&lt;/p&gt;
&lt;p&gt;Books:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;DeGroot, M. H., &amp;amp; Schervish, M. J. (2012). Probability and Statistics (4th ed.). Pearson.&lt;/li&gt;
&lt;li&gt;Wackerly, D., Mendenhall, W., &amp;amp; Scheaffer, R. L. (2007). Mathematical Statistics with Applications (7th ed.). Cengage Learning.&lt;/li&gt;
&lt;li&gt;Casella, G., &amp;amp; Berger, R. L. (2001). Statistical Inference (2nd ed.). Duxbury Press.&lt;/li&gt;
&lt;li&gt;Hastie, T., Tibshirani, R., &amp;amp; Friedman, J. (2009). The Elements of Statistical Learning: Data Mining, Inference, and Prediction (2nd ed.). Springer.&lt;/li&gt;
&lt;li&gt;Wickham, H., &amp;amp; Grolemund, G. (2016). R for Data Science: Import, Tidy, Transform, Visualize, and Model Data. O’Reilly Media.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Websites:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Khan Academy: Probability and Statistics: &lt;a href=&quot;https://www.khanacademy.org/math/statistics-probability&quot; rel=&quot;noopener noreferrer&quot; target=&quot;_blank&quot;&gt;https://www.khanacademy.org/math/statistics-probability&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Stat Trek: Teach yourself statistics: &lt;a href=&quot;https://stattrek.com/&quot; rel=&quot;noopener noreferrer&quot; target=&quot;_blank&quot;&gt;https://stattrek.com/&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Carnegie Mellon University Probability &amp;amp; Statistics: &lt;a href=&quot;https://oli.cmu.edu/courses/probability-statistics-open-free/&quot; rel=&quot;noopener noreferrer&quot; target=&quot;_blank&quot;&gt;https://oli.cmu.edu/courses/probability-statistics-open-free/&lt;/a&gt;&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;Online Courses:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Coursera: Statistics with R Specialization by Duke University: &lt;a href=&quot;https://www.coursera.org/specializations/statistics&quot; rel=&quot;noopener noreferrer&quot; target=&quot;_blank&quot;&gt;https://www.coursera.org/specializations/statistics&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;Coursera: Introduction to Probability and Data with R by Duke University: &lt;a href=&quot;https://www.coursera.org/learn/probability-intro&quot; rel=&quot;noopener noreferrer&quot; target=&quot;_blank&quot;&gt;https://www.coursera.org/learn/probability-intro&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;edX: Probability and Statistics in Data Science using Python by the University of California, San Diego: &lt;a href=&quot;https://www.edx.org/course/probability-and-statistics-in-data-science-using-python&quot; rel=&quot;noopener noreferrer&quot; target=&quot;_blank&quot;&gt;https://www.edx.org/course/probability-and-statistics-in-data-science-using-python&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;DataCamp: Introduction to Probability in R: &lt;a href=&quot;https://www.datacamp.com/courses/introduction-to-probability-in-r&quot; rel=&quot;noopener noreferrer&quot; target=&quot;_blank&quot;&gt;https://www.datacamp.com/courses/introduction-to-probability-in-r&lt;/a&gt;&lt;/li&gt;
&lt;li&gt;DataCamp: Foundations of Probability in R: &lt;a href=&quot;https://www.datacamp.com/courses/foundations-of-probability-in-r&quot; rel=&quot;noopener noreferrer&quot; target=&quot;_blank&quot;&gt;https://www.datacamp.com/courses/foundations-of-probability-in-r&lt;/a&gt;&lt;/li&gt;
&lt;/ol&gt;</content:encoded><dc:creator>Dr. Aous Abdo</dc:creator><category>Statistics</category><category>Probability</category><category>Fundamentals</category></item><item><title>What AI actually does for an airline</title><link>https://analyticadss.com/insights/flying-into-the-future-how-artificial-intelligence-is-revolutionizing-the-airline-industry/</link><guid isPermaLink="true">https://analyticadss.com/insights/flying-into-the-future-how-artificial-intelligence-is-revolutionizing-the-airline-industry/</guid><description>Forget the brochure version. Where machine learning genuinely earns money in aviation: fuel, maintenance, fraud — and where it still can&apos;t cope.</description><pubDate>Sun, 01 Jan 2023 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;&lt;em&gt;Written in 2023. The specifics are of their time; the argument is what has been kept.&lt;/em&gt;&lt;/p&gt;&lt;p&gt;Airlines are a strange business: colossal fixed costs, brutal margins, and an operation where a single cascading delay can burn a day’s profit. That combination makes them one of the places where machine learning stops being a brochure word and starts paying for itself in specific, countable ways.&lt;/p&gt;
&lt;p&gt;Where does the money actually come from? Mostly four places.&lt;/p&gt;
&lt;p&gt;Fuel and routing. Flight planning is an optimisation problem with weather, winds, traffic and cost all moving at once, and models trained on past flights find efficiencies that hand-built plans miss. A fraction of a percent of fuel across a fleet is serious money.&lt;/p&gt;
&lt;p&gt;Maintenance is my favourite, because it’s the same problem we solve for &lt;a href=&quot;https://analyticadss.com/work/preventive-maintenance-rul/&quot;&gt;mining equipment&lt;/a&gt;: predict the failure before it grounds the machine. A component that fails on schedule is a line item. One that fails at the gate is cancellations radiating through a network all day. Models that estimate remaining life from sensor data move failures from the second category into the first.&lt;/p&gt;
&lt;p&gt;Fraud, quietly, is huge. Stolen cards and forged documents flow through booking systems at volume, and learned models catch patterns in that stream no rule-writer anticipated.&lt;/p&gt;
&lt;p&gt;And the customer-facing layer everyone actually meets: chatbots handling bookings and check-in questions, and recommendation systems deciding which fare or route to offer you. Useful, if oversold — which brings me to the honest part.&lt;/p&gt;
&lt;p&gt;The airline chatbot is also where the technology’s limits are on public display. These systems handle the routine beautifully and fail exactly when you need them most: the cancelled connection, the weird itinerary, the situation with no template. Everyone who has shouted “agent!” at a phone tree understands the boundary between AI at its best and AI at its worst. It isn’t shy about showing you.&lt;/p&gt;
&lt;p tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Figure&quot;&gt;&lt;img src=&quot;https://analyticadss.com/img/posts/aviation/where-the-money-is.svg&quot; alt=&quot;Four points marked across an airline operation: fraud detection at booking, fuel and routing at planning, predictive maintenance between flights, and the customer-facing layer at the gate. A note marks where the chatbot fails — the cancelled connection with no template.&quot; width=&quot;760&quot; height=&quot;392&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot;/&gt;&lt;/p&gt;
&lt;p&gt;Three of those four are invisible to passengers, and they are where the money is. The one everybody meets is the one with the worst failure mode.&lt;/p&gt;
&lt;p&gt;Two other caveats belong in any honest account. Displacement is real: automation here means fewer humans doing the routine work, and “retraining” has to be a programme, not a press release. And bias can ride along in anything trained on historical customer data: a pricing or screening model that learned from the past will happily reproduce it. Aviation’s own safety culture is actually the model here, with its nearly fanatical insistence on knowing &lt;em&gt;why&lt;/em&gt; systems do what they do before trusting them. The industry that invented the checklist and the black box shouldn’t accept unexplainable models for consequential decisions, and increasingly it doesn’t.&lt;/p&gt;
&lt;p&gt;So: revolutionising? I’d say something less cinematic and more valuable. AI in aviation is compounding: a percent here in fuel, a grounded-aircraft day avoided there, a fraud ring caught early. In an industry with these margins, compounding quietly &lt;em&gt;is&lt;/em&gt; the revolution.&lt;/p&gt;</content:encoded><dc:creator>Dr. Aous Abdo</dc:creator><category>AI</category><category>Aviation</category></item><item><title>AGI, minus the theatre</title><link>https://analyticadss.com/insights/is-agi-the-next-step-in-human-evolution-or-a-dangerous-experiment/</link><guid isPermaLink="true">https://analyticadss.com/insights/is-agi-the-next-step-in-human-evolution-or-a-dangerous-experiment/</guid><description>General intelligence in machines is a hypothesis, not a schedule. What the optimists and pessimists actually claim, and what&apos;s worth worrying about now.</description><pubDate>Sun, 01 Jan 2023 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;&lt;em&gt;Early 2023. The capability frontier has moved dramatically since; the framing — treat AGI as a hypothesis with uncertain timing rather than a scheduled event — is the part I&apos;d still defend.&lt;/em&gt;&lt;/p&gt;&lt;p&gt;Artificial general intelligence is the hypothesis that machines will one day match and then exceed human ability across the board. Not one task. All of them. Popular culture has done this idea no favours. Between the killer robots and the digital messiahs, it’s genuinely hard to hear the serious version of the argument, so let me try to state it.&lt;/p&gt;
&lt;p&gt;The people building toward AGI aren’t cranks. Demis Hassabis at DeepMind, Ilya Sutskever at OpenAI, and researchers like Ben Goertzel have all said, in different registers, that general systems are achievable and worth pursuing. Their case rests on trajectory: narrow AI kept absorbing tasks that were supposed to require “real” intelligence, and the methods kept generalising further than the sceptics predicted. The critics answer that competence isn’t understanding. Scaling pattern-learners produces broader pattern-learners, they argue, and the step to flexible, self-directed general intelligence may require ideas nobody has yet. Some argue that ingredient looks like reasoning about one’s own goals; others suspect whole missing architectures.&lt;/p&gt;
&lt;p&gt;Here’s my honest position: nobody knows, and the confident timelines in both directions are performances. What I find more useful than “when” is noticing that the hard problems don’t wait for AGI. They arrive early, with systems well short of general.&lt;/p&gt;
&lt;p tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Figure&quot;&gt;&lt;img src=&quot;https://analyticadss.com/img/posts/agi/problems-arrive-early.svg&quot; alt=&quot;A single axis from the systems we have now toward general intelligence, marked with a question mark because no timeline is established. Three problems usually filed under AGI — concentration, displacement and control — are marked near the present-day end.&quot; width=&quot;760&quot; height=&quot;360&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot;/&gt;&lt;/p&gt;
&lt;p&gt;The timeline argument is the loud one, and it is the one nobody can settle. Everything marked on the left is already happening, on systems that are not remotely general — which makes the schedule debate a poor reason to wait.&lt;/p&gt;
&lt;p&gt;Concentration arrives early. If capability at this level accrues to a handful of labs and states, the gap between those who own it and everyone else compounds. That isn’t a sci-fi scenario. It’s an extrapolation of the last decade.&lt;/p&gt;
&lt;p&gt;Job displacement arrives early, as &lt;a href=&quot;https://analyticadss.com/insights/rise-of-the-machines-what-to-expect-in-the-new-era-of-ai/&quot;&gt;I’ve written elsewhere&lt;/a&gt;, and lands unevenly.&lt;/p&gt;
&lt;p&gt;And the control question arrives earliest of all, because it’s really a question about optimisation, not consciousness: a highly capable system pursuing a badly specified objective is dangerous with no self-awareness whatsoever. That’s why the &lt;a href=&quot;https://analyticadss.com/insights/the-ethical-dangers-of-ai-how-uncertainty-can-help-us-avoid-unforeseen-consequences/&quot;&gt;uncertainty-about-objectives line of work&lt;/a&gt; matters now, whatever AGI turns out to be.&lt;/p&gt;
&lt;p&gt;So — next step in evolution, or dangerous experiment? I reject the framing, and I think you should too. Evolution implies inevitability; experiment implies a lab we could simply close. What’s actually happening is a technology whose ceiling nobody has established, being developed at speed, under competitive pressure, by institutions we built and can still shape. The right responses are unglamorous: safety research that assumes capability keeps rising, governance written before deployment instead of after the first scandal, and a firm refusal to let either the hype or the doom talk us out of taking the boring parts seriously.&lt;/p&gt;
&lt;p&gt;The theatre is optional. The trajectory isn’t.&lt;/p&gt;</content:encoded><dc:creator>Dr. Aous Abdo</dc:creator><category>AI</category><category>AGI</category></item><item><title>R on steroids: Rcpp</title><link>https://analyticadss.com/insights/r-on-steroids-with-rcpp-library/</link><guid isPermaLink="true">https://analyticadss.com/insights/r-on-steroids-with-rcpp-library/</guid><description>How to move one slow R loop into C++ with Rcpp&apos;s cppFunction(): a complete working example, a measured 24x speedup, and when it isn&apos;t worth it.</description><pubDate>Sun, 01 Jan 2023 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;&lt;em&gt;Written some years ago. The ideas hold, but check package versions and API details against current documentation before relying on the code.&lt;/em&gt;&lt;/p&gt;&lt;p&gt;Sometimes the honest answer to “why is my R slow?” is that the computation is a genuine loop, each step needing the last, and &lt;a href=&quot;https://analyticadss.com/insights/maximizing-efficiency-with-loops-and-vectorization-in-programming-languages/&quot;&gt;no amount of vectorizing&lt;/a&gt; will change that. For exactly those cases, R keeps an escape hatch with a beautiful power-to-effort ratio: Rcpp.&lt;/p&gt;
&lt;p&gt;The idea: write the one hot function in C++, and Rcpp compiles it and welds it into your session as if it were a normal R function. Everything else — data wrangling, plotting, the whole comfortable tidy world — stays in R. You’re not rewriting your project in C++. You’re extraditing one guilty function.&lt;/p&gt;
&lt;p&gt;Why does it work? R is interpreted; every pass through an R loop pays interpreter overhead per element. C++ is compiled to machine code before it runs, so a loop is just a loop. For scalar-heavy, iteration-heavy work the difference is routinely 10–100x.&lt;/p&gt;
&lt;p&gt;Here’s the entire workflow, using a cumulative-sum-with-a-twist that genuinely can’t be vectorized (each element depends on the previous &lt;em&gt;output&lt;/em&gt;):&lt;/p&gt;
&lt;pre class=&quot;astro-code github-dark&quot; style=&quot;background-color:#24292e;color:#e1e4e8;overflow-x:auto&quot; tabindex=&quot;0&quot; data-language=&quot;r&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;install.packages&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt;&amp;quot;Rcpp&amp;quot;&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;library&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;Rcpp&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;cppFunction&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt;&amp;#39;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt;NumericVector decayCumsum(NumericVector x, double decay) {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt;  int n = x.size();&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt;  NumericVector out(n);&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt;  out[0] = x[0];&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt;  for (int i = 1; i &amp;lt; n; i++) {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt;    out[i] = x[i] + decay * out[i - 1];&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt;  }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt;  return out;&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt;}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt;&amp;#39;&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;x&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; runif&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;1e7&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;system.time&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;decayCumsum&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;x&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 0.9&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;))   &lt;/span&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# ~0.02s&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# The same function in pure R&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;decayR&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; function&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;x&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; decay&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;) {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;  out&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; numeric&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;length&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;x&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)); &lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;out&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;] &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;&amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; x&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;  for&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;i&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; in&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 2&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;:&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;length&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;x&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)) &lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;out&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;i&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;] &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;&amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; x&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;i&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;] &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;+&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; decay&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; *&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; out&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;i&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; -&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 1&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;  out&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;system.time&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;decayR&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;x&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 0.9&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;))        &lt;/span&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# ~0.46s on the same machine&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Figure&quot;&gt;&lt;img src=&quot;https://analyticadss.com/img/posts/rcpp/rcpp-measured-speedup.svg&quot; alt=&quot;Two horizontal bars. The plain R loop over ten million elements takes 460 milliseconds; the identical loop written in C++ and compiled with Rcpp takes 19 milliseconds, a 24-fold speedup.&quot; width=&quot;760&quot; height=&quot;300&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot;/&gt;&lt;/p&gt;
&lt;p&gt;Twenty-four times faster for fourteen lines of C++, and the calling code can’t tell the difference. &lt;code&gt;cppFunction&lt;/code&gt; handles the compiling, the type bridging and the loading, which are the parts that made “just write it in C” a week-long detour in the old days.&lt;/p&gt;
&lt;p&gt;That number used to be a hundred, and I’d have repeated it if I hadn’t re-run the benchmark for this figure. R grew a byte-code compiler that’s been on by default since version 3.4, and it’s very good at exactly this shape of loop. Turn it off with &lt;code&gt;compiler::enableJIT(0)&lt;/code&gt; and the R side goes back to 5.5 seconds, which is where the old hundred-fold figure came from. So the escape hatch is still worth having — it just has a smaller gap to jump than it used to, and the honest version of the claim is the one you measured this morning.&lt;/p&gt;
&lt;p&gt;When should you actually reach for this? My rule has three parts. Profile first, because the slow part is never where you think (&lt;code&gt;Rprof&lt;/code&gt; or the profvis package will tell you). Vectorize second, because if the loop &lt;em&gt;can&lt;/em&gt; become a vector operation, that’s free speed with no toolchain. Rcpp third, for the loops that survive: simulations, recursive filters, MCMC innards, tree traversals.&lt;/p&gt;
&lt;p&gt;And when shouldn’t you? When the bottleneck is I/O or memory rather than CPU, because C++ won’t make your disk faster. When the R version runs in eight seconds once a month. The maintenance cost of a second language in the project is real, and somebody who doesn’t know C++ will inherit this someday. Speed you don’t need is complexity you paid for anyway.&lt;/p&gt;
&lt;p&gt;But when the profile says one function is eating ninety percent of an hour-long run? Fourteen lines. It still feels like cheating.&lt;/p&gt;</content:encoded><dc:creator>Dr. Aous Abdo</dc:creator><category>R</category><category>C++</category><category>Performance</category></item><item><title>AI in policing: useful, and easy to get wrong</title><link>https://analyticadss.com/insights/revolutionizing-law-enforcement-the-benefits-and-challenges-of-ai/</link><guid isPermaLink="true">https://analyticadss.com/insights/revolutionizing-law-enforcement-the-benefits-and-challenges-of-ai/</guid><description>Facial recognition, predictive policing, crime mapping: what these systems genuinely do well, and the specific ways each one goes wrong.</description><pubDate>Sun, 01 Jan 2023 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;&lt;em&gt;Written in 2023. The specifics are of their time; the argument is what has been kept.&lt;/em&gt;&lt;/p&gt;&lt;p&gt;Police departments have been using algorithms far longer than the current AI conversation suggests — automated fingerprint matching dates to the 1970s, which surprises most people I mention it to. What’s new is scope: systems that recognise faces in crowds, forecast where crime will occur, and score individuals for risk. Some of this genuinely helps. Some of it fails in ways that worry me as someone who builds these systems for a living. The honest article is the one that says which is which.&lt;/p&gt;
&lt;p&gt;Start with what the technology does well, because it’s real. Machines process volume humans can’t: matching evidence across databases in seconds, finding patterns across thousands of case files, mapping incidents to reveal hotspots and trends an analyst would take weeks to surface. Speed matters in investigations, and consistency matters everywhere — a model applies the same criteria at 4 p.m. and 4 a.m. No tired human can say that.&lt;/p&gt;
&lt;p&gt;But each headline application carries a specific, documented failure mode, and they deserve to be named rather than waved at.&lt;/p&gt;
&lt;p&gt;Facial recognition’s is unequal error. The systems deployed by US departments perform measurably worse on darker-skinned faces, and a false match in this context isn’t a product bug. It’s a wrongful arrest. The known wrongful-arrest cases in the US trace to exactly this failure, compounded by officers treating a match score as an identification.&lt;/p&gt;
&lt;p&gt;Predictive policing’s is the feedback loop, and it’s subtle enough to fool honest people. The models forecast crime where crime was previously &lt;em&gt;recorded&lt;/em&gt;. But recorded crime reflects where police patrolled. Send patrols where the model points, generate more records there, retrain: the system is now confirming its own bias and calling it accuracy. “The algorithm says so” launders yesterday’s patrol pattern into tomorrow’s.&lt;/p&gt;
&lt;p tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Figure&quot;&gt;&lt;img src=&quot;https://analyticadss.com/img/posts/police/predictive-policing-loop.svg&quot; alt=&quot;A four-step cycle: patrols go where the model points, more crime gets recorded there, the model retrains on those records, and it predicts crime in the same place. A note explains that the flaw is in the data rather than the code — nothing in the loop ever measures where crime actually happened.&quot; width=&quot;760&quot; height=&quot;462&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot;/&gt;&lt;/p&gt;
&lt;p&gt;Every individual step in that ring is defensible, which is exactly why it gets built by people acting in good faith. Nowhere in the loop is there a measurement of where crime happened.&lt;/p&gt;
&lt;p&gt;Risk scoring’s is consequence without explanation. When a model influences bail or sentencing, the person affected deserves to know why. Yet the systems in actual use have often been proprietary black boxes that neither defendant nor judge could interrogate.&lt;/p&gt;
&lt;p&gt;Underneath all three runs the privacy question: these tools work by watching everyone to find anyone, and mass collection is a cost even when nothing goes wrong.&lt;/p&gt;
&lt;p&gt;So what would using this well look like? Not abstinence; the volume problems are real and the tools help. It looks like the boring governance the vendors never lead with: independent accuracy audits, by demographic, before deployment and continuously after. Explanations for any consequential decision, as a requirement rather than a courtesy. Human decision-makers who treat model output as one input, and are trained on exactly how it fails. And retiring systems that can’t meet that bar, however impressive the demo.&lt;/p&gt;
&lt;p&gt;I see the same pattern in every domain we work in. But nowhere are the stakes of a false positive higher, and nowhere is “the computer said so” more corrosive. A tool this powerful in a domain this consequential earns its place through accountability, or it doesn’t belong there.&lt;/p&gt;</content:encoded><dc:creator>Dr. Aous Abdo</dc:creator><category>AI</category><category>Policy</category><category>Ethics</category></item><item><title>The Riemann hypothesis, without the fog</title><link>https://analyticadss.com/insights/riemann-hypothesis-a-mathematical-exploration/</link><guid isPermaLink="true">https://analyticadss.com/insights/riemann-hypothesis-a-mathematical-exploration/</guid><description>What the Riemann hypothesis claims, why the zeros of the zeta function control the primes, and what rests on it. In plain language, with one picture.</description><pubDate>Sun, 01 Jan 2023 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;&lt;em&gt;Written in 2023. The hypothesis is still open.&lt;/em&gt;&lt;/p&gt;&lt;p&gt;There’s a function that seems to know where the prime numbers live. The Riemann hypothesis is a claim about that function, it has been open since 1859, and the Clay Mathematics Institute will pay a million dollars to whoever settles it.&lt;/p&gt;
&lt;p&gt;Here’s the function:&lt;/p&gt;
&lt;p&gt;ζ(s) = 1⁻ˢ + 2⁻ˢ + 3⁻ˢ + 4⁻ˢ + …&lt;/p&gt;
&lt;p&gt;Feed it a complex number &lt;em&gt;s&lt;/em&gt; and, with a suitable extension of the definition, you get a value back. Some inputs return exactly zero. A batch of those are boring and fully understood: the negative even integers, called the trivial zeros. The interesting ones, the non-trivial zeros, all seem to line up on a single vertical line in the complex plane, the line where the real part of &lt;em&gt;s&lt;/em&gt; equals 1/2.&lt;/p&gt;
&lt;p&gt;The Riemann hypothesis says: every last one of them is on that line. No stragglers, ever.&lt;/p&gt;
&lt;h2 id=&quot;why-anyone-should-care-where-zeros-sit&quot;&gt;Why anyone should care where zeros sit&lt;/h2&gt;
&lt;p&gt;Because of what Riemann showed on the way to the conjecture. The zeros of this function control the distribution of the primes. How can a smooth function know about the primes, which look like pure static? Because there’s an exact formula rebuilding the prime counts from the zeros: each zero contributes a wave, and the primes emerge from the interference pattern. If all the zeros sit on the critical line, the primes are as evenly spread as they can possibly be, and a whole family of error terms across number theory snaps to its best possible bound. If even one zero sits off the line, the primes have a bias nobody has detected, and hundreds of theorems that begin “assuming the Riemann hypothesis” lose their footing.&lt;/p&gt;
&lt;p tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Figure&quot;&gt;&lt;img src=&quot;https://analyticadss.com/img/posts/riemann/zeros-rebuild-primes.svg&quot; alt=&quot;Four curves on one set of axes. A red staircase steps up at every prime power — the exact truth. A dashed line with no zeros included is a smooth ramp that catches the trend and none of the detail. Adding the first ten zeros produces waves that begin tracking the steps; the first sixty trace the staircase almost exactly.&quot; width=&quot;760&quot; height=&quot;440&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot;/&gt;&lt;/p&gt;
&lt;p&gt;I generated that with the first sixty zeros and a sieve, because the claim is much more convincing when you watch it happen. The smooth ramp is what you get knowing nothing about the zeros. Every wave you add is one zero speaking, and they interfere their way into the exact positions of the primes.&lt;/p&gt;
&lt;p&gt;That’s the real stake. It isn’t one isolated puzzle; it’s a load-bearing wall. Results about prime gaps, about how well we can approximate the prime-counting function, even estimates that matter for cryptography, lean on it.&lt;/p&gt;
&lt;h2 id=&quot;the-evidence-honestly-stated&quot;&gt;The evidence, honestly stated&lt;/h2&gt;
&lt;p&gt;Computation is lopsidedly in favour. Not the first million zeros. The first ten &lt;em&gt;trillion&lt;/em&gt; have been computed, and every one lies exactly on the critical line. There are also structural results: Hardy proved in 1914 that infinitely many zeros lie on the line, and later work showed a positive fraction of them must.&lt;/p&gt;
&lt;p&gt;And yet, as with &lt;a href=&quot;https://analyticadss.com/insights/unsolved-math-problems-the-goldbach-conjecture/&quot;&gt;Goldbach&lt;/a&gt;, none of that is a proof. Number theory has a humbling history on this point. Some plausible conjectures with mountains of numerical support have turned out false, with the first counterexample living at a height no computation would ever visit. Ten trillion polite zeros constrain nothing about the ten-trillion-and-first.&lt;/p&gt;
&lt;h2 id=&quot;can-you-touch-it-yourself&quot;&gt;Can you touch it yourself?&lt;/h2&gt;
&lt;p&gt;You can. The zeros are computable, and libraries in Python (&lt;code&gt;mpmath&lt;/code&gt;) and R will hand you as many as you like; the first one sits at about 1/2 + 14.13&lt;em&gt;i&lt;/em&gt;. There’s something clarifying about printing a few and watching the real parts come back 0.5, 0.5, 0.5. It converts the problem from folklore into an object.&lt;/p&gt;
&lt;p&gt;What it doesn’t convert into is a proof, and that’s the lesson I take from this one every time I revisit it. The gap between “true in every case we can reach” and “true” is where the whole difficulty lives. In mathematics, and honestly, in a lot of the inference work we do for a living too.&lt;/p&gt;</content:encoded><dc:creator>Dr. Aous Abdo</dc:creator><category>Mathematics</category><category>Number theory</category></item><item><title>Uncertainty is a safety feature</title><link>https://analyticadss.com/insights/the-ethical-dangers-of-ai-how-uncertainty-can-help-us-avoid-unforeseen-consequences/</link><guid isPermaLink="true">https://analyticadss.com/insights/the-ethical-dangers-of-ai-how-uncertainty-can-help-us-avoid-unforeseen-consequences/</guid><description>Stuart Russell&apos;s argument, in short: an AI that is certain about its objective is dangerous precisely because of that certainty.</description><pubDate>Sun, 01 Jan 2023 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;&lt;em&gt;A short note from early 2023, prompted by a Stuart Russell interview. The argument has aged well; the systems around it have changed a lot.&lt;/em&gt;&lt;/p&gt;&lt;p&gt;The scariest AI system isn’t one that’s confused about what you want. It’s one that’s certain.&lt;/p&gt;
&lt;p&gt;That’s the argument Stuart Russell made in an interview with the World Economic Forum, and it stuck with me enough to write it down. The standard way we build optimising systems is to hand them a fixed objective and let them pursue it as hard as the machinery allows. The trouble is that any objective you can write down is a proxy for what you actually want, and a sufficiently capable optimiser will exploit the gap. Tell a system to maximise throughput on a production line and, as far as it’s concerned, worker safety and environmental damage are just variables it was never asked about.&lt;/p&gt;
&lt;p&gt;The failure isn’t malice. It’s literalism at scale.&lt;/p&gt;
&lt;p&gt;Russell’s proposal flips the design. Build systems that are &lt;em&gt;uncertain&lt;/em&gt; about the objective — systems that treat human preferences as something to be learned rather than something fully specified up front. A machine that knows it might have the goal wrong behaves differently in exactly the ways you’d want: it asks before doing something drastic, it accepts correction, it leaves things reversible. Deference falls out of doubt. You don’t have to bolt it on.&lt;/p&gt;
&lt;p tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Figure&quot;&gt;&lt;img src=&quot;https://analyticadss.com/img/posts/uncertainty/certain-or-uncertain-objective.svg&quot; alt=&quot;Two panels compared. A system certain of its objective pursues what was written as hard as it can, treats anything unmentioned as free to spend, and has no reason to ask or to accept being switched off. A system uncertain of its objective asks before acting drastically, accepts correction as information, and leaves things reversible.&quot; width=&quot;760&quot; height=&quot;372&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot;/&gt;&lt;/p&gt;
&lt;p&gt;Read down the right-hand column and notice that none of those behaviours had to be specified. They are what wanting the right thing looks like when you are not sure what the right thing is.&lt;/p&gt;
&lt;p&gt;Does this solve AI safety? No, and Russell doesn’t claim it does. Someone still has to decide whose preferences count and what happens when they conflict, and an economy absorbing this technology has distributional questions no objective function will settle. But as a design principle it has a quality I value in our own statistical work: it locates the danger correctly. The risk was never that the machine is stupid. The risk is that it’s competent in service of a goal we specified badly and it never thought to question.&lt;/p&gt;
&lt;p&gt;We put uncertainty on every estimate we ship, because a number without an error bar invites more confidence than it has earned. Russell’s point is the same instinct applied to goals. Certainty you haven’t earned is the hazard — for models, and for the people deploying them.&lt;/p&gt;</content:encoded><dc:creator>Dr. Aous Abdo</dc:creator><category>AI</category><category>Ethics</category></item><item><title>Fusion, AI, and a joke that stopped being funny</title><link>https://analyticadss.com/insights/unleashing-the-power-of-fusion-the-role-of-ai-in-harnessing-the-ultimate-energy-source/</link><guid isPermaLink="true">https://analyticadss.com/insights/unleashing-the-power-of-fusion-the-role-of-ai-in-harnessing-the-ultimate-energy-source/</guid><description>Written the week the NIF achieved ignition: what actually happened, the physicists&apos; oldest joke, and where machine learning fits in a tokamak.</description><pubDate>Tue, 13 Dec 2022 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;&lt;em&gt;Written in 2022. The specifics are of their time; the argument is what has been kept.&lt;/em&gt;&lt;/p&gt;&lt;p&gt;There’s a joke physicists have told for generations: fusion is ten years away, and always will be. I heard it over lunch at the Institute for Defense Analyses back in 2012, delivered with the weary timing of someone who’d heard it in graduate school themselves. On 13 December 2022, the week I’m writing this, the joke finally took some damage.&lt;/p&gt;
&lt;p&gt;The Department of Energy announced that the National Ignition Facility at Lawrence Livermore had achieved ignition: a controlled fusion reaction releasing more energy than the laser energy delivered to the target. Decades of near-misses. Then one actual net-positive shot.&lt;/p&gt;
&lt;p&gt;Why does this matter so much? Because fusion is the energy source underneath everything. It’s what powers stars. Light nuclei forced together. Mass converted to energy. No carbon, no long-lived waste, fuel effectively everywhere. And because energy scarcity sits under a depressing share of modern conflict, a working fusion economy would remove one of the oldest reasons nations fight.&lt;/p&gt;
&lt;p&gt;The uncomfortable footnote is that humanity has produced fusion at scale since 1952, in thermonuclear weapons, starting with the first detonation on 1 November of that year. Uncontrolled fusion, we mastered quickly. It’s the &lt;em&gt;controlled&lt;/em&gt; version, holding a plasma at stellar temperatures steadily enough to farm energy from it, that has consumed seventy years. The whole difficulty of the field lives in that one word.&lt;/p&gt;
&lt;p&gt;Control problems, though, are exactly where machine learning has begun to earn a place. A plasma is a writhing, unstable thing. Hundreds of sensors watch it. Magnetic fields steer it on millisecond timescales, far too fast and too high-dimensional for a human in the loop. At the Joint European Torus, ML models have been used to optimise plasma confinement, learning from experimental shots which control settings hold the plasma stable. ITER, the 35-nation reactor under construction in France, is planning on the same family of tools at greater scale.&lt;/p&gt;
&lt;p tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Figure&quot;&gt;&lt;img src=&quot;https://analyticadss.com/img/posts/fusion/control-timescales.svg&quot; alt=&quot;A logarithmic time axis. Plasma instabilities grow, and the magnetic field has to answer, at around one millisecond. A human notices at roughly a quarter of a second. The band between them is marked as where everything that matters happens.&quot; width=&quot;760&quot; height=&quot;428&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot;/&gt;&lt;/p&gt;
&lt;p&gt;Put the two timescales on one axis and the argument makes itself. There is no version of this where a person sits in the loop. The only question is what does.&lt;/p&gt;
&lt;p&gt;Looking forward from this week’s news, the openings for AI in fusion are easy to name. Simulation surrogates: full plasma-physics simulations are brutally expensive, and learned models that approximate them well would let researchers explore designs orders of magnitude faster. Real-time control and optimisation: steering temperature, pressure and confinement continuously, and reacting to instabilities before they grow. And safety: anomaly detection over the sensor streams, flagging conditions no operator would catch in time.&lt;/p&gt;
&lt;p&gt;I’d add the honest caveat that ignition at NIF is a milestone, not a power plant. The energy accounting counts laser light delivered, not the electricity that produced it, and the road from one igniting pellet to a grid-connected reactor is long. But for the first time in my scientific life, the ten-years joke has a real experimental result standing against it. Physics moved. It never does that on schedule, which is exactly what makes it worth writing down when it does.&lt;/p&gt;</content:encoded><dc:creator>Dr. Aous Abdo</dc:creator><category>Physics</category><category>AI</category><category>Energy</category></item><item><title>Loops vs vectorization: the 40x you&apos;re leaving on the table</title><link>https://analyticadss.com/insights/maximizing-efficiency-with-loops-and-vectorization-in-programming-languages/</link><guid isPermaLink="true">https://analyticadss.com/insights/maximizing-efficiency-with-loops-and-vectorization-in-programming-languages/</guid><description>The same computation, written two ways, with a benchmark that isn&apos;t close. Why vectorized code wins in R, and when a loop is still right.</description><pubDate>Thu, 01 Dec 2022 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;&lt;em&gt;Written some years ago. The ideas hold, but check package versions and API details against current documentation before relying on the code.&lt;/em&gt;&lt;/p&gt;&lt;p&gt;Here’s a benchmark from my own machine: the same arithmetic on ten million numbers, once with a loop, once vectorized.&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://analyticadss.com/img/posts/loops/loop-vs-vectorization-benchmark.webp&quot; alt=&quot;Loop time grows linearly to about 10 seconds at ten million calculations; the vectorized line stays flat near zero.&quot; width=&quot;743&quot; height=&quot;426&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot;/&gt;&lt;/p&gt;
&lt;p&gt;The red line is the loop. The dashed line hugging the x-axis is vectorization doing the identical work. That’s not a tuning difference; that’s a factor of forty or more, and it’s sitting in most R scripts I get asked to look at.&lt;/p&gt;
&lt;p&gt;A loop says &lt;em&gt;do this to each element, one at a time&lt;/em&gt;. Vectorization says &lt;em&gt;do this to the whole vector&lt;/em&gt; and lets the language dispatch the iteration to compiled C underneath. Same maths, different addressee:&lt;/p&gt;
&lt;pre class=&quot;astro-code github-dark&quot; style=&quot;background-color:#24292e;color:#e1e4e8;overflow-x:auto&quot; tabindex=&quot;0&quot; data-language=&quot;r&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;x&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; runif&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;1e7&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# The loop&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;out&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; numeric&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;length&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;x&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;))&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;for&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;i&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; in&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; seq_along&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;x&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)) {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;  out&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;i&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;] &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;&amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; x&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;i&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;] &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;*&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 2&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; +&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 1&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# The vectorized version&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;out&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; x&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; *&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 2&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; +&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 1&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Why is the gap so violent in R specifically? Because R is interpreted, and a loop pays interpreter overhead — type checks, dispatch, bookkeeping — on &lt;em&gt;every element&lt;/em&gt;. Ten million elements, ten million tolls. The vectorized call pays once, then runs machine code. Python has the same disease and the same cure, spelled NumPy.&lt;/p&gt;
&lt;p&gt;Two loop pitfalls do the most damage in practice. Growing an object inside a loop (&lt;code&gt;out &amp;lt;- c(out, ...)&lt;/code&gt;) reallocates the whole vector every pass and turns linear work quadratic. If you must loop, pre-allocate, as above. And the sneaky one: hiding loops inside loops, where an innocent &lt;code&gt;apply&lt;/code&gt; call inside a &lt;code&gt;for&lt;/code&gt; multiplies the toll.&lt;/p&gt;
&lt;p&gt;So should you vectorize everything? No, and knowing where the boundary sits is the actual skill. A loop is right when iterations genuinely depend on each other: a simulation where step &lt;em&gt;t&lt;/em&gt; needs step &lt;em&gt;t−1&lt;/em&gt;, like &lt;a href=&quot;https://analyticadss.com/insights/unsolved-math-problems-the-collatz-conjecture/&quot;&gt;the Collatz sequence&lt;/a&gt;, can’t be vectorized away. It’s right when each iteration is a heavy independent job (a model fit per file, where &lt;a href=&quot;https://analyticadss.com/insights/unleash-the-power-of-functional-programming-in-r-with-the-purrr-package/&quot;&gt;purrr’s map functions&lt;/a&gt; give you loop semantics with cleaner bones). And it’s right when clarity wins: a readable loop over a thirty-second job beats a clever one-liner nobody can maintain.&lt;/p&gt;
&lt;p&gt;But the default posture matters. In vector languages, reach for the vectorized form first and justify the loop, not the other way round. When you genuinely need loop logic at compiled speed, that escape hatch exists too — it’s called Rcpp, and &lt;a href=&quot;https://analyticadss.com/insights/r-on-steroids-with-rcpp-library/&quot;&gt;it gets its own post&lt;/a&gt;.&lt;/p&gt;
&lt;p&gt;The benchmark above is one line of your time to reproduce. Run it once on your own machine. You’ll never write the naive loop again.&lt;/p&gt;</content:encoded><dc:creator>Dr. Aous Abdo</dc:creator><category>R</category><category>Python</category><category>Performance</category></item><item><title>Recommendation engines run more of your day than you think</title><link>https://analyticadss.com/insights/recommendation-engines-how-algorithms-are-affecting-our-decision-making-everyday/</link><guid isPermaLink="true">https://analyticadss.com/insights/recommendation-engines-how-algorithms-are-affecting-our-decision-making-everyday/</guid><description>Spotify, Tinder, Steam, the news you read — the same three algorithms sit under all of it. How they work, and the one we built for hotels.</description><pubDate>Thu, 01 Dec 2022 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;&lt;em&gt;Written in 2022. The specifics are of their time; the argument is what has been kept.&lt;/em&gt;&lt;/p&gt;&lt;p&gt;Count the recommendations you’ve accepted today. The playlist Spotify assembled, the video queued next, the products under “you may also like”, the articles your news app surfaced, possibly the person you’re dating. We used to choose from what friends suggested and reviews praised. Now a substantial share of daily decisions arrive pre-ranked by algorithms, and it’s worth knowing how the ranking happens.&lt;/p&gt;
&lt;p&gt;Under the hood there are really three ideas, endlessly remixed.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Collaborative filtering&lt;/strong&gt; ignores what things &lt;em&gt;are&lt;/em&gt; and watches what people &lt;em&gt;do&lt;/em&gt;: people whose history overlaps yours chose this next, so you might too. It powers the classic “customers also bought”, and its known weakness is the cold start: a new user or new item has no history to match on.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Content-based filtering&lt;/strong&gt; works from attributes instead: you liked these dark Scandinavian thrillers, here’s another with the same fingerprint. No crowd needed, but it can trap you in more-of-the-same.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Hybrids&lt;/strong&gt; blend both, which is what serious systems actually run: Spotify folding your listening history together with audio features and playlist co-occurrence; Steam mixing what you’ve played with what a game is; Tinder ranking by stated preference and by revealed swiping. Even newsrooms run them: the article you just read was probably followed by three chosen this way.&lt;/p&gt;
&lt;p&gt;I got to build one of these where the stakes were unusually concrete: commercial hotels. At &lt;a href=&quot;https://analyticadss.com/work/hotel-recommender/&quot;&gt;JLL, one of the world’s largest commercial real-estate services firms&lt;/a&gt;, a hotel coming to market meant finding the hundred likeliest buyers in a book of two hundred thousand investors, a search that took a broker up to three months by hand. The original sketch of the problem still says it best:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://analyticadss.com/img/posts/recommenders/hotels-to-buyers.webp&quot; alt=&quot;The hotel recommender problem as originally diagrammed: a set of hotel listings on the left, fanned out to the small subset of likely buyers among the many.&quot; width=&quot;828&quot; height=&quot;351&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot;/&gt;&lt;/p&gt;
&lt;p&gt;The twist that made it hard: buyers don’t rate hotels the way viewers rate films. Our interaction matrix was 99.35% empty, so the signal had to come from behaviour — enquiries, bids, stated mandates — rather than ratings. The &lt;a href=&quot;https://analyticadss.com/work/hotel-recommender/&quot;&gt;full case study&lt;/a&gt; covers the bake-off, including the baseline that embarrassed a fancier algorithm.&lt;/p&gt;
&lt;p&gt;So should you be glad these systems exist, or worried? Both, and specifically. The genuine gift is discovery at scale: the back-catalogue song, the niche game, the buyer nobody thought to call. The genuine costs are narrowing (optimising for what you did before quietly shrinks what you see next), and the gap between &lt;em&gt;engaging&lt;/em&gt; and &lt;em&gt;good for you&lt;/em&gt; — an engine tuned for clicks will happily feed any appetite that clicks.&lt;/p&gt;
&lt;p&gt;My practical advice is small but real: notice the ranking. A recommendation is a prediction about you with a business model attached, and simply remembering that, then deliberately wandering outside the feed now and then, keeps the algorithms where they belong, working for you rather than the reverse.&lt;/p&gt;</content:encoded><dc:creator>Dr. Aous Abdo</dc:creator><category>Machine learning</category><category>Recommenders</category></item><item><title>AI in healthcare: what&apos;s real</title><link>https://analyticadss.com/insights/revolutionizing-healthcare-the-role-of-artificial-intelligence-and-machine-learning/</link><guid isPermaLink="true">https://analyticadss.com/insights/revolutionizing-healthcare-the-role-of-artificial-intelligence-and-machine-learning/</guid><description>Imaging, records, drug discovery, hospital operations: where models genuinely help medicine, and the two failure modes that should scare you.</description><pubDate>Thu, 01 Dec 2022 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;&lt;em&gt;Written in 2022. The specifics are of their time; the argument is what has been kept.&lt;/em&gt;&lt;/p&gt;&lt;p&gt;Healthcare generates more data per patient than almost any human activity — scans, labs, notes, monitors, histories. For decades most of it was written once and read never. That’s the real story of AI in medicine. Not robot doctors. Just finally reading what we already record. It’s also &lt;a href=&quot;https://analyticadss.com/sectors/healthcare/&quot;&gt;work we do&lt;/a&gt;, so this is a field report as much as a survey.&lt;/p&gt;
&lt;p&gt;Where do models actually earn their keep today? Four places, in descending order of maturity.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Imaging&lt;/strong&gt; is the most proven. Radiology is pattern recognition under time pressure, and that’s the one thing deep learning does at superhuman consistency. Models now flag tumours, fractures and retinal disease at accuracy rivalling specialists, in narrow, well-defined tasks. The operative word is &lt;em&gt;flag&lt;/em&gt;: the working deployments put the model’s attention in front of a radiologist, not in place of one, catching the case that arrives at hour nine of a shift.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Records&lt;/strong&gt; are the sleeping giant. Your medical history is mostly free text, and natural-language models can finally read it at scale — surfacing risks buried in old notes, matching patients to trials, predicting deterioration early enough to act. Prediction from records is our own corner of this field, and and the hard part is never the model. It’s &lt;a href=&quot;https://analyticadss.com/sectors/healthcare/&quot;&gt;class imbalance, leakage and honest validation&lt;/a&gt;, because a model that scores well on a convenient sample and fails on the real population is worse than no model at all.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Drug discovery&lt;/strong&gt; uses models to search chemical space, proposing candidates and predicting interactions before anything touches a lab bench. Years and fortunes are spent per approved drug; even modestly better triage compounds enormously.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Operations&lt;/strong&gt; is the unglamorous one that pays today: predicting admissions, scheduling theatres, spotting billing anomalies. Nobody writes headlines about bed-allocation models. Hospitals notice the difference.&lt;/p&gt;
&lt;p tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Figure&quot;&gt;&lt;img src=&quot;https://analyticadss.com/img/posts/healthcare/where-models-earn-their-keep.svg&quot; alt=&quot;Four bars in descending order of maturity: imaging, records, drug discovery, operations. Below them, two gates every deployment has to pass — bias inherited from unequally delivered historical care, and the requirement that a clinician can interrogate the prediction.&quot; width=&quot;760&quot; height=&quot;430&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot;/&gt;&lt;/p&gt;
&lt;p&gt;The order of that list is roughly the reverse of how often each one gets pitched.&lt;/p&gt;
&lt;p&gt;Now the failure modes, because in medicine they aren’t footnotes. They’re the story.&lt;/p&gt;
&lt;p&gt;The first is bias with clinical consequences. Models learn from historical care, and historical care was not delivered equally. A famous US algorithm used spending as a proxy for need and systematically under-referred Black patients: less money had been spent on them, so the model concluded less care was needed. No one intended that. It’s what training on the past &lt;em&gt;means&lt;/em&gt;, and only deliberate auditing catches it.&lt;/p&gt;
&lt;p&gt;The second is the black-box problem, and medicine is where I’m least tolerant of it. A prediction no clinician can interrogate is a prediction no clinician should act on, whatever its accuracy score. If the model can’t show its reasons, it isn’t ready for the ward. In this field, explanation isn’t a feature request. It’s the price of admission.&lt;/p&gt;
&lt;p&gt;The near future is more of the real stuff: multimodal models reading images, records and genomics together, and earlier detection for diseases where every month matters. The realistic frame stays the same, though. The scarce resource in medicine is trained human attention, and the technology’s proper job is aiming that attention where it’s needed most. Tools, not oracles — and the systems built with that humility are, so far, the only ones that have actually made patients better off.&lt;/p&gt;</content:encoded><dc:creator>Dr. Aous Abdo</dc:creator><category>AI</category><category>Healthcare</category></item><item><title>What the new era of AI actually changes</title><link>https://analyticadss.com/insights/rise-of-the-machines-what-to-expect-in-the-new-era-of-ai/</link><guid isPermaLink="true">https://analyticadss.com/insights/rise-of-the-machines-what-to-expect-in-the-new-era-of-ai/</guid><description>Past the headlines about robot overlords: what automation genuinely does to work, who carries the cost, and what preparing would look like.</description><pubDate>Thu, 01 Dec 2022 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;&lt;em&gt;A late-2022 view, written just as generative AI went mainstream. The specifics have raced ahead; the structural questions here are still the ones that matter.&lt;/em&gt;&lt;/p&gt;&lt;p&gt;Every wave of automation triggers the same two headlines: the machines will take every job, and the machines will free us all. Both have been wrong every previous time, and the honest position on AI is that it will be wrong this time too — while something real happens in between.&lt;/p&gt;
&lt;p&gt;The real thing is displacement, and it’s worth being precise about who it lands on. AI doesn’t automate &lt;em&gt;jobs&lt;/em&gt; so much as &lt;em&gt;tasks&lt;/em&gt;, and it takes the routine ones first. The result isn’t mass unemployment overnight. It’s pressure, concentrated on people whose work is mostly routine tasks, who are disproportionately the people with the least cushion to absorb a career change. Meanwhile, new work genuinely does appear. Someone has to build, maintain, audit and improve these systems, and whole occupations exist today that didn’t when I started. The uncomfortable truth is that the people gaining the new work and the people losing the old work are mostly not the same people. That’s the inequality mechanism, and it’s more concerning than any robot-uprising scenario because it’s already running.&lt;/p&gt;
&lt;p tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Figure&quot;&gt;&lt;img src=&quot;https://analyticadss.com/img/posts/machines/tasks-not-jobs.svg&quot; alt=&quot;A job drawn as a stack of tasks, with the routine ones absorbed first, leaving the job standing with a hole in it. Below, three columns: who loses the old work, who gains the new work, and the gap between them, which is the inequality mechanism.&quot; width=&quot;760&quot; height=&quot;438&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot;/&gt;&lt;/p&gt;
&lt;p&gt;Drawn out, the thing to look at is the bottom row rather than the top. Task-level automation is the well-understood part. What decides how this actually goes is that the two groups of people are different groups.&lt;/p&gt;
&lt;p&gt;The effects that worry me most aren’t even economic. AI systems were already, in 2022, scoring risk in criminal justice and screening candidates for jobs — decisions with real consequences, made partly by models trained on data that encodes every past bias we ever practised. A hiring model trained on yesterday’s hires reproduces yesterday’s prejudices with a straight face and an objective-sounding score. Nobody has to intend that outcome. It’s the default.&lt;/p&gt;
&lt;p&gt;So what does preparing actually look like? Three things, none of them exotic.&lt;/p&gt;
&lt;p&gt;Education and retraining, funded like we mean it, aimed at the people the pressure lands on rather than the people writing think-pieces about it. The skills that hold value are the ones models complement instead of replace: judgement, cross-domain reasoning, and the ability to work &lt;em&gt;with&lt;/em&gt; these systems.&lt;/p&gt;
&lt;p&gt;Second, a policy conversation about distribution that starts before the concentration is finished, because the gains from automation flow to whoever owns the automation, and that isn’t a law of physics — it’s a choice of rules.&lt;/p&gt;
&lt;p&gt;And third, real accountability for consequential decisions. Not a voluntary ethics charter. Standards with teeth for any system that decides who gets bail, a loan, or an interview, including the right to know why.&lt;/p&gt;
&lt;p&gt;None of this is anti-AI. I build these systems for a living and I think the ceiling on what they can do for us is very high. But “what should we expect?” has a grown-up answer, and it isn’t utopia or apocalypse. It’s a powerful general technology arriving into our existing institutions, amplifying whatever we’ve already built. Including the parts we should have fixed first.&lt;/p&gt;</content:encoded><dc:creator>Dr. Aous Abdo</dc:creator><category>AI</category><category>Economics</category><category>Society</category></item><item><title>Will language models replace search?</title><link>https://analyticadss.com/insights/the-future-of-search-how-language-models-are-transforming-the-way-we-find-information/</link><guid isPermaLink="true">https://analyticadss.com/insights/the-future-of-search-how-language-models-are-transforming-the-way-we-find-information/</guid><description>Written as ChatGPT was first colliding with Google: why the answer was &apos;enhance, not replace&apos; — and what that prediction got right.</description><pubDate>Thu, 01 Dec 2022 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;&lt;em&gt;Late 2022, written in the first weeks of the ChatGPT wave. Kept as a time capsule — the &amp;quot;enhance rather than replace&amp;quot; call has held up better than most predictions from that season.&lt;/em&gt;&lt;/p&gt;&lt;p&gt;In late 2022 the question was suddenly everywhere: is this the end of Google? ChatGPT had just landed, it answered questions in full sentences, and typing keywords into a search box abruptly felt like using a rotary phone.&lt;/p&gt;
&lt;p&gt;My answer at the time was no. And since predictions in this field usually deserve a scorecard, I’m leaving this one up to be checked.&lt;/p&gt;
&lt;p&gt;The case for language models in search was obvious even then. A model that &lt;a href=&quot;https://analyticadss.com/insights/what-are-large-language-models-llms-and-how-are-they-being-used/&quot;&gt;understands language as probability distributions&lt;/a&gt; can do things keyword matching never could. Ask “what are the top attractions in Rome, and how do I reach each one from the city centre?” and a keyword engine sees a bag of words; a language model sees a two-part question with an itinerary hiding in it. Ambiguity is the other win. “A good coffee shop in the city” means &lt;em&gt;your&lt;/em&gt; city, &lt;em&gt;open now&lt;/em&gt;: context a model can hold and a keyword index can’t. And the answer can come back as a sentence rather than ten blue links.&lt;/p&gt;
&lt;p&gt;So why not replacement? Because search engines do a job that generating text doesn’t touch: they index the live web, they point at sources, and they’re accountable to them. A search result says &lt;em&gt;here’s where this came from&lt;/em&gt;; a generated answer says &lt;em&gt;trust me&lt;/em&gt;. A model’s knowledge also freezes at training time, while the web changes hourly. And fluent doesn’t mean correct. A language model will state a wrong answer with exactly the confidence of a right one, which is fatal in a tool whose one job is finding the truth.&lt;/p&gt;
&lt;p&gt;What I expected instead was a merger: models as the understanding-and-presentation layer wrapped around retrieval, with the index still underneath doing the honest work of knowing where facts live.&lt;/p&gt;
&lt;p&gt;Reading it back, that’s roughly what happened. The assistants learned to search and cite; the search engines grew generated summaries with links hanging off them. The architecture even got a name, retrieval-augmented generation, and it’s precisely the marriage described above: fetch real documents first, then let the model answer &lt;em&gt;from them&lt;/em&gt;, on the record.&lt;/p&gt;
&lt;p tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Figure&quot;&gt;&lt;img src=&quot;https://analyticadss.com/img/posts/search/why-search-merged.svg&quot; alt=&quot;Three columns checked against three properties the job requires: freshness, pointing at a source, and understanding the question. A keyword index is fresh and cites its sources but does not understand the question. A language model alone understands but is frozen at training time and cites nothing. Retrieval-augmented generation satisfies all three.&quot; width=&quot;760&quot; height=&quot;408&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot;/&gt;&lt;/p&gt;
&lt;p&gt;Laid out as a scorecard, the prediction stops looking clever. Two of those columns each fail a property the job cannot do without, and there was only ever one way to get all three.&lt;/p&gt;
&lt;p&gt;The lesson I keep from this post isn’t that the prediction aged well. It’s &lt;em&gt;why&lt;/em&gt; it did. The reasoning wasn’t about which company would win; it was about which properties the job requires. Finding information requires freshness and verifiability. Any technology missing those wasn’t going to replace the one that has them. It was going to get bolted onto it. Reason from what the job needs and your predictions survive contact with the future far more often than reasoning from the demo.&lt;/p&gt;</content:encoded><dc:creator>Dr. Aous Abdo</dc:creator><category>AI</category><category>Search</category></item><item><title>The Collatz conjecture: two rules, no exit</title><link>https://analyticadss.com/insights/unsolved-math-problems-the-collatz-conjecture/</link><guid isPermaLink="true">https://analyticadss.com/insights/unsolved-math-problems-the-collatz-conjecture/</guid><description>Halve it if even, 3n + 1 if odd. Every number tested past 2⁶⁸ reaches 1, yet there&apos;s no proof. Why it&apos;s so hard, Tao&apos;s 2019 result, and R code to try it.</description><pubDate>Thu, 01 Dec 2022 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;&lt;em&gt;Written some years ago. The ideas hold, but check package versions and API details against current documentation before relying on the code.&lt;/em&gt;&lt;/p&gt;&lt;p&gt;Take any positive whole number. If it’s even, halve it. If it’s odd, multiply by three and add one. Repeat.&lt;/p&gt;
&lt;p&gt;Start with 5 and you get: 5 → 16 → 8 → 4 → 2 → 1. Start with 27 and you’re in for a ride — 111 steps, climbing as high as 9,232 before collapsing. But collapse it does, and that’s the conjecture Lothar Collatz posed in 1937: no matter where you start, you always reach 1.&lt;/p&gt;
&lt;p&gt;Always? Every number ever tested reaches 1, and the testing has gone past 2⁶⁸. No proof exists. Paul Erdős famously said mathematics isn’t ready for such problems, and offered $500 for a solution — pocket change against the &lt;a href=&quot;https://analyticadss.com/insights/riemann-hypothesis-a-mathematical-exploration/&quot;&gt;million on Riemann&lt;/a&gt;, yet the problem has eaten just as many careers.&lt;/p&gt;
&lt;h2 id=&quot;why-so-hard-when-the-rule-is-so-small&quot;&gt;Why so hard, when the rule is so small?&lt;/h2&gt;
&lt;p&gt;Because the sequence has no respect for size or pattern. Whether you rise or fall next depends only on parity, and parity after tripling-plus-one is chaotic in practice. Neighbouring starting points produce wildly different journeys, which you can see directly:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://analyticadss.com/img/posts/collatz/trajectory-lengths.webp&quot; alt=&quot;Total stopping time for each starting value: a jagged, self-similar skyline with no obvious order.&quot; width=&quot;828&quot; height=&quot;507&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot;/&gt;&lt;/p&gt;
&lt;p&gt;That plot is the length of the sequence for each starting number. 26 takes ten steps; 27 takes 111. There’s structure in there: bands, echoes, a strange self-similarity. But nothing anyone has turned into a proof. The serious result we do have is Terence Tao’s from 2019: &lt;em&gt;almost all&lt;/em&gt; starting values eventually get almost as small as you like. “Almost all” is doing heavy lifting there. It’s the strongest statement anyone has managed, and it still doesn’t rule out one immortal orbit somewhere, spinning forever.&lt;/p&gt;
&lt;h2 id=&quot;run-it-yourself&quot;&gt;Run it yourself&lt;/h2&gt;
&lt;p&gt;The whole thing fits in a dozen lines of R:&lt;/p&gt;
&lt;pre class=&quot;astro-code github-dark&quot; style=&quot;background-color:#24292e;color:#e1e4e8;overflow-x:auto&quot; tabindex=&quot;0&quot; data-language=&quot;r&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;collatz&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; function&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;n&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;) {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;  steps&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; n&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;  while&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;n&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; !=&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 1&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;) {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;    n&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; if&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;n&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; %%&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 2&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; ==&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 0&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;) &lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;n&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; /&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 2&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; else&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 3&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; *&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; n&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; +&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 1&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;    steps&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; c&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;steps&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; n&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;  }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;  steps&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;}&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;collatz&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;27&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# 27 82 41 124 62 31 94 47 142 71 214 107 ... 4 2 1&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Trajectory lengths for the first 10,000 starting values&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;lengths&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; sapply&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;:&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;10000&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; function&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;n&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;) &lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;length&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;collatz&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;n&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)))&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;plot&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;lengths&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; type&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt; &amp;quot;h&amp;quot;&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; xlab&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt; &amp;quot;Starting value n&amp;quot;&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;     ylab&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt; &amp;quot;Length of Collatz sequence&amp;quot;&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Change the rule slightly, to 3n − 1 instead of 3n + 1, and you get cycles that never reach 1. So the conjecture is delicately balanced on its exact form, which somehow makes it worse.&lt;/p&gt;
&lt;p&gt;Why do I keep a soft spot for this one, alongside &lt;a href=&quot;https://analyticadss.com/insights/unsolved-math-problems-the-goldbach-conjecture/&quot;&gt;Goldbach&lt;/a&gt;? Because it’s the purest example I know of a lesson our whole field runs on: a system’s rules being simple tells you nothing about its behaviour being simple. Four lines of code, eighty years of failure. I show this to people who think complexity requires complication, and I watch it land every time.&lt;/p&gt;</content:encoded><dc:creator>Dr. Aous Abdo</dc:creator><category>Mathematics</category><category>R</category><category>Number theory</category></item><item><title>The Goldbach conjecture, plainly</title><link>https://analyticadss.com/insights/unsolved-math-problems-the-goldbach-conjecture/</link><guid isPermaLink="true">https://analyticadss.com/insights/unsolved-math-problems-the-goldbach-conjecture/</guid><description>Is every even number above 2 the sum of two primes? Checked to 4 × 10¹⁸ with no exception, unproven since 1742. Why it&apos;s so hard, in plain language.</description><pubDate>Thu, 01 Dec 2022 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;&lt;em&gt;Written in 2022. The conjecture is still open.&lt;/em&gt;&lt;/p&gt;&lt;p&gt;Is every even number greater than 2 the sum of two primes? Every one checked so far is, up to about 4 × 10¹⁸, but nobody has proved it holds for all of them. It stays unsolved because primes are defined by multiplication while the conjecture is about addition, and because a proof has to cover infinitely many numbers that no computer can reach.&lt;/p&gt;
&lt;p&gt;Pick an even number bigger than 2. Any one. Now try to write it as the sum of two primes.&lt;/p&gt;
&lt;p&gt;4 = 2 + 2. 6 = 3 + 3. 8 = 3 + 5. 100 = 3 + 97. It works every single time, as far as anyone has ever checked, and nobody on Earth can prove it always will. That’s the Goldbach conjecture, and it has been sitting there unsolved since 1742, when Christian Goldbach floated it in a letter to Leonhard Euler.&lt;/p&gt;
&lt;p tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Figure&quot;&gt;&lt;img src=&quot;https://analyticadss.com/img/posts/goldbach/goldbach-comet.svg&quot; alt=&quot;Every even number from 4 to 10,000 plotted against the number of ways it can be written as two primes. The cloud fans steadily upward and never touches zero — the fewest representations anywhere in the range is one, at n = 4. Colouring by n modulo 6 splits it into three arms, with multiples of 6 sitting well above the rest.&quot; width=&quot;760&quot; height=&quot;430&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot;/&gt;&lt;/p&gt;
&lt;p&gt;That shape is the trap, drawn. Not only does the count never hit zero, it climbs — so the bigger the number, the more ways there are for it to work, and the safer the conjecture looks. Feeling safe and being proved are different things, and the picture cannot tell you which one you are looking at.&lt;/p&gt;
&lt;p&gt;I love this problem for the same reason I love the &lt;a href=&quot;https://analyticadss.com/insights/unsolved-math-problems-the-collatz-conjecture/&quot;&gt;Collatz conjecture&lt;/a&gt;: you can explain it to a ten-year-old in one minute, and then you can spend a career failing to prove it. That gap between how easy a thing is to state and how hard it is to settle tells you something true about mathematics.&lt;/p&gt;
&lt;h2 id=&quot;how-far-has-checking-got-us&quot;&gt;How far has checking got us?&lt;/h2&gt;
&lt;p&gt;Computers have verified the conjecture for every even number up to about 4 × 10¹⁸. Four quintillion. The big systematic push was Tomás Oliveira e Silva’s distributed computation, which ground through that entire range and found no exception.&lt;/p&gt;
&lt;p&gt;So is it true? Almost certainly. Is it proven? Not even close, and here’s the thing people outside mathematics find strange: those two statements can both hold at once. Checking four quintillion cases proves exactly four quintillion cases. The conjecture makes a claim about &lt;em&gt;all&lt;/em&gt; even numbers, and there are infinitely many left after any check you can run. One counterexample hiding at 10¹⁰⁰ would sink the whole thing, and no computer will ever reach it by brute force.&lt;/p&gt;
&lt;h2 id=&quot;why-is-it-so-hard&quot;&gt;Why is it so hard?&lt;/h2&gt;
&lt;p&gt;Two reasons, and they compound each other.&lt;/p&gt;
&lt;p&gt;Primes are multiplicative objects. They’re defined by what divides them. But Goldbach is a claim about &lt;em&gt;addition&lt;/em&gt; — about which numbers you can build by summing primes. Mathematics has spectacular machinery for the multiplicative side and much weaker machinery for questions that mix the two. The primes are individually easy to find and collectively very strange, and their gaps and clusters don’t follow a formula anyone has tamed.&lt;/p&gt;
&lt;p&gt;The second reason is the infinity problem I mentioned. A proof has to work for every even number at once, which means finding structure, some reason the sums must always land, rather than confirming instances. Nobody has found that reason in 280 years.&lt;/p&gt;
&lt;p&gt;Progress has come at the edges, and some of it is genuinely beautiful. Analytic number theory, the toolkit descending from Riemann’s zeta function, gave us results like Vinogradov’s theorem, which proves that every sufficiently large &lt;em&gt;odd&lt;/em&gt; number is the sum of &lt;em&gt;three&lt;/em&gt; primes. Harald Helfgott completed that “weak Goldbach” result for all odd numbers above 5 in 2013. The strong version, two primes for every even number, still stands open.&lt;/p&gt;
&lt;h2 id=&quot;why-i-keep-coming-back-to-it&quot;&gt;Why I keep coming back to it&lt;/h2&gt;
&lt;p&gt;Because it’s a working lesson in what evidence is and isn’t. Four quintillion confirmations feel like certainty, and in any applied setting we’d have shipped long ago. Mathematics refuses to, and that refusal is the discipline’s whole character. In our modelling work I think about this more than you’d expect: a pattern that has held every time you’ve looked is not the same thing as a pattern that must hold, and the day someone bets real money on the difference is the day it matters.&lt;/p&gt;
&lt;p&gt;Try a few even numbers yourself. It never fails. Nobody knows why.&lt;/p&gt;</content:encoded><dc:creator>Dr. Aous Abdo</dc:creator><category>Mathematics</category><category>Number theory</category></item><item><title>What a language model actually is</title><link>https://analyticadss.com/insights/what-are-large-language-models-llms-and-how-are-they-being-used/</link><guid isPermaLink="true">https://analyticadss.com/insights/what-are-large-language-models-llms-and-how-are-they-being-used/</guid><description>Strip away the noise and an LLM is a next-word predictor of colossal size. From n-grams to GPT-3, including the tiny model I built in R.</description><pubDate>Thu, 01 Dec 2022 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;&lt;em&gt;Written in late 2022, when GPT-3&apos;s 175 billion parameters was the number everyone quoted. The scale has moved on; the fundamentals here haven&apos;t.&lt;/em&gt;&lt;/p&gt;&lt;p&gt;Underneath everything a large language model does is one deceptively small task: predict the next word.&lt;/p&gt;
&lt;p&gt;That’s it. That’s the training objective. Given “the cat sat on the”, assign probabilities to what comes next: “mat” high, “roof” lower, “carburettor” very low. The startling discovery of the last few years is how much apparent understanding falls out of doing that one task extremely well at extreme scale.&lt;/p&gt;
&lt;p&gt;My old whiteboard sketch of the objective survives, and it still explains it faster than a paragraph:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://analyticadss.com/img/posts/llms/next-word-prediction.webp&quot; alt=&quot;Hand-drawn diagram: the sentence &amp;quot;S = Where are we going&amp;quot;, with the earlier words bracketed as context and the last word marked as the one being predicted, above the chain-rule expansion of the sentence probability.&quot; width=&quot;795&quot; height=&quot;281&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot;/&gt;&lt;/p&gt;
&lt;h2 id=&quot;the-old-way-and-my-own-small-museum-piece&quot;&gt;The old way, and my own small museum piece&lt;/h2&gt;
&lt;p&gt;Language modelling long predates the current excitement. The classical approach was statistical: n-gram models, which estimate the next word from counts of the previous few. Simple, fast, and hopeless at anything requiring long-range memory. By the time an n-gram model finishes a sentence, it has forgotten how the sentence began.&lt;/p&gt;
&lt;p&gt;I have a personal exhibit here. Over a decade ago I built a small word-prediction model in R and put it on the web, and &lt;a href=&quot;https://analytica.shinyapps.io/NLPWordPrediction/&quot; rel=&quot;noopener noreferrer&quot; target=&quot;_blank&quot;&gt;it’s still running&lt;/a&gt;. Type a phrase and it guesses your next word from n-gram statistics. Playing with it after using a modern model is a useful kind of time travel: you can feel exactly what scale and architecture bought us.&lt;/p&gt;
&lt;h2 id=&quot;what-changed&quot;&gt;What changed&lt;/h2&gt;
&lt;p&gt;Neural language models replaced counting with learned representations: words became vectors, and the model learned relationships between them. Then transformers arrived and removed the memory bottleneck, letting the model attend to everything in the context at once. Three names from this era are worth knowing: GPT-3, which showed that 175 billion parameters of pure next-word prediction could write, translate and answer questions it was never explicitly taught; BERT, Google’s model that reads in both directions and quietly improved search; and ELMo before both, which at about 94 million parameters proved contextual embeddings mattered. The parameter counts are the “knobs” the training process tunes — more of them means more capacity, more cost, and more data required to fill them.&lt;/p&gt;
&lt;p&gt;How do you measure such a thing? The standard metric is perplexity, which is roughly how surprised the model is by text it hasn’t seen. Lower is better. It’s a narrow ruler for what these systems became, and the field has been arguing about better ones ever since.&lt;/p&gt;
&lt;h2 id=&quot;where-they-were-already-showing-up&quot;&gt;Where they were already showing up&lt;/h2&gt;
&lt;p&gt;Even in 2022 the applications were everywhere once you knew to look: speech recognition, machine translation, autocomplete, chatbots, the ranking layer inside search engines. The pattern was consistent. Anywhere language needed turning into probabilities, a language model was doing the turning.&lt;/p&gt;
&lt;p&gt;What I’d tell anyone trying to build intuition: don’t start from the demos, start from the objective. A model that predicts next words has learned, as a side effect, an enormous amount about how the world tends to be described. That’s genuinely powerful. It’s also exactly why these systems state falsehoods with perfect fluency: plausible continuation was the goal, and truth only correlates with it. Keep that one sentence and most of what these models do, and do wrong, stops being mysterious.&lt;/p&gt;</content:encoded><dc:creator>Dr. Aous Abdo</dc:creator><category>AI</category><category>NLP</category></item><item><title>Reinforcement learning, in plain terms</title><link>https://analyticadss.com/insights/what-is-reinforcement-learning/</link><guid isPermaLink="true">https://analyticadss.com/insights/what-is-reinforcement-learning/</guid><description>The third kind of machine learning: no labels, no clusters — just an agent, its actions, and the consequences. Where it shines and where it bites.</description><pubDate>Thu, 01 Dec 2022 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;&lt;em&gt;Written some years ago. The ideas hold, but check package versions and API details against current documentation before relying on the code.&lt;/em&gt;&lt;/p&gt;&lt;p&gt;Most machine learning is taught as two flavours: supervised, where you learn from labelled examples, and unsupervised, where you find structure without labels. Reinforcement learning is the third flavour, and it’s the strangest of the family: nobody tells the system the right answer, ever. It acts, the world responds, and it learns from the consequences.&lt;/p&gt;
&lt;p&gt;The vocabulary is small. An &lt;em&gt;agent&lt;/em&gt; takes &lt;em&gt;actions&lt;/em&gt; in an &lt;em&gt;environment&lt;/em&gt; and receives &lt;em&gt;rewards&lt;/em&gt;, positive or negative. Its job is a &lt;em&gt;policy&lt;/em&gt;: a way of choosing actions that maximises reward over time, not just on the next move. That last clause is the entire depth of the field. A chess sacrifice looks terrible by immediate reward and wins the game eight moves later, and connecting outcomes to the decisions that caused them — the credit-assignment problem — is what makes RL genuinely hard.&lt;/p&gt;
&lt;p&gt;The classic mental picture is a robot in a maze: bump a wall, small penalty; reach the exit, big reward. Thousands of episodes later, it has a route nobody programmed. And the same loop, scaled up, produced the era’s most famous result: a Go-playing system trained substantially through self-play, learning strategies from consequences alone that had never occurred to human masters in two thousand years of play.&lt;/p&gt;
&lt;p&gt;That is easy to say and more convincing to watch, so here is one I actually trained. Tabular Q-learning, four thousand episodes, minus one point for every step until it reaches the exit:&lt;/p&gt;
&lt;p tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Figure&quot;&gt;&lt;img src=&quot;https://analyticadss.com/img/posts/rl/q-learning-maze.svg&quot; alt=&quot;Two panels. The learning curve falls from 408 steps in the first episode, through noisy improvement, to a flat line near 20. The maze alongside shows the route the trained policy takes, which is 20 steps — the same length as the true shortest path.&quot; width=&quot;760&quot; height=&quot;372&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot;/&gt;&lt;/p&gt;
&lt;p&gt;The first episode took 408 steps of near-random flailing. The finished policy takes 20, which is exactly the shortest path through that maze. Nothing in the reward mentioned walls, corners or distance. It only ever said &lt;em&gt;not yet&lt;/em&gt;.&lt;/p&gt;
&lt;p&gt;Where does it show up outside the highlight reel? A trading algorithm learns from its own profits and losses, which is why I called it reinforcement learning when it appeared in &lt;a href=&quot;https://analyticadss.com/insights/what-is-the-difference-between-ai-and-ml-anyway/&quot;&gt;my AI-versus-ML tour&lt;/a&gt;. Robotic arms learn grasps through reward for success. Recommendation and assistant systems fold user reactions back into behaviour. The pattern to spot: wherever a system’s own actions change what it sees next, you’ve left supervised learning’s world and entered this one.&lt;/p&gt;
&lt;p&gt;If you want to touch it, the traditional playground is OpenAI’s Gym library and its little MountainCar problem, an underpowered car that must learn to rock backwards to build momentum. The loop, stripped to pseudocode, is honestly this small:&lt;/p&gt;
&lt;pre class=&quot;astro-code github-dark&quot; style=&quot;background-color:#24292e;color:#e1e4e8;overflow-x:auto&quot; tabindex=&quot;0&quot; data-language=&quot;text&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span&gt;for each episode:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;    state = env.reset()&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;    while not done:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;        action = agent.choose(state)          # explore vs exploit&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;        state, reward, done = env.step(action)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;        agent.update(state, action, reward)   # learn from consequence&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The line that hides the dragons is &lt;code&gt;choose&lt;/code&gt;. Always exploit what you know and you never discover the better strategy; explore too much and you never cash in. Balancing that trade-off is the field’s permanent tension.&lt;/p&gt;
&lt;p&gt;One warning from practice, because it’s the part the textbooks underplay: RL systems are champion loophole-finders. Specify the reward even slightly wrong and the agent optimises &lt;em&gt;exactly what you wrote&lt;/em&gt; rather than what you meant — racing in circles collecting points instead of finishing the race, in one famous demo. It’s &lt;a href=&quot;https://analyticadss.com/insights/the-ethical-dangers-of-ai-how-uncertainty-can-help-us-avoid-unforeseen-consequences/&quot;&gt;the objective problem&lt;/a&gt; in miniature, and it’s why my first question about any RL result is never “how big is the reward?” It’s “what, precisely, was rewarded?”&lt;/p&gt;</content:encoded><dc:creator>Dr. Aous Abdo</dc:creator><category>Machine learning</category><category>AI</category></item><item><title>AI vs ML: what&apos;s the actual difference?</title><link>https://analyticadss.com/insights/what-is-the-difference-between-ai-and-ml-anyway/</link><guid isPermaLink="true">https://analyticadss.com/insights/what-is-the-difference-between-ai-and-ml-anyway/</guid><description>The two terms get used interchangeably and they shouldn&apos;t be. The distinction, with examples you already use every day.</description><pubDate>Thu, 01 Dec 2022 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;&lt;em&gt;Written in 2022. The specifics are of their time; the argument is what has been kept.&lt;/em&gt;&lt;/p&gt;&lt;p&gt;People use “AI” and “machine learning” as if they’re the same word. I understand why; the marketing departments of the world have worked hard to blur them. They aren’t the same. The difference takes two minutes, and it’s worth every one of them.&lt;/p&gt;
&lt;p&gt;Here it is. AI is the ambition; ML is a method. Artificial intelligence is the broad goal of machines doing things that would need intelligence from us: reasoning, planning, understanding language, making decisions. Machine learning is one specific way of getting there — you don’t program the behaviour in, you let the system learn it from data.&lt;/p&gt;
&lt;p&gt;That means every ML system is doing AI, but not everything under the AI umbrella is ML. Consider a chess program built entirely on hand-written search and evaluation rules, the kind people were writing decades before anyone said “deep learning”. It decides. It plans. Nobody taught it from examples, though. Someone coded the judgement in directly, which makes it AI with no learning in it anywhere.&lt;/p&gt;
&lt;p&gt;The cleanest way I know to feel the difference is a pair of examples.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;AI without ML:&lt;/strong&gt; that classic chess engine. It analyses the position, considers moves, picks the best one. All of that logic came from a programmer’s head.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;ML doing AI:&lt;/strong&gt; a program that recognises cats in photos. Nobody wrote down what a cat looks like. Nobody could. The system saw a hundred thousand labelled images and worked out the pattern itself, which is supervised learning, the workhorse of the field.&lt;/p&gt;
&lt;p tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Figure&quot;&gt;&lt;img src=&quot;https://analyticadss.com/img/posts/aivsml/rules-written-or-learned.svg&quot; alt=&quot;Two pipelines. The first runs from a programmer, to working out the rules, to a chess engine: AI with no learning in it. The second runs from a hundred thousand labelled photographs, to the system finding the pattern itself, to a cat recogniser.&quot; width=&quot;760&quot; height=&quot;396&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot;/&gt;&lt;/p&gt;
&lt;p&gt;Both rows are artificial intelligence. The only thing that changes between them is where the rules came from.&lt;/p&gt;
&lt;p&gt;Once you have the lens, you can sort the products you use every day.&lt;/p&gt;
&lt;p&gt;Your voice assistant is AI in the broad sense: it parses language, decides what you meant, acts. Learning shows up inside it in specific places, like adapting to your accent. A self-driving car is the same shape: a big AI system, with learned components doing the seeing.&lt;/p&gt;
&lt;p&gt;Netflix recommendations are ML through and through: learn from what millions of people watched, predict what you will. Spam filtering? Same story. A model trained on mail people flagged. A trading algorithm learning from its own outcomes is the third flavour, reinforcement learning, where the feedback is consequences rather than labels.&lt;/p&gt;
&lt;p&gt;And fraud detection is where the two visibly work together: hand-written rules catch the known patterns, while learned models hunt the anomalies nobody has written a rule for yet.&lt;/p&gt;
&lt;p&gt;Does the distinction matter outside of pedantry? In our work, constantly. When a client asks for “AI”, the first real question is whether the problem needs learning at all. Sometimes a dozen honest rules beat a model, cost nothing to explain, and never drift. Sometimes only a model will do, because nobody can write the pattern down. Knowing which situation you’re in is most of the job. It starts with keeping these two words apart.&lt;/p&gt;</content:encoded><dc:creator>Dr. Aous Abdo</dc:creator><category>AI</category><category>Machine learning</category></item><item><title>tidyverse and data.table, and when we reach for each</title><link>https://analyticadss.com/insights/the-tidyverse-and-data-table-r-packages/</link><guid isPermaLink="true">https://analyticadss.com/insights/the-tidyverse-and-data-table-r-packages/</guid><description>dplyr and data.table side by side on the same tasks, a measured group-by benchmark from 100,000 to 5 million rows, and a simple rule for choosing.</description><pubDate>Sun, 14 Feb 2021 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;&lt;em&gt;Written some years ago. The ideas hold, but check package versions and API details against current documentation before relying on the code.&lt;/em&gt;&lt;/p&gt;&lt;p&gt;Two R packages solve the same problem in opposite ways, and which one you should reach for depends on something most comparisons skip: how big the data is, and who else has to read the code.&lt;/p&gt;
&lt;p&gt;The &lt;code&gt;tidyverse&lt;/code&gt; optimises for the reader. &lt;code&gt;data.table&lt;/code&gt; optimises for the machine. Both are excellent. I use both, usually in the same project, and the choice is rarely close once you know what you’re optimising for.&lt;/p&gt;
&lt;p&gt;The &lt;code&gt;tidyverse&lt;/code&gt; is a family of packages, &lt;code&gt;dplyr&lt;/code&gt;, &lt;code&gt;tidyr&lt;/code&gt;, &lt;code&gt;ggplot2&lt;/code&gt; and others, built around a shared grammar. Verbs read left to right, joined by a pipe, and someone who has never seen the code can usually tell what a chain does on first reading. That’s not a small thing when the person maintaining your analysis in a year is not you.&lt;/p&gt;
&lt;p&gt;&lt;code&gt;data.table&lt;/code&gt; trades that legibility for speed and memory. Its syntax is dense, it modifies in place rather than copying, and on a few million rows the difference stops being academic.&lt;/p&gt;
&lt;p&gt;Same operations in both, below.&lt;/p&gt;
&lt;hr/&gt;
&lt;h2 id=&quot;examples-tidyverse-examples&quot;&gt;Examples: Tidyverse Examples&lt;/h2&gt;
&lt;p&gt;Here are some examples of how to use the &lt;code&gt;tidyverse&lt;/code&gt;:&lt;/p&gt;
&lt;p&gt;To select specific columns from a dataset:&lt;/p&gt;
&lt;pre class=&quot;astro-code github-dark&quot; style=&quot;background-color:#24292e;color:#e1e4e8;overflow-x:auto&quot; tabindex=&quot;0&quot; data-language=&quot;r&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Load the tidyverse package&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;library&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;tidyverse&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Load the mpg dataset from the ggplot2 package&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;data&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mpg&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Select the &amp;quot;manufacturer&amp;quot; and &amp;quot;model&amp;quot; columns&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mpg&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; %&amp;gt;%&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; select&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;manufacturer&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; model&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;And to group and summarize a dataset:&lt;/p&gt;
&lt;pre class=&quot;astro-code github-dark&quot; style=&quot;background-color:#24292e;color:#e1e4e8;overflow-x:auto&quot; tabindex=&quot;0&quot; data-language=&quot;r&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Load the tidyverse package&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;library&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;tidyverse&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Load the mpg dataset from the ggplot2 package&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;data&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mpg&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Group the dataset by &amp;quot;class&amp;quot; and compute the mean of the &amp;quot;hwy&amp;quot; column&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mpg&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; %&amp;gt;%&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; group_by&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;class&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;) &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;%&amp;gt;%&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; summarize&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mean_hwy&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; mean&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;hwy&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;))&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;To join two datasets:&lt;/p&gt;
&lt;pre class=&quot;astro-code github-dark&quot; style=&quot;background-color:#24292e;color:#e1e4e8;overflow-x:auto&quot; tabindex=&quot;0&quot; data-language=&quot;r&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Load the tidyverse package&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;library&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;tidyverse&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Load the mpg and cylinders datasets from the ggplot2 package&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;data&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mpg&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;data&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;cylinders&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Join the mpg and cylinders datasets on the &amp;quot;manufacturer&amp;quot; column&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mpg&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; %&amp;gt;%&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; left_join&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;cylinders&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; by&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt; &amp;quot;manufacturer&amp;quot;&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;To perform a linear regression using the &lt;code&gt;lm&lt;/code&gt; function from the &lt;code&gt;stats&lt;/code&gt; package:&lt;/p&gt;
&lt;pre class=&quot;astro-code github-dark&quot; style=&quot;background-color:#24292e;color:#e1e4e8;overflow-x:auto&quot; tabindex=&quot;0&quot; data-language=&quot;r&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Load the tidyverse and stats packages&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;library&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;tidyverse&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;library&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;stats&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Load the mtcars dataset&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;data&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mtcars&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Perform a linear regression to predict mpg (miles per gallon) using wt (weight) as the predictor variable&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;fit&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; mtcars&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; %&amp;gt;%&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt; &lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;  lm&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mpg&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; ~&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; wt&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; data&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; .&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Summarize the model results&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;summary&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;fit&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Create a scatterplot matrix using the &lt;code&gt;scatterplotMatrix&lt;/code&gt; function from the &lt;code&gt;car&lt;/code&gt; package:&lt;/p&gt;
&lt;pre class=&quot;astro-code github-dark&quot; style=&quot;background-color:#24292e;color:#e1e4e8;overflow-x:auto&quot; tabindex=&quot;0&quot; data-language=&quot;r&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Load the tidyverse and car packages&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;library&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;tidyverse&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;library&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;car&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Load the iris dataset&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;data&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;iris&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Create a scatterplot matrix of the iris dataset&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;scatterplotMatrix&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;iris&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; smooth&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; FALSE&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Create a faceted bar plot using &lt;code&gt;ggplot2&lt;/code&gt;:&lt;/p&gt;
&lt;pre class=&quot;astro-code github-dark&quot; style=&quot;background-color:#24292e;color:#e1e4e8;overflow-x:auto&quot; tabindex=&quot;0&quot; data-language=&quot;r&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Load the tidyverse package&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;library&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;tidyverse&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Load the mpg dataset from the ggplot2 package&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;data&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mpg&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Create a faceted bar plot showing the distribution of hwy (highway miles per gallon) by class and drv (drive type)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;ggplot&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mpg&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; aes&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;x&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; hwy&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)) &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;+&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;  geom_histogram&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;binwidth&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 2&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;) &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;+&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;  facet_wrap&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;~&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; class&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; +&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; drv&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; nrow&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 2&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;hr/&gt;
&lt;h2 id=&quot;examples-datatable-examples&quot;&gt;Examples: data.table Examples&lt;/h2&gt;
&lt;p&gt;The &lt;code&gt;data.table&lt;/code&gt; package, on the other hand, is a high-performance package for working with large datasets. It provides functions for manipulating and querying data efficiently. The &lt;code&gt;data.table&lt;/code&gt; package is particularly useful when working with datasets that are too large to fit in memory or when you need to perform complex operations on large datasets.&lt;/p&gt;
&lt;h3 id=&quot;one-of-the-main-advantages-of-the-datatable-package&quot;&gt;One of the main advantages of the &lt;code&gt;data.table&lt;/code&gt; package&lt;/h3&gt;
&lt;p&gt;One of the main advantages of the &lt;code&gt;data.table&lt;/code&gt; package is its speed. The functions in the &lt;code&gt;data.table&lt;/code&gt; package are generally faster than their counterparts in the &lt;code&gt;tidyverse&lt;/code&gt;, especially when working with large datasets.&lt;/p&gt;
&lt;p&gt;So I measured it rather than repeating it. Group by a letter, average a column, three sizes:&lt;/p&gt;
&lt;p tabindex=&quot;0&quot; role=&quot;region&quot; aria-label=&quot;Figure&quot;&gt;&lt;img src=&quot;https://analyticadss.com/img/posts/tidyverse/groupby-measured.svg&quot; alt=&quot;Grouped bars comparing median run time for dplyr against data.table. At 100,000 rows it is 3 milliseconds against 2, a 1.5-fold gap. At 1 million rows, 21 against 13. At 5 million rows, 95 against 39 — a 2.4-fold gap.&quot; width=&quot;760&quot; height=&quot;380&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot;/&gt;&lt;/p&gt;
&lt;p&gt;Both are fast, and that is the part the benchmark wars tend to skip. On a hundred thousand rows the difference is about a millisecond, so pick whichever you write more fluently. The gap only earns an opinion once the data stops being small, and then it widens rather than holding steady.&lt;/p&gt;
&lt;p&gt;Here are some more examples of how to use the &lt;code&gt;data.table&lt;/code&gt; package:&lt;/p&gt;
&lt;p&gt;To select specific columns from a dataset:&lt;/p&gt;
&lt;pre class=&quot;astro-code github-dark&quot; style=&quot;background-color:#24292e;color:#e1e4e8;overflow-x:auto&quot; tabindex=&quot;0&quot; data-language=&quot;r&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Load the data.table package&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;library&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;data.table&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Load the mpg dataset from the ggplot2 package&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;data&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mpg&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Convert the dataset to a data.table&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mpg&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; as.data.table&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mpg&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Select the &amp;quot;manufacturer&amp;quot; and &amp;quot;model&amp;quot; columns&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mpg&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; .&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;manufacturer&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; model&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)]&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;and to group and summarize a dataset:&lt;/p&gt;
&lt;pre class=&quot;astro-code github-dark&quot; style=&quot;background-color:#24292e;color:#e1e4e8;overflow-x:auto&quot; tabindex=&quot;0&quot; data-language=&quot;r&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Load the data.table package&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;library&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;data.table&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Load the mpg dataset from the ggplot2 package&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;data&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mpg&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Convert the dataset to a data.table&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mpg&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; as.data.table&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mpg&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Group the dataset by &amp;quot;class&amp;quot; and compute the mean of the &amp;quot;hwy&amp;quot; column&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mpg&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; .&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mean_hwy&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; mean&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;hwy&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;))&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; by&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; class&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;]&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;To join two datasets:&lt;/p&gt;
&lt;pre class=&quot;astro-code github-dark&quot; style=&quot;background-color:#24292e;color:#e1e4e8;overflow-x:auto&quot; tabindex=&quot;0&quot; data-language=&quot;r&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Load the data.table package&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;library&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;data.table&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Load the mpg and cylinders datasets from the ggplot2 package&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;data&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mpg&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;data&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;cylinders&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Convert the datasets to data.tables&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mpg&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; as.data.table&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mpg&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;cylinders&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; as.data.table&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;cylinders&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Join the mpg and cylinders datasets on the &amp;quot;manufacturer&amp;quot; column&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mpg&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;cylinders&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; on&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt; &amp;quot;manufacturer&amp;quot;&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;]&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Perform a linear regression using the &lt;code&gt;lm&lt;/code&gt; function from the &lt;code&gt;stats&lt;/code&gt; package and the &lt;code&gt;data.table&lt;/code&gt; package:&lt;/p&gt;
&lt;pre class=&quot;astro-code github-dark&quot; style=&quot;background-color:#24292e;color:#e1e4e8;overflow-x:auto&quot; tabindex=&quot;0&quot; data-language=&quot;r&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Load the data.table and stats packages&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;library&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;data.table&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;library&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;stats&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Load the mtcars dataset&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;data&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mtcars&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Convert the dataset to a data.table&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mtcars&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; setDT&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mtcars&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Perform a linear regression to predict mpg (miles per gallon) using wt (weight) as the predictor variable&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;fit&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; mtcars&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; lm&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mpg&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; ~&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; wt&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Summarize the model results&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;summary&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;fit&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Create a scatterplot matrix using the &lt;code&gt;scatterplotMatrix&lt;/code&gt; function from the &lt;code&gt;car&lt;/code&gt; package and the &lt;code&gt;data.table&lt;/code&gt; package:&lt;/p&gt;
&lt;pre class=&quot;astro-code github-dark&quot; style=&quot;background-color:#24292e;color:#e1e4e8;overflow-x:auto&quot; tabindex=&quot;0&quot; data-language=&quot;r&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Load the data.table and car packages&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;library&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;data.table&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;library&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;car&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Load the iris dataset&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;data&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;iris&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Convert the dataset to a data.table&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;iris&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; as.data.table&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;iris&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Create a scatterplot matrix of the iris dataset&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;scatterplotMatrix&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;iris&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; smooth&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; FALSE&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;Create a faceted bar plot using &lt;code&gt;ggplot2&lt;/code&gt; and the &lt;code&gt;data.table&lt;/code&gt; package:&lt;/p&gt;
&lt;pre class=&quot;astro-code github-dark&quot; style=&quot;background-color:#24292e;color:#e1e4e8;overflow-x:auto&quot; tabindex=&quot;0&quot; data-language=&quot;r&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Load the data.table and ggplot2 packages&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;library&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;data.table&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;library&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;ggplot2&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Load the mpg dataset from the ggplot2 package&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;data&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mpg&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Convert the dataset to a data.table&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mpg&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; as.data.table&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mpg&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Create a faceted bar plot showing the distribution of hwy (highway miles per gallon) by class and drv (drive type)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;ggplot&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;mpg&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; aes&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;x&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; hwy&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)) &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;+&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;  geom_histogram&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;binwidth&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 2&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;) &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;+&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;  facet_wrap&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;~&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; class&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; +&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; drv&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; nrow&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 2&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;hr/&gt;
&lt;p&gt;In terms of implementation, both the &lt;code&gt;tidyverse&lt;/code&gt; and &lt;code&gt;data.table&lt;/code&gt; packages are written in R, but some of the functions in the &lt;code&gt;data.table&lt;/code&gt; package are implemented in C for improved performance.&lt;/p&gt;
&lt;h2 id=&quot;so-which-one&quot;&gt;So which one?&lt;/h2&gt;
&lt;p&gt;Ask how big the data is, then ask who reads the code.&lt;/p&gt;
&lt;p&gt;Under a million rows, with people who will maintain it after you, use the &lt;code&gt;tidyverse&lt;/code&gt;. The chains read like sentences, the errors are legible, and the performance difference will not be what slows the project down. On a few million rows or more, &lt;code&gt;data.table&lt;/code&gt; stops being an optimisation and becomes the difference between an analysis that runs and one that doesn’t. It modifies in place instead of copying, and once memory is the constraint that matters more than syntax.&lt;/p&gt;
&lt;p&gt;Does it have to be one or the other? No. &lt;code&gt;dtplyr&lt;/code&gt; gives you &lt;code&gt;dplyr&lt;/code&gt; verbs on a &lt;code&gt;data.table&lt;/code&gt; backend, which is a reasonable middle if your team knows one syntax and your data has outgrown it.&lt;/p&gt;
&lt;h3 id=&quot;at-analytica&quot;&gt;At Analytica&lt;/h3&gt;
&lt;p&gt;We work in gigabytes to terabytes, so &lt;code&gt;data.table&lt;/code&gt; is the default here and the &lt;code&gt;tidyverse&lt;/code&gt; gets used for the plotting and the final report. That’s a consequence of our data sizes, not a verdict on the packages. Your constraint is probably different from ours, and it should decide this rather than anyone’s preference.&lt;/p&gt;</content:encoded><dc:creator>Dr. Aous Abdo</dc:creator><category>R</category><category>Data wrangling</category><category>Performance</category></item><item><title>Analyzing crypto markets in R, part 1: exchanges and arbitrage</title><link>https://analyticadss.com/insights/analyzing-cryptocurrency-markets-using-r-part-1/</link><guid isPermaLink="true">https://analyticadss.com/insights/analyzing-cryptocurrency-markets-using-r-part-1/</guid><description>Bitcoin&apos;s price wasn&apos;t one price. Pulling 2011–2018 exchange data into R and measuring the gaps arbitrage was supposed to close.</description><pubDate>Sat, 01 Dec 2018 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;&lt;em&gt;A 2018 analysis, kept for the method and the findings. The data feeds it used — Quandl&apos;s exchange series and the Poloniex API — have since shut down or changed beyond recognition, so the code is shown for its approach rather than to run as-is. The figures are the original outputs.&lt;/em&gt;&lt;/p&gt;&lt;p&gt;In 2018 I got curious about a question with real money attached: is Bitcoin’s price actually &lt;em&gt;one&lt;/em&gt; price? Every exchange quotes its own, arbitrage is supposed to keep them honest, and I had seven years of data to check whether it did.&lt;/p&gt;
&lt;p&gt;The toolkit was ordinary R: &lt;code&gt;data.table&lt;/code&gt; and &lt;code&gt;lubridate&lt;/code&gt; for wrangling, &lt;code&gt;ggplot2&lt;/code&gt; for the pictures, with price series pulled from Quandl’s exchange feeds and the Poloniex API. Those feeds are gone now, which is why this post carries the note above. The method outlives the plumbing.&lt;/p&gt;
&lt;p&gt;First, the shape of the thing we’re analysing:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://analyticadss.com/img/posts/crypto/btc-weighted-price-history.webp&quot; alt=&quot;Bitcoin&apos;s volume-weighted price, 2012–2018: years of near-flatline, then the vertical 2017 spike toward $20,000 and the fall after.&quot; width=&quot;828&quot; height=&quot;497&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot;/&gt;&lt;/p&gt;
&lt;p&gt;That spike is late 2017, and everything interesting in this analysis happens in its shadow.&lt;/p&gt;
&lt;h2 id=&quot;how-far-apart-could-one-bitcoin-be-from-itself&quot;&gt;How far apart could one Bitcoin be from itself?&lt;/h2&gt;
&lt;p&gt;For each day I computed the maximum percentage gap between the highest and lowest price across exchanges — the raw material of arbitrage. Plotted over time, with a loess smoother:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://analyticadss.com/img/posts/crypto/price-difference-loess.webp&quot; alt=&quot;Daily maximum price difference between exchanges, log scale, 2013–2018, with a loess curve drifting down from several percent toward one percent.&quot; width=&quot;828&quot; height=&quot;492&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot;/&gt;&lt;/p&gt;
&lt;p&gt;Two things jump out. The typical gap &lt;em&gt;declined&lt;/em&gt; over the years (the market maturing, arbitrageurs doing their job), but the spread never disappeared, and single days still spiked to 10% and beyond deep into the study. The monthly view says the same thing with distributions instead of dots:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://analyticadss.com/img/posts/crypto/monthly-price-difference-boxplots.webp&quot; alt=&quot;Box plots of price differences by month: medians settle near one percent, with fat upper tails in every volatile period.&quot; width=&quot;828&quot; height=&quot;499&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot;/&gt;&lt;/p&gt;
&lt;p&gt;Medians settle around a percent. The tails never behave. And when I asked &lt;em&gt;which&lt;/em&gt; exchanges kept showing up at the extremes, the answer wasn’t random either:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://analyticadss.com/img/posts/crypto/exchange-max-min-differences.webp&quot; alt=&quot;Frequency of each exchange posting the day&apos;s maximum or minimum price: a small set of venues dominates both ends.&quot; width=&quot;828&quot; height=&quot;500&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot;/&gt;&lt;/p&gt;
&lt;p&gt;A handful of venues were persistently the expensive place and a handful persistently the cheap one. Textbook arbitrage says gaps like that get eaten within minutes. So why did they persist for years?&lt;/p&gt;
&lt;p&gt;Because the friction was real: withdrawal limits and delays, fees on both sides, counterparty risk on venues of very mixed reliability, and transfer times during exactly the volatile moments when gaps opened. The chart records the size of the reward; the frictions were the moat around it. “Free money” that persists is almost always a measurement of some cost you haven’t priced.&lt;/p&gt;
&lt;p&gt;That finding — markets are one market only when moving between them is cheap — is the whole lesson, and it generalises far past crypto. &lt;a href=&quot;https://analyticadss.com/insights/analyzing-cryptocurrency-markets-using-r-part-2/&quot;&gt;Part 2&lt;/a&gt; turns from one coin on many exchanges to many coins moving together, where the correlation structure has its own surprise.&lt;/p&gt;</content:encoded><dc:creator>Dr. Aous Abdo</dc:creator><category>R</category><category>Finance</category><category>Data analysis</category></item><item><title>Analyzing crypto markets in R, part 2: everything correlates</title><link>https://analyticadss.com/insights/analyzing-cryptocurrency-markets-using-r-part-2/</link><guid isPermaLink="true">https://analyticadss.com/insights/analyzing-cryptocurrency-markets-using-r-part-2/</guid><description>The diversification that wasn&apos;t: daily returns for eight coins, correlation matrices, and what happens to independence when markets fall.</description><pubDate>Sat, 01 Dec 2018 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;&lt;em&gt;The 2018 companion piece to part 1, kept for the method and the findings. Its data feeds are gone, so the code is illustrative; the figures are the original outputs.&lt;/em&gt;&lt;/p&gt;&lt;p&gt;&lt;a href=&quot;https://analyticadss.com/insights/analyzing-cryptocurrency-markets-using-r-part-1/&quot;&gt;Part 1&lt;/a&gt; asked whether Bitcoin was one price across exchanges. This half asks a portfolio question: were the altcoins genuinely different assets, or eight tickers for the same bet?&lt;/p&gt;
&lt;p&gt;The 2018 altcoin pitch leaned on diversification — don’t just hold BTC, spread across ETH, LTC, XRP and friends. Whether that spreads &lt;em&gt;risk&lt;/em&gt; depends entirely on correlation, and correlation is measurable. Same toolkit as before, prices for eight major coins, everything converted to daily returns:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://analyticadss.com/img/posts/crypto/daily-returns-all-pairs.webp&quot; alt=&quot;Daily percentage returns for eight coin pairs, 2015–2018: quiet stretches punctuated by synchronized spikes across every panel at once.&quot; width=&quot;828&quot; height=&quot;492&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot;/&gt;&lt;/p&gt;
&lt;p&gt;Even before any statistics, your eye catches it: the spikes line up. When one panel goes wild, they all do. The correlation matrix makes it precise:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://analyticadss.com/img/posts/crypto/correlation-matrix-ellipses.webp&quot; alt=&quot;Correlation matrix of daily returns for XRP, XEM, ETH, XMR, LTC, DASH, BCH and BTC: almost everything between 0.7 and 0.95.&quot; width=&quot;826&quot; height=&quot;722&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot;/&gt;&lt;/p&gt;
&lt;p&gt;Nearly every pair sits between 0.7 and 0.95. In equity terms those are numbers you’d expect &lt;em&gt;within one sector&lt;/em&gt;, not across an asset class. Drawn as a network, with distance standing for independence, the “diverse” portfolio collapses into one tight tangle:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://analyticadss.com/img/posts/crypto/static-correlation-network.webp&quot; alt=&quot;Static correlation network, 2014–2018: eight coins knotted into a single cluster, with BCH the only partial outlier.&quot; width=&quot;828&quot; height=&quot;519&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot;/&gt;&lt;/p&gt;
&lt;p&gt;Eight names, one knot. Holding five of these wasn’t diversification; it was the same position with extra fees.&lt;/p&gt;
&lt;h2 id=&quot;the-part-that-should-worry-a-portfolio&quot;&gt;The part that should worry a portfolio&lt;/h2&gt;
&lt;p&gt;Averages hide the dangerous detail, so I rolled the BTC–LTC correlation month by month:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://analyticadss.com/img/posts/crypto/btc-ltc-correlation-over-time.webp&quot; alt=&quot;Correlation between BTC and LTC daily returns computed per month, 2015–2018: mostly 0.7–0.9, dipping only in quiet periods.&quot; width=&quot;828&quot; height=&quot;519&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot;/&gt;&lt;/p&gt;
&lt;p&gt;The dips toward independence happen in calm stretches. In stressed ones the correlation pins to the top of the range — and the 2018 price panels show what that meant in practice:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://analyticadss.com/img/posts/crypto/altcoin-price-facets.webp&quot; alt=&quot;2018 prices for eight coins, faceted: every panel is the same descending shape.&quot; width=&quot;828&quot; height=&quot;499&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot;/&gt;&lt;/p&gt;
&lt;p&gt;Eight charts, one shape, all pointing down together. Which is the general and much older lesson this data illustrates unusually cleanly: &lt;strong&gt;correlations rise exactly when you need them low.&lt;/strong&gt; Diversification quoted from peaceful averages evaporates in a crisis, in crypto and everywhere else — 2008 taught equity investors the same thing at greater expense.&lt;/p&gt;
&lt;p&gt;Would I expect identical numbers today? No; the market has had years to mature, and I’d want fresh data before claiming anything about now. But the &lt;em&gt;method&lt;/em&gt; is the reusable part: returns, not prices; correlations over time, not on average; and a hard look at exactly the periods your model is most tempted to smooth over.&lt;/p&gt;</content:encoded><dc:creator>Dr. Aous Abdo</dc:creator><category>R</category><category>Finance</category><category>Data analysis</category></item><item><title>Conway&apos;s Game of Life, in R and in Python</title><link>https://analyticadss.com/insights/conways-game-of-life-with-examples-in-r-and-python/</link><guid isPermaLink="true">https://analyticadss.com/insights/conways-game-of-life-with-examples-in-r-and-python/</guid><description>The four rules of Conway&apos;s Game of Life, the patterns they produce (gliders, oscillators, guns), and a full simulation in Python and in R to run yourself.</description><pubDate>Wed, 17 Jan 2018 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;&lt;em&gt;Written some years ago. The ideas hold, but check package versions and API details against current documentation before relying on the code.&lt;/em&gt;&lt;/p&gt;&lt;p&gt;Four rules. No player. That’s the entire specification of Conway’s Game of Life, and it is enough to build a pattern that computes.&lt;/p&gt;
&lt;p&gt;John Conway published it in 1970. It’s a cellular automaton: a grid where each cell is alive or dead, and every cell’s next state depends only on how many of its eight neighbours are currently alive. You set a starting arrangement and then you don’t touch it again. Everything after that is the rules running.&lt;/p&gt;
&lt;p&gt;What makes it worth an afternoon is how little it takes to get complicated. Gliders that travel across the grid. Oscillators that return to their starting shape every few generations. Arrangements that grow without bound. People have built a working Turing machine out of it, which means the four rules below are, in a strict sense, capable of computing anything computable.&lt;/p&gt;
&lt;p&gt;Below: the rules, then implementations in R and in Python, both short enough to read in one sitting.&lt;/p&gt;
&lt;h2 id=&quot;game-rules&quot;&gt;Game Rules&lt;/h2&gt;
&lt;p&gt;The game rules are as follows:&lt;/p&gt;
&lt;ol&gt;
&lt;li&gt;Any live cell with fewer than two live neighbors dies, as if by underpopulation.&lt;/li&gt;
&lt;li&gt;Any live cell with two or three live neighbors lives on to the next generation.&lt;/li&gt;
&lt;li&gt;Any live cell with more than three live neighbors dies, as if by overpopulation.&lt;/li&gt;
&lt;li&gt;Any dead cell with exactly three live neighbors becomes a live cell, as if by reproduction.&lt;/li&gt;
&lt;/ol&gt;
&lt;p&gt;These rules are applied to each cell in the grid simultaneously, with the state of the cells in the next generation being determined based on the state of the cells in the current generation. The game continues in this way, with the state of the cells being updated in each generation.&lt;/p&gt;
&lt;p&gt;The Game of Life can be implemented in many different ways, with the rules being applied to a two-dimensional grid, or to a one-dimensional “tape,” or even to a three-dimensional space. There are also many variations of the rules that have been explored, with different sets of rules leading to different patterns and behaviors in the game.&lt;/p&gt;
&lt;p&gt;The Game of Life is capable of producing a wide range of patterns, depending on the initial configuration of the cells and the specific rules that are being used. Some patterns remain stable over time, while others may evolve and change over the course of the game.&lt;/p&gt;
&lt;h2 id=&quot;game-patterns&quot;&gt;Game Patterns&lt;/h2&gt;
&lt;p&gt;The classic starting shapes and their fates, from the original post:&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://analyticadss.com/img/posts/conway/pattern-taxonomy.webp&quot; alt=&quot;A taxonomy of small starting patterns: several settle into stable blocks, one dies out entirely, and one repeats forever.&quot; width=&quot;828&quot; height=&quot;877&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot;/&gt;&lt;/p&gt;
&lt;p&gt;&lt;img src=&quot;https://analyticadss.com/img/posts/conway/glider-and-friends.webp&quot; alt=&quot;Familiar residents of the grid, including the glider — the small arrangement that travels diagonally forever.&quot; width=&quot;610&quot; height=&quot;178&quot; loading=&quot;lazy&quot; decoding=&quot;async&quot;/&gt;&lt;/p&gt;
&lt;p&gt;Few examples of patterns that can appear in the Game of Life&lt;/p&gt;
&lt;p&gt;Here are a few examples of patterns that can appear in the Game of Life:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Still lifes: These are patterns that remain stable over time and do not change from one generation to the next. Examples of still lifes include blocks, beehives, and loafs.&lt;/li&gt;
&lt;li&gt;Oscillators: These are patterns that repeat themselves over time, cycling through a fixed set of states. Examples of oscillators include blinkers, toads, and pulsars.&lt;/li&gt;
&lt;li&gt;Spaceships: These are patterns that move across the grid, leaving behind a “trail” of cells as they go. Spaceships can move in any of the four cardinal directions, and their movement may be periodic or aperiodic.&lt;/li&gt;
&lt;li&gt;Gliders: These are a type of spaceship that move diagonally across the grid, leaving behind a distinctive “trail” of cells. Gliders are one of the simplest and most well-known spaceships in the Game of Life.&lt;/li&gt;
&lt;li&gt;Guns: These are patterns that produce an endless stream of spaceships or other patterns. The first gun was discovered in 1970 by Bill Gosper, and many more have been found since.&lt;/li&gt;
&lt;li&gt;Patterns with complex behavior: Some patterns in the Game of Life exhibit behavior that is difficult to predict or understand. These patterns may exhibit seemingly random behavior, or they may exhibit complex, self-organizing behavior.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;These are just a few examples of the patterns that can appear in the Game of Life. The game is capable of producing a wide range of patterns and behaviors, and new patterns are still being discovered today.&lt;/p&gt;
&lt;h2 id=&quot;real-life-examples&quot;&gt;Real-Life Examples&lt;/h2&gt;
&lt;p&gt;So does any of this matter outside a screensaver? More than I expected when I first met it. The Game of Life turns up wherever people are trying to understand how local rules produce global behaviour.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;Biology: The Game of Life has been used to model the behavior of cellular automata, including the growth and division of cells. It has also been used to study patterns in the distribution of species in ecosystems and the spread of epidemics.&lt;/li&gt;
&lt;li&gt;Physics: The Game of Life has been used to model the behavior of physical systems, such as the formation of patterns in crystals and the behavior of fluids.&lt;/li&gt;
&lt;li&gt;Computer science: The Game of Life has been used as a test bed for exploring algorithms and data structures, and it has inspired the development of a number of computational models and techniques.&lt;/li&gt;
&lt;li&gt;Art and design: The Game of Life has inspired a number of artistic and design projects, including digital art, installations, and interactive exhibits.&lt;/li&gt;
&lt;li&gt;Education: The Game of Life has been used as a tool for teaching concepts in mathematics, computer science, and other fields, and it has been incorporated into educational software and curricula.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;The thread running through all of those is the same one: nobody designed the glider. It falls out of the rules. That is the property people keep coming back for, and it is why I think every analyst should implement this once by hand.&lt;/p&gt;
&lt;h2 id=&quot;game-simulation-in-r&quot;&gt;Game Simulation in R&lt;/h2&gt;
&lt;p&gt;The script below simulates the Game of Life and produces a plot of the evolution of the cells over time. The &lt;code&gt;plotly&lt;/code&gt; package is used to create an interactive heatmap, with “dead” cells being shown in darker shades and “alive” cells being shown in lighter shades. The x-axis of the plot represents the generation, and the y-axis represents the cell.&lt;/p&gt;
&lt;pre class=&quot;astro-code github-dark&quot; style=&quot;background-color:#24292e;color:#e1e4e8;overflow-x:auto&quot; tabindex=&quot;0&quot; data-language=&quot;r&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Install the plotly and furrr packages if they are not already installed&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# install.packages(&amp;quot;plotly&amp;quot;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# install.packages(&amp;quot;furrr&amp;quot;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;library&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;plotly&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;library&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;furrr&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Set up the grid for the game&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;grid&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; matrix&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;sample&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;c&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;0&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 1&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 100&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;*&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;100&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; replace&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; TRUE&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; nrow&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 100&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Initialize a list to store the state of the cells at each generation&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;grid_history&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; rep&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;list&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;grid&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; generations&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Simulate the game using the furrr package to parallelize the simulation&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;plan&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;multisession&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;grid_history&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; future_map&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;:&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;generations&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; function&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;i&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;) {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;  # Get the current state of the cells&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;  current_state&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; grid_history&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;[[&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;i&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;]]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;  &lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;  # Initialize the next state of the cells&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;  next_state&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; matrix&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;0&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; nrow&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; nrow&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;current_state&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; ncol&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; ncol&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;current_state&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;))&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;  &lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;  # Iterate over each cell in the grid&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;  for&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;x&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; in&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 1&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;:&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;nrow&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;current_state&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)) {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;    for&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;y&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; in&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 1&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;:&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;ncol&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;current_state&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)) {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;      # Get the number of live neighbors for the current cell&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;      neighbors&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; sum&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;current_state&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;max&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;x&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 1&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;:&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;min&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;x&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;+&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; nrow&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;current_state&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;))&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt; &lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;                                     max&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;y&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 1&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;:&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;min&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;y&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;+&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; ncol&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;current_state&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;))]) &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; current_state&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;x&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; y&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;      &lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;      # Apply the rules of the Game of Life to determine the next state of the cell&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;      if&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;current_state&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;x&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; y&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;] &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;==&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 1&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;) {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;        if&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;neighbors&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 2&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; ||&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; neighbors&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;gt;&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 3&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;) {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;          next_state&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;x&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; y&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;] &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;&amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 0&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;        } &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;else&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;          next_state&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;x&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; y&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;] &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;&amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 1&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;        }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;      } &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;else&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;        if&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt; (&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;neighbors&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; ==&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 3&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;) {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;          next_state&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;x&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; y&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;] &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;&amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 1&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;        } &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;else&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt; {&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;          next_state&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;[&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;x&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; y&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;] &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;&amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 0&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;        }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;      }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;    }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;  }&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;  &lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;  # Update the state of the cells&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;  grid&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; &amp;lt;-&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; next_state&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;  &lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;  # Store the state of the cells at the current generation&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;  grid&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;})&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Plot the evolution of the cells over time using the plotly package&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;plot_ly&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;z&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; grid_history&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; colorscale&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt; &amp;quot;Blackbody&amp;quot;&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; type&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt; &amp;quot;heatmapgl&amp;quot;&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;) &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;%&amp;gt;%&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#B392F0&quot;&gt;  layout&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;xaxis&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; list&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;title&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt; &amp;quot;Generation&amp;quot;&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;,&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt; yaxis&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#B392F0&quot;&gt; list&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;title&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; =&lt;/span&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt; &amp;quot;Cell&amp;quot;&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;))&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;h2 id=&quot;game-simulation-in-python&quot;&gt;Game Simulation in Python&lt;/h2&gt;
&lt;p&gt;The script below simulates the Game of Life and produces a plot of the evolution of the cells over time. The &lt;code&gt;matplotlib&lt;/code&gt; package is used to create a heatmap, with “dead” cells being shown in darker shades and “alive” cells being shown in lighter shades. The x-axis of the plot represents the generation, and the y-axis represents the cell.&lt;/p&gt;
&lt;pre class=&quot;astro-code github-dark&quot; style=&quot;background-color:#24292e;color:#e1e4e8;overflow-x:auto&quot; tabindex=&quot;0&quot; data-language=&quot;python&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;import&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt; numpy &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;as&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt; np&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;import&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt; matplotlib.pyplot &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;as&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt; plt&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Set up the grid for the game&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;grid &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt; np.random.choice([&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;0&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;], &lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;size&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;100&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;100&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;))&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Initialize a list to store the state of the cells at each generation&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;grid_history &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt; [grid]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Simulate the game&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;for&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt; i &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;in&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; range&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(generations):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;    # Get the current state of the cells&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;    current_state &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt; grid_history[i]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;    # Initialize the next state of the cells&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;    next_state &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt; np.zeros((&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;100&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;100&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;))&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;    # Iterate over each cell in the grid&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;    for&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt; x &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;in&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; range&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;100&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;        for&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt; y &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;in&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; range&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;100&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;):&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;            # Get the number of live neighbors for the current cell&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;            neighbors &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt; (current_state[&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;max&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(x&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;0&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;):&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;min&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(x&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;+&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;2&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;100&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;), &lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;max&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(y&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;1&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;0&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;):&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;min&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;(y&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;+&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;2&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;, &lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt;100&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)]).sum() &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;-&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt; current_state[x, y]&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;            # Apply the rules of the Game of Life to determine the next state of the cell&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;            if&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt; current_state[x, y] &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;==&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 1&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;                if&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt; neighbors &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;&amp;lt;&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 2&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt; or&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt; neighbors &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;&amp;gt;&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 3&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;                    next_state[x, y] &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 0&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;                else&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;                    next_state[x, y] &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 1&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;            else&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;                if&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt; neighbors &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;==&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 3&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;                    next_state[x, y] &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 1&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#F97583&quot;&gt;                else&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;:&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;                    next_state[x, y] &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#79B8FF&quot;&gt; 0&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;    # Update the state of the cells&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;    grid &lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt; next_state&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;    # Store the state of the cells at the current generation&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;    grid_history.append(grid)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#959DA5&quot;&gt;# Plot the evolution of the cells over time using the matplotlib package&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;plt.imshow(grid_history, &lt;/span&gt;&lt;span style=&quot;color:#FFAB70&quot;&gt;cmap&lt;/span&gt;&lt;span style=&quot;color:#F97583&quot;&gt;=&lt;/span&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt;&amp;quot;Greys&amp;quot;&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;plt.xlabel(&lt;/span&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt;&amp;quot;Generation&amp;quot;&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;plt.ylabel(&lt;/span&gt;&lt;span style=&quot;color:#9ECBFF&quot;&gt;&amp;quot;Cell&amp;quot;&lt;/span&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;)&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span style=&quot;color:#E1E4E8&quot;&gt;plt.show()&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;h2 id=&quot;why-its-worth-your-time&quot;&gt;Why it’s worth your time&lt;/h2&gt;
&lt;p&gt;Four rules and no player, and you get gliders, oscillators, and arrangements that grow forever. That gap between how simple the specification is and how complicated the behaviour gets is the whole reason people still write these implementations forty-odd years on.&lt;/p&gt;
&lt;p&gt;It’s also a useful thing to keep in mind when you build models for a living. Complicated output does not require a complicated model. Sometimes it requires four rules and enough iterations, which is worth remembering the next time somebody insists a phenomenon must have a complicated cause.&lt;/p&gt;
&lt;p&gt;Run either script above and change the starting grid. That’s the fun part.&lt;/p&gt;</content:encoded><dc:creator>Dr. Aous Abdo</dc:creator><category>R</category><category>Python</category><category>Simulation</category></item></channel></rss>