RTUG

mdbplyr: a tidy interface to MongoDB

Paolo Bosetti — Thu, 16 Apr 2026 00:00:00 GMT

Introduction

We are developing mdbplyr, an R package that provides a disciplined, lazy dplyr-style interface for MongoDB aggregation pipelines.

mdbplyr logo

mdbplyr sits between raw mongolite usage and broader compatibility layers. Compared with writing aggregation JSON by hand, it lets you express supported queries with tidy verbs and inspect the generated pipeline before execution. Compared with approaches that try to hide MongoDB behind table-like semantics, it stays explicit about scope, keeps translation conservative, and fails clearly when a verb or expression is outside the supported subset.

The practical advantages are:

lazy query construction until collect(),
direct translation to MongoDB aggregation stages,
show_query() for inspectable pipeline output,
explicit unsupported-feature errors instead of silent fallback,
a smaller, testable semantic surface aligned with MongoDB rather than SQL.

Important

This post is a quick introduction to mdbplyr with examples. The package is still in early development, so expect some rough edges and missing features. The API is not yet stable, and the documentation is still a work in progress. If you’re interested in trying it out or contributing, check out the GitHub repository.

Usage examples

Installation

Since April 22, 20226, mdbplyr is available on CRAN. You can install it with:

install.packages("mdbplyr")

Since mdbplyr is still in early development, you can alternatively install the latest development version from GitHub using the devtools package:

install.packages("devtools")
devtools::install_github("pbosetti/mdbplyr")

Loading data

Suppose there is a running MongoDB instance on localhost with default port and no authentication. The code below loads dplyr::starwars into a collection named starwars.

library(dplyr)
library(mongolite)
library(mdbplyr)

starwars_collection <- mongolite::mongo(
  collection = "starwars",
  db = "mdbplyr"
)

starwars_collection$drop()
starwars_collection$insert(dplyr::starwars)

List of 5
 $ nInserted  : num 87
 $ nMatched   : num 0
 $ nRemoved   : num 0
 $ nUpserted  : num 0
 $ writeErrors: list()

starwars_tbl <- tbl_mongo(
  starwars_collection,
  schema = names(dplyr::starwars)
)

Once the collection is loaded, starwars_tbl is the lazy table used in the examples below.

library(dplyr)
library(mdbplyr)

starwars_collection <- mongolite::mongo(
  collection = "starwars",
  db = "mdbplyr"
)

starwars_tbl <- tbl_mongo(starwars_collection) %>% 
  infer_schema()

Basic inspection

Inspect the known schema and the generated pipeline without executing the query.

schema_fields(starwars_tbl)

 [1] "_id"        "name"       "height"     "mass"       "hair_color"
 [6] "skin_color" "eye_color"  "birth_year" "sex"        "gender"    
[11] "homeworld"  "species"    "films"      "vehicles"   "starships"

starwars_tbl |>
  filter(species == "Human", height > 180) |>
  select(name, height, mass) |>
  show_query()

[
  {
    "$match": {
      "$expr": {
        "$and": {
          "1": {
            "$eq": [
              "$species",
              "Human"
            ]
          },
          "2": {
            "$gt": [
              "$height",
              180
            ]
          }
        }
      }
    }
  },
  {
    "$project": {
      "name": 1,
      "height": 1,
      "mass": 1,
      "_id": 0
    }
  }
]

Supported verbs

Each subsection below shows one of the supported dplyr-like verbs on the starwars collection.

`filter()`

starwars_tbl |>
  filter(species == "Droid", height > 100) |>
  collect()

# A tibble: 87 × 15
   `_id`     name  height  mass hair_color skin_color eye_color birth_year sex  
                                   
 1 6a1ffb87… Luke…    172    77 blond      fair       blue            19   male 
 2 6a1ffb87… C-3PO    167    75        gold       yellow         112   none 
 3 6a1ffb87… R2-D2     96    32        white, bl… red             33   none 
 4 6a1ffb87… Dart…    202   136 none       white      yellow          41.9 male 
 5 6a1ffb87… Leia…    150    49 brown      light      brown           19   fema…
 6 6a1ffb87… Owen…    178   120 brown, gr… light      blue            52   male 
 7 6a1ffb87… Beru…    165    75 brown      light      blue            47   fema…
 8 6a1ffb87… R5-D4     97    32        white, red red             NA   none 
 9 6a1ffb87… Bigg…    183    84 black      light      brown           24   male 
10 6a1ffb87… Obi-…    182    77 auburn, w… fair       blue-gray       57   male 
# ℹ 77 more rows
# ℹ 6 more variables: gender , homeworld , species ,
#   films , vehicles , starships

`select()`

starwars_tbl |>
  select(name, species, homeworld) |>
  collect()

# A tibble: 87 × 3
   name               homeworld species
                        
 1 Luke Skywalker     Tatooine  Human  
 2 C-3PO              Tatooine  Droid  
 3 R2-D2              Naboo     Droid  
 4 Darth Vader        Tatooine  Human  
 5 Leia Organa        Alderaan  Human  
 6 Owen Lars          Tatooine  Human  
 7 Beru Whitesun Lars Tatooine  Human  
 8 R5-D4              Tatooine  Droid  
 9 Biggs Darklighter  Tatooine  Human  
10 Obi-Wan Kenobi     Stewjon   Human  
# ℹ 77 more rows

`rename()`

starwars_tbl |>
  rename(character_name = name, planet = homeworld) |>
  collect()

# A tibble: 87 × 15
   `_id`  character_name height  mass hair_color skin_color eye_color birth_year
                                        
 1 6a1ff… Luke Skywalker    172    77 blond      fair       blue            19  
 2 6a1ff… C-3PO             167    75        gold       yellow         112  
 3 6a1ff… R2-D2              96    32        white, bl… red             33  
 4 6a1ff… Darth Vader       202   136 none       white      yellow          41.9
 5 6a1ff… Leia Organa       150    49 brown      light      brown           19  
 6 6a1ff… Owen Lars         178   120 brown, gr… light      blue            52  
 7 6a1ff… Beru Whitesun…    165    75 brown      light      blue            47  
 8 6a1ff… R5-D4              97    32        white, red red             NA  
 9 6a1ff… Biggs Darklig…    183    84 black      light      brown           24  
10 6a1ff… Obi-Wan Kenobi    182    77 auburn, w… fair       blue-gray       57  
# ℹ 77 more rows
# ℹ 7 more variables: sex , gender , planet , species ,
#   films , vehicles , starships

`mutate()`

starwars_tbl |>
  mutate(height_m = height / 100) |>
  mutate(bmi_like = mass / height_m^2 ) |> # can't be in the same mutate()!
  select(name, height, mass, height_m, bmi_like) |>
  collect()

# A tibble: 87 × 5
   name               height  mass height_m bmi_like
                           
 1 Luke Skywalker        172    77     1.72     26.0
 2 C-3PO                 167    75     1.67     26.9
 3 R2-D2                  96    32     0.96     34.7
 4 Darth Vader           202   136     2.02     33.3
 5 Leia Organa           150    49     1.5      21.8
 6 Owen Lars             178   120     1.78     37.9
 7 Beru Whitesun Lars    165    75     1.65     27.5
 8 R5-D4                  97    32     0.97     34.0
 9 Biggs Darklighter     183    84     1.83     25.1
10 Obi-Wan Kenobi        182    77     1.82     23.2
# ℹ 77 more rows

Important

Currently, mdbplyr implements mapping between MongoDB operators and:

Arithmetic/predicate: %%, ^, %in%
Logs/math: log(x, base), log10(), floor(), ceiling(), trunc()
Trig/hyperbolic: sin(), cos(), tan(), asin(), acos(), atan(), atan2(), sinh(), cosh(), tanh(), asinh(), acosh(), atanh()
Scalar min/max: pmin(), pmax()
String ops: tolower(), toupper(), nchar(), paste(), paste0(), substr(), substring()
Literal vector support for expressions like x %in% c(1, 3, 5)

Note though, that within the same mutate call you cannot use a newly created field in another expression, as the translation is currently one-to-one with MongoDB’s $addFields stage. So you need to split the BMI-like calculation into two mutate() calls, as in te above example.

`transmute()`

starwars_tbl |>
  transmute(name = name, height_m = height / 100) |>
  collect()

# A tibble: 87 × 2
   name               height_m
                    
 1 Luke Skywalker         1.72
 2 C-3PO                  1.67
 3 R2-D2                  0.96
 4 Darth Vader            2.02
 5 Leia Organa            1.5 
 6 Owen Lars              1.78
 7 Beru Whitesun Lars     1.65
 8 R5-D4                  0.97
 9 Biggs Darklighter      1.83
10 Obi-Wan Kenobi         1.82
# ℹ 77 more rows

`arrange()`

starwars_tbl |>
  arrange(desc(height), name) |>
  select(name, height) |>
  slice_head(n = 10) |>
  collect()

# A tibble: 10 × 2
   name         height
            
 1 Yarael Poof     264
 2 Tarfful         234
 3 Lama Su         229
 4 Chewbacca       228
 5 Roos Tarpals    224
 6 Grievous        216
 7 Taun We         213
 8 Rugor Nass      206
 9 Tion Medon      206
10 Darth Vader     202

`group_by()`

starwars_tbl |>
  group_by(species)

 collection 
  Filters: 0 
  Projection:  
  Computed:  
  Groups: species 
  Summaries:  
  Slice:  
  Manual stages: 0

`summarise()`

starwars_tbl |>
  group_by(species) |>
  summarise(
    n = n(),
    avg_height = mean(height),
    max_mass = max(mass)
  ) |>
  arrange(desc(n)) |>
  collect()

# A tibble: 38 × 4
   species      n avg_height max_mass
                 
 1 Human       35       178     136  
 2 Droid        6       131.    140  
 3          4       175     110  
 4 Gungan       3       209.     82  
 5 Zabrak       2       173      80  
 6 Twi'lek      2       179      55  
 7 Mirialan     2       168      56.2
 8 Wookiee      2       231     136  
 9 Kaminoan     2       221      88  
10 Cerean       1       198      82  
# ℹ 28 more rows

`slice_head()`

starwars_tbl |>
  select(name, species) |>
  slice_head(n = 5) |>
  collect()

# A tibble: 5 × 2
  name           species
              
1 Luke Skywalker Human  
2 C-3PO          Droid  
3 R2-D2          Droid  
4 Darth Vader    Human  
5 Leia Organa    Human

`head()`

head(starwars_tbl, 5) |>
  collect()

# A tibble: 5 × 15
  `_id`      name  height  mass hair_color skin_color eye_color birth_year sex  
                                   
1 6a1ffb87d… Luke…    172    77 blond      fair       blue            19   male 
2 6a1ffb87d… C-3PO    167    75        gold       yellow         112   none 
3 6a1ffb87d… R2-D2     96    32        white, bl… red             33   none 
4 6a1ffb87d… Dart…    202   136 none       white      yellow          41.9 male 
5 6a1ffb87d… Leia…    150    49 brown      light      brown           19   fema…
# ℹ 6 more variables: gender , homeworld , species ,
#   films , vehicles , starships

Notes

The examples above stay within the currently supported subset:

explicit field references,
scalar expressions for mutate() and transmute(),
conservative grouped summaries,
no joins, window functions, or automatic client-side fallback.

If a query falls outside that subset, mdbplyr is designed to fail explicitly rather than guess or silently change execution semantics.

That’s all, folks!

RTUG::25.2 meeting

Paolo Bosetti — Mon, 02 Jun 2025 00:00:00 GMT

It is now time for planning the next R-Trento Users Meeting, code RTUG::25.2. The event is loosely planned for late Fall 2025. Content and location are still to be defined, but we are looking for contributions from the community, i.e. YOU!

Details will be announced here and on our Meetup page.

Remember: RTUG meetings are open to contributions, in the form of:

presentations
workshops or hands-on sessions
discussions on open topics
request for help or collaboration on specific issues

Drop your proposals to .

Topics

To be defined.

Venue

To be defined.

Details on Topics

To be defined.

Tukey vs. Student

Paolo Bosetti — Thu, 10 Apr 2025 00:00:00 GMT

Packages that we need

In this example, we are using the packages tidyverse and adas.utils version 1.1.4 (see https://github.com/pbosetti/adas.utils)

library(tidyverse)
library(adas.utils)

Repeated T-test vs. Tukey’s test

The dataset

Let us compare the result of a Tukey’s test with a repeated Student’s T-test on all combinations. We consider the cotton dataset, which is included in the adas.utils package from version 1.1.4. The dataset contains the tensile strength of mixed cotton-synthetic yarns with different cotton content:

cotton %>% 
  ggplot(aes(x=Cotton, y=Strength, group=Cotton)) +
  geom_boxplot()

Inference on `Strength`

Now we want to compare all the possibile combinations of treatmentswith a set of pairwise T-tests.

First, we create the list of pairwise combinations, sorting each pair in descending order, as it is done by the TukeyHSD function:

lvl <- levels(cotton$Cotton) %>% 
  combn(2, FUN=sort, decreasing=T) %>% 
  as_tibble(.name_repair="minimal") %>% 
  as.list() %>% glimpse()

List of 10
 $ : chr [1:2] "20" "15"
 $ : chr [1:2] "25" "15"
 $ : chr [1:2] "30" "15"
 $ : chr [1:2] "35" "15"
 $ : chr [1:2] "25" "20"
 $ : chr [1:2] "30" "20"
 $ : chr [1:2] "35" "20"
 $ : chr [1:2] "30" "25"
 $ : chr [1:2] "35" "25"
 $ : chr [1:2] "35" "30"

Now, for each pair we do a T-test on the corresponding cotton data-frame subset, and accumulate into a new tibble the values of interest. We get the df table that is analogous to the TukeyHSD output:

df <- lvl %>% reduce(\(acc, pair) {
  tt <- cotton %>% 
    filter(Cotton %in% pair) %>% 
    t.test(Strength~Cotton, data=., var.equal=TRUE)
  bind_rows(acc, list(
    pair = paste0(pair[1], "-", pair[2]),
    diff = -median(tt$conf.int),
    lwr = -tt$conf.int[2],
    upr = -tt$conf.int[1],
    p.value = tt$p.value
  ))
}, .init=tibble()) 

df %>% knitr::kable()

pair	diff	lwr	upr	p.value
20-15	5.6	0.8740978	10.3259022	0.0257453
25-15	7.8	3.7398608	11.8601392	0.0021967
30-15	11.8	7.4246648	16.1753352	0.0002541
35-15	1.0	-3.5423014	5.5423014	0.6253800
25-20	2.2	-1.6724395	6.0724395	0.2265324
30-20	6.2	1.9982605	10.4017395	0.0093233
35-20	-4.6	-8.9753352	-0.2246648	0.0415629
30-25	4.0	0.5641312	7.4358688	0.0277266
35-25	-6.8	-10.4461127	-3.1538873	0.0026133
35-30	-10.8	-14.7941163	-6.8058837	0.0002496

To be compared with Tukey’s values:

ttdf <- TukeyHSD(aov(lm(Strength~Cotton, data=cotton)))$Cotton %>% 
  as.data.frame() %>% 
  rownames_to_column(var="pair") %>% 
  rename(p.value=`p adj`)
ttdf %>% knitr::kable()

pair	diff	lwr	upr	p.value
20-15	5.6	0.2270417	10.9729583	0.0385024
25-15	7.8	2.4270417	13.1729583	0.0025948
30-15	11.8	6.4270417	17.1729583	0.0000190
35-15	1.0	-4.3729583	6.3729583	0.9797709
25-20	2.2	-3.1729583	7.5729583	0.7372438
30-20	6.2	0.8270417	11.5729583	0.0188936
35-20	-4.6	-9.9729583	0.7729583	0.1162970
30-25	4.0	-1.3729583	9.3729583	0.2101089
35-25	-6.8	-12.1729583	-1.4270417	0.0090646
35-30	-10.8	-16.1729583	-5.4270417	0.0000624

Now let’s join both tables and make a common plot:

compared <- df %>% 
  left_join(ttdf, by=join_by(pair), suffix=c(".student", ".tukey")) %>% 
  pivot_longer(-pair, names_to = c("stat", "test"), names_pattern = "(.*)\\.(student|tukey)$") %>% 
  pivot_wider(names_from = stat)

compared %>% knitr::kable()

pair	test	diff	lwr	upr	p.value
20-15	student	5.6	0.8740978	10.3259022	0.0257453
20-15	tukey	5.6	0.2270417	10.9729583	0.0385024
25-15	student	7.8	3.7398608	11.8601392	0.0021967
25-15	tukey	7.8	2.4270417	13.1729583	0.0025948
30-15	student	11.8	7.4246648	16.1753352	0.0002541
30-15	tukey	11.8	6.4270417	17.1729583	0.0000190
35-15	student	1.0	-3.5423014	5.5423014	0.6253800
35-15	tukey	1.0	-4.3729583	6.3729583	0.9797709
25-20	student	2.2	-1.6724395	6.0724395	0.2265324
25-20	tukey	2.2	-3.1729583	7.5729583	0.7372438
30-20	student	6.2	1.9982605	10.4017395	0.0093233
30-20	tukey	6.2	0.8270417	11.5729583	0.0188936
35-20	student	-4.6	-8.9753352	-0.2246648	0.0415629
35-20	tukey	-4.6	-9.9729583	0.7729583	0.1162970
30-25	student	4.0	0.5641312	7.4358688	0.0277266
30-25	tukey	4.0	-1.3729583	9.3729583	0.2101089
35-25	student	-6.8	-10.4461127	-3.1538873	0.0026133
35-25	tukey	-6.8	-12.1729583	-1.4270417	0.0090646
35-30	student	-10.8	-14.7941163	-6.8058837	0.0002496
35-30	tukey	-10.8	-16.1729583	-5.4270417	0.0000624

compared %>%
  ggplot(aes(x=diff, y=pair, color=test)) + 
  geom_point() + 
  geom_errorbar(aes(xmin=lwr, xmax=upr), width=0.5, position=position_dodge()) + 
  geom_vline(xintercept=0, color="red") +
  labs(x="Difference", y="Pair", title="95% pairwise confidence level")

The Family-Wise Error Rate

As expected, the Tukey’s test in the last plot shows larger confidence intervals, that is, it has reduced chances of a false positive (Type I Error). More specifically, Tukey’s test controls the family-wise error rate (FWER) — the probability of making any false positive in the full set of comparisons.

Let’s see why. If we set a confidence level of 0.95, it means that the probability of not making a Type I error on a single T-test is 0.95.

For 3 independent tests, the probability of no Type I error at all (in any of the tests) is: So the chance of making at least one Type I error is: That’s almost triple the risk you thought you were accepting! Furthermore, this risk increases exponentially with the number of comparisons. Given elements, the number of possible combinations of elements is given by the binomial coefficient: In R, the latter is provided by the choose(n, k) function:

choose(5, 2)

[1] 10

so, with increasing number of classes to be compared, this is what happens to the probability of committing at least one Type-I error:

2:15 %>% reduce(\(acc, k) {
    nt <- choose(k, 2)
    bind_rows(acc,
      list(n=k, p=1-0.95^nt)
    )
  }, .init=tibble()) %>% 
  ggplot(aes(x=n, y=p)) +
  geom_point() +
  geom_line() +
  ylim(0, 1) +
  labs(
    x="number of classes", 
    y="probability of Type-I Error")

And what happens if we change the confidence level? Let’s see, by creating a parametric plot similar to the ast one, but with different confidence levels. First we factor the last reduce opration into a function, FWER, that takes the confidence level as an argument. The function returns a tibble with the number of classes and the corresponding probability of Type I error.

FWER <- function(levels, conf.int=0.95) {
  reduce(levels, \(acc, k) {
    nt <- choose(k, 2)
    bind_rows(acc,
      list(n=k, p=1-conf.int^nt)
    )
  }, .init=tibble()) 
}

Then we apply the FWER function to a set of confidence levels, and join the results into a single tibble via the usual reduce, and finally, we plot the results:

cl <- c(0.9, 0.95, 0.99, 0.995, 0.999) 
N <- 2:15
cl %>% 
  reduce(\(acc, ci) {
    fwer <- FWER(N, ci) %>% rename(!!paste0("cl-", ci):=p)
    left_join(acc, fwer, by=join_by(n))
  }, .init=tibble(n=N)) %>% 
  pivot_longer(-n, names_to = c(NA, "cl"), names_pattern="(cl-)(.*)") %>% 
  ggplot(aes(x=n, y=value, color=cl)) + 
  geom_point() + 
  geom_line() + 
  ylim(0, 1) + 
  labs(
    x="number of classes", 
    y="probability of Type-I Error", 
    color="Conf. level")

That’s all, folks!

R-Consortium supporting RTUG

Paolo Bosetti — Fri, 28 Mar 2025 00:00:00 GMT

We’re supported by the R-Consortium

An important milestone

Today, March 28th, 2025, we are proud to announce that the R-Trento Users Group has been accepted by the R-Consortium. This means that we are now officially recognized as a community of R users and developers, and we will receive support from the R-Consortium for our activities.

We thank the R-Consortium for the financial support that will allow us to organize our first in-person meeting in the next months and to have our meeting announcements published on .

Tukey’s test plot in adas.utils

Paolo Bosetti — Thu, 27 Mar 2025 00:00:00 GMT

Rationale

The new version v1.1.1 of the adas.utils package includes a new function to plot the results of Tukey’s test. The function is called ggTukey and it is used to plot the results of Tukey’s test provided by the stas::TukeyHSD function.

John W. Tukey

The standard stats::TukeyHSD function returns an S3 object that has the print and plot methods. The result of the plot method, though, is honestly not really appealing. Let’s see how it works, by loading a dataset and running a Tukey’s test on it. We load an online dataset by using the adas.utils::examples_url function¹, and we begin with a simple boxplot of the data.

library(tidyverse)
library(adas.utils)

data <- examples_url("anova.dat") %>% 
  read.table(header=TRUE) %>% 
  mutate(Cotton=factor(Cotton)) %>% 
  glimpse()

Rows: 25
Columns: 3
$ Cotton       15, 15, 15, 15, 15, 20, 20, 20, 20, 20, 25, 25, 25, 25, 25…
$ Observation  1, 2, 3, 4, 5, 1, 2, 3, 4, 5, 1, 2, 3, 4, 5, 1, 2, 3, 4, 5…
$ Strength     7, 7, 15, 11, 9, 12, 17, 12, 18, 18, 14, 18, 18, 19, 19, 1…

data %>% 
  ggplot(aes(x=Cotton, y=Strength, group=Cotton)) +
  geom_boxplot()

The Tukey’s test is built by using the aov function and the TukeyHSD function. The results are then plotted by the plot method of the TukeyHSD object:

data %>% 
  aov(Strength ~ Cotton, data=.) %>%
  TukeyHSD() %>% 
  plot()

The biggest problem with plot.TukeyHSD is that the labels of the differences are often partially hidden if there are many groups or the plot is too squat. This is the main reason for implementing an analogous function based on GGplot2 in adas.utils.

Enter `ggTukey`

Labels are much more readable in the ggTukey plot. Let’s see how it works:

data %>% 
  aov(Strength ~ Cotton, data=.) %>%
  TukeyHSD() %>% 
  ggTukey()

There’s actually more: you can also pass to ggTukey the data frame and the formula to be used in the aov model:

data %>% 
  ggTukey(Strength ~ Cotton)

Which is nicely more tidiverse-y and readable, isn’t it?

Now let’s look at a different, slightly more complex dataset:

data <- examples_url("battery.dat") %>% 
  read.table(header=TRUE) %>% 
  mutate(Material=factor(LETTERS[Material]), Temperature=factor(Temperature)) %>%
  glimpse()

Rows: 36
Columns: 6
$ RunOrder       34, 25, 16, 7, 8, 1, 26, 36, 6, 13, 3, 31, 27, 29, 12, 1…
$ StandardOrder  1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 1…
$ Temperature    15, 70, 125, 15, 70, 125, 15, 70, 125, 15, 70, 125, 15, …
$ Material       A, A, A, B, B, B, C, C, C, A, A, A, B, B, B, C, C, C, A,…
$ Repeat         1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2, 2, 2, 2, 2, 2, 2, 2, 3,…
$ Response       130, 34, 20, 150, 136, 25, 138, 174, 96, 155, 40, 70, 18…

This one shows discharge test results for batteries with different dielectric materials (qualitative factors) and operating at different temperatures (quantitative factors). So inn this case we have two predictors. We want to create a set of Tukey’s tests for each level of Material. We need to pass ggTukey with the formula Response ~ Temperature (for the aov model) and the splt argument with another, one side formula that specifies the grouping factor: ~Material. In a terse and clear way, we can write:

data %>% 
  ggTukey(Response ~ Temperature, splt=~Material)

And that was not possible with the standard plot.TukeyHSD method, at least not in two lines of code.

That’s all, folks!

Footnotes

This function can load any data file listed on https://paolobosetti.quarto.pub/data↩︎

Moving to a new domain

Paolo Bosetti — Wed, 26 Feb 2025 00:00:00 GMT

Starting from February 26, 2025, RTUG website has a new domain under the University of Trento domain: rtug.unitn.it. So the group’s homepage is now accessible as https://rtug.unitn.it.

Furthermore, there is a new contact email address:

Note

The old address https://r-trento.github.io will continue to work, automatically redirecting to the new domain name.

That’s all, folks!

Introducing the adas.utils package

Paolo Bosetti — Mon, 10 Feb 2025 00:00:00 GMT

Factorial plans

The package provides tools for dealing with factorial plan according to the Design of Experiments (DoE) protocols. The functions for dealing with DoE have names starting with fp_. As much as possible, we are aiming at a tidyverse-like syntax, so that the functions can be used in a pipe.

We are following conventions and techniques illustrated in the book Design and Analysis of Experiments by Douglas C. Montgomery.

Full factorial plan

A hypercube, clearly impossible to represent in 2D

You can create a full factorial plan with the fp_design_matrix function, passing the number of factors:

(dm <- fp_design_matrix(2, rep=2) %>% 
  mutate(Y=rnorm(n())))
#>  Factorial Plan Design Matrix
#>  Defining Relationship:  ~ A * B 
#>  Factors:  A B 
#>  Levels:  -1 1 
#>  Fraction:  NA 
#>  Type:  plain 
#>  
#> # A tibble: 8 × 7
#>   StdOrder RunOrder .treat  .rep     A     B      Y
#>                 
#> 1        1        5 (1)        1    -1    -1  1.06 
#> 2        2        3 a          1     1    -1 -0.670
#> 3        3        6 b          1    -1     1 -0.800
#> 4        4        1 ab         1     1     1  1.27 
#> 5        5        8 (1)        2    -1    -1  0.417
#> 6        6        4 a          2     1    -1  2.29 
#> 7        7        2 b          2    -1     1 -1.71 
#> 8        8        7 ab         2     1     1  0.942

In this case, the factors are the first n capital letters.

If you want different factor names, use a right-side-only formula combining all the named factors with *:

fp_design_matrix(~Speed*Weight)
#>  Factorial Plan Design Matrix
#>  Defining Relationship:  ~ Speed * Weight 
#>  Factors:  Speed Weight 
#>  Levels:  -1 1 
#>  Fraction:  NA 
#>  Type:  plain 
#>  
#> # A tibble: 4 × 7
#>   StdOrder RunOrder .treat       .rep Speed Weight Y    
#>                      
#> 1        1        4 (1)             1    -1     -1 NA   
#> 2        2        3 speed           1     1     -1 NA   
#> 3        3        2 weight          1    -1      1 NA   
#> 4        4        1 speedweight     1     1      1 NA

NOTE, though, that using custom factor names is discouraged, and won’t work as expected if you are using the functions for dealing with fractional factorial plans, especially for the analysis of alias structures among factors.

The yield column Y must then be completed according to the randomized RunOrder column.

It is possible to add custom scales to the factors, and also add names to the factors:

fp_design_matrix(2) %>% 
  fp_add_names(A="Temperature", B="Pressure") %>%
  fp_add_scale(A=c(20, 25), B=c(75, 125), suffix=".scaled")
#>  Factorial Plan Design Matrix
#>  Defining Relationship:  ~ A * B 
#>  Factors:  A B 
#>  Levels:  -1 1 
#>  Fraction:  NA 
#>  Type:  plain 
#>  Scales suffix: .scaled
#>  Scaled factors:
#>     A.scaled: [20, 25]
#>     B.scaled: [75, 125]
#>  Factor names:
#>     A: Temperature
#>     B: Pressure
#>  
#> # A tibble: 4 × 9
#>   StdOrder RunOrder .treat  .rep     A     B A.scaled B.scaled Y    
#>                        
#> 1        1        4 (1)        1    -1    -1       20       75 NA   
#> 2        2        1 a          1     1    -1       25       75 NA   
#> 3        3        3 b          1    -1     1       20      125 NA   
#> 4        4        2 ab         1     1     1       25      125 NA

Custom levels

If you want a factorial plan with custom levels, pass the levels argument. In this case, though, the .treat column with Yates’ treatment codes would be NA:

fp_design_matrix(2, levels=-1:1)
#>  Factorial Plan Design Matrix
#>  Defining Relationship:  ~ A * B 
#>  Factors:  A B 
#>  Levels:  -1 0 1 
#>  Fraction:  NA 
#>  Type:  plain 
#>  
#> # A tibble: 9 × 6
#>   StdOrder RunOrder  .rep     A     B Y    
#>              
#> 1        1        5     1    -1    -1 NA   
#> 2        2        4     1     0    -1 NA   
#> 3        3        1     1     1    -1 NA   
#> 4        4        2     1    -1     0 NA   
#> 5        5        8     1     0     0 NA   
#> 6        6        6     1     1     0 NA   
#> 7        7        7     1    -1     1 NA   
#> 8        8        9     1     0     1 NA   
#> 9        9        3     1     1     1 NA

Augment a plan

You can augment a plan by adding a central point, typically repeated:

fp_design_matrix(3) %>%
  fp_augment_center(rep=4)
#>  Factorial Plan Design Matrix
#>  Defining Relationship:  ~ A * B * C 
#>  Factors:  A B C 
#>  Levels:  -1 1 
#>  Fraction:  NA 
#>  Type:  centered 
#>  
#> # A tibble: 12 × 8
#>    StdOrder RunOrder .treat  .rep     A     B     C Y    
#>                  
#>  1        1        2 (1)        1    -1    -1    -1 NA   
#>  2        2        8 a          1     1    -1    -1 NA   
#>  3        3        5 b          1    -1     1    -1 NA   
#>  4        4        1 ab         1     1     1    -1 NA   
#>  5        5        6 c          1    -1    -1     1 NA   
#>  6        6        4 ac         1     1    -1     1 NA   
#>  7        7        3 bc         1    -1     1     1 NA   
#>  8        8        7 abc        1     1     1     1 NA   
#>  9        9       12 center     1     0     0     0 NA   
#> 10       10        9 center     2     0     0     0 NA   
#> 11       11       11 center     3     0     0     0 NA   
#> 12       12       10 center     4     0     0     0 NA

Then if needed (because the analysis show low p-value for the quadratic term) you can add axial points to get a central composite design:

fp_design_matrix(3) %>% 
  fp_augment_center(rep=3) %>% 
  fp_augment_axial(rep=2)
#>  Factorial Plan Design Matrix
#>  Defining Relationship:  ~ A * B * C 
#>  Factors:  A B C 
#>  Levels:  -1 1 
#>  Fraction:  NA 
#>  Type:  composite 
#>  
#> # A tibble: 23 × 8
#>    StdOrder RunOrder .treat  .rep     A     B     C Y    
#>                  
#>  1        1        5 (1)        1    -1    -1    -1 NA   
#>  2        2        3 a          1     1    -1    -1 NA   
#>  3        3        2 b          1    -1     1    -1 NA   
#>  4        4        7 ab         1     1     1    -1 NA   
#>  5        5        6 c          1    -1    -1     1 NA   
#>  6        6        8 ac         1     1    -1     1 NA   
#>  7        7        4 bc         1    -1     1     1 NA   
#>  8        8        1 abc        1     1     1     1 NA   
#>  9        9       11 center     1     0     0     0 NA   
#> 10       10       10 center     2     0     0     0 NA   
#> # ℹ 13 more rows

Full example

Let’s see a full example using the ccd_experiment_yield dataset, which contains a list of the yield data for three sequential experiments in a central composite design.

First, we design a factorial plan, with two factors and two levels each:

fp <- fp_design_matrix(2, rep=3)

Ideally, we would then sort the table according to the RunOrder column, and complete the Y column with the yield data from the the real experiments. For the sake of documenting the package, we can directly add the yield data from the base field of the ccd_experiment_yield dataset (which holds values in standard Yates’ order):

fp$Y <- ccd_experiment_yield$base

Now we can fit a linear model to the data, and check the p-values of the ANOVA:

fp %>% 
  lm(Y ~ A*B, data=.) %>% 
  anova()
#> Analysis of Variance Table
#> 
#> Response: Y
#>           Df  Sum Sq Mean Sq F value    Pr(>F)    
#> A          1  49.331  49.331  105.06 7.058e-06 ***
#> B          1 131.224 131.224  279.46 1.659e-07 ***
#> A:B        1 216.931 216.931  461.99 2.311e-08 ***
#> Residuals  8   3.756   0.470                      
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

All factors and their interactions are significant. But is the two-level model enough? Let’s check for the quadratic terms, by augmenting the plan with a central point repeated 4 times. We also load the center field from the ccd_experiment_yield dataset:

fpc <- fp %>% 
  fp_augment_center(rep=4)

fpc$Y[fpc$.treat == "center"] <- ccd_experiment_yield$center

Now we can fit a model with the quadratic term, using either or : since we only have a central point, we cannot discriminate which factor is contributing to the curvature in the response surface. We get:

fpc %>% 
  lm(Y ~ A*B+I(A^2), data=.) %>% 
  anova()
#> Analysis of Variance Table
#> 
#> Response: Y
#>           Df  Sum Sq Mean Sq F value    Pr(>F)    
#> A          1  49.331  49.331  96.801 8.695e-07 ***
#> B          1 131.224 131.224 257.494 5.592e-09 ***
#> I(A^2)     1  15.204  15.204  29.834 0.0001972 ***
#> A:B        1 216.931 216.931 425.673 3.827e-10 ***
#> Residuals 11   5.606   0.510                      
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

So the contribution of the quadratic term is significant. This means that we have to further augment the plan with axial points and investigate a Central Composite Design (CCD). Note: if the quadratic term contribution were not significant, we would have to remove the quadratic term from the model and accept the two-level model.

So let’s load the axial points from the axial field of the ccd_experiment_yield dataset, and fit a model with the quadratic terms and their interactions:

fpccd <- fpc %>% 
  fp_augment_axial(rep=2)

fpccd$Y[fpccd$.treat == "axial"] <- ccd_experiment_yield$axial

fpccd %>% 
  lm(Y ~ A*B*I(A^2)*I(B^2), data=.) %>% 
  anova()
#> Analysis of Variance Table
#> 
#> Response: Y
#>               Df  Sum Sq Mean Sq  F value    Pr(>F)    
#> A              1  75.196  75.196  94.1952 7.395e-08 ***
#> B              1 194.702 194.702 243.8964 1.097e-10 ***
#> I(A^2)         1 101.355 101.355 126.9638 1.017e-08 ***
#> I(B^2)         1   3.551   3.551   4.4479   0.05216 .  
#> A:B            1 216.931 216.931 271.7423 5.087e-11 ***
#> A:I(A^2)       1   0.235   0.235   0.2945   0.59530    
#> B:I(A^2)       1   1.046   1.046   1.3106   0.27022    
#> I(A^2):I(B^2)  1   0.490   0.490   0.6142   0.44542    
#> Residuals     15  11.974   0.798                       
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

So we can finally state that a proper model would be Y ~ A*B+I(A^2):

fpccd %>% 
  lm(Y ~ A*B+I(A^2), data=.) %>% 
  summary()
#> 
#> Call:
#> lm(formula = Y ~ A * B + I(A^2), data = .)
#> 
#> Residuals:
#>      Min       1Q   Median       3Q      Max 
#> -1.64925 -0.60624  0.00919  0.65165  1.67506 
#> 
#> Coefficients:
#>             Estimate Std. Error t value Pr(>|t|)    
#> (Intercept)   1.0156     0.3061   3.318  0.00362 ** 
#> A             1.9390     0.2134   9.088 2.40e-08 ***
#> B             3.1201     0.2134  14.624 8.59e-12 ***
#> I(A^2)        2.9905     0.2834  10.552 2.20e-09 ***
#> A:B           4.2518     0.2754  15.437 3.32e-12 ***
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
#> 
#> Residual standard error: 0.9541 on 19 degrees of freedom
#> Multiple R-squared:  0.9714, Adjusted R-squared:  0.9654 
#> F-statistic: 161.5 on 4 and 19 DF,  p-value: 2.191e-14

Save to/load from a file

Once the design matrix is prepared, you typically want to save it to a file and use it for collecting data form experiments. You can use the write.csv function, but it is recommended to use the fp_write_csv function, which will also save the design matrix properties as comments:

dm <-  fp_design_matrix(2) %>% 
  fp_add_names(A="Temperature", B="Pressure") %>% 
  fp_add_scale(A=c(2, 12), B=c(40, 60), suffix="_s") %>%
  fp_write_csv("design_matrix.csv")

Note that the fp_write_csv function invisibly returns the same design matrix, so you can use it in a pipe chain. Also, the CVS files has the rows arranged in the same order as the RunOrder column (i.e. randomized).

Once the CSV file has been completed, you can load it back into R using the fp_read_csv function:

dm <- dm %>% 
  fp_read_csv("design_matrix.csv")

Note that fp_read_csv returns the design matrix reordered according to Yates’ standard order.

Fractional factorial plan

It is possible to divide a design matrix into a fractional factorial plan using the fp_fraction function. The fraction uses a defining relationship (dr) as , which is mapped in R as a one side formula ~A*B*C*D.

Any fraction is added to the factorial.plan object in the fraction attribute.

A full factorial plan can be reduced to a fractional factorial plan by applying the fp_fraction function times. For example, to get a plan with the defining relationships and :

fp_design_matrix(5) %>% 
  fp_fraction(~A*B*C*D) %>% 
  fp_fraction(~B*C*D*E)
#>  Factorial Plan Design Matrix
#>  Defining Relationship:  ~ A * B * C * D * E 
#>  Factors:  A B C D E 
#>  Levels:  -1 1 
#>  Fraction:  I=ABCD I=BCDE 
#>  Type:  fractional 
#>  
#> # A tibble: 8 × 12
#>   StdOrder RunOrder .treat  .rep     A     B     C     D     E Y      ABCD  BCDE
#>                     
#> 1        1        7 (1)        1    -1    -1    -1    -1    -1 NA        1     1
#> 2        7       29 bc         1    -1     1     1    -1    -1 NA        1     1
#> 3       11       28 bd         1    -1     1    -1     1    -1 NA        1     1
#> 4       13       23 cd         1    -1    -1     1     1    -1 NA        1     1
#> 5       20       20 abe        1     1     1    -1    -1     1 NA        1     1
#> 6       22       16 ace        1     1    -1     1    -1     1 NA        1     1
#> 7       26       26 ade        1     1    -1    -1     1     1 NA        1     1
#> 8       32       10 abcde      1     1     1     1     1     1 NA        1     1

Note that with the remove option you can control if you want to keep both fractions, and later on filter(ABC==1) them out.

fp_design_matrix(3) %>% 
  fp_fraction(~A*B*C, remove=FALSE)
#>  Factorial Plan Design Matrix
#>  Defining Relationship:  ~ A * B * C 
#>  Factors:  A B C 
#>  Levels:  -1 1 
#>  Fraction:  I=ABC 
#>  Type:  fractional 
#>  
#> # A tibble: 8 × 9
#>   StdOrder RunOrder .treat  .rep     A     B     C Y       ABC
#>                  
#> 1        1        8 (1)        1    -1    -1    -1 NA       -1
#> 2        2        6 a          1     1    -1    -1 NA        1
#> 3        3        3 b          1    -1     1    -1 NA        1
#> 4        4        4 ab         1     1     1    -1 NA       -1
#> 5        5        2 c          1    -1    -1     1 NA        1
#> 6        6        7 ac         1     1    -1     1 NA       -1
#> 7        7        5 bc         1    -1     1     1 NA       -1
#> 8        8        1 abc        1     1     1     1 NA        1

Also, note that the remove option is sticky, so that when you can apply the fp_fraction function multiple times and the first time has the option set to remove=FALSE, then all the following fp_fraction calls will have the same option set to FALSE. Setting remove=FALSE to any of the following calls can have unexpected behavior.

Alias structure

Any fraction of a factorial plan results in a set of aliases among effects. The package provides the following functions to deal with alias structures:

fp_alias_matrix: returns a matrix with the alias structure of the factors in the design matrix. The alias matrix has a plot method.
fp_all_drs: given a set of defining relationships, returns the dependent one.
fp_merge_drs: given a set of defining relationships, returns the merged one, i.e. the one having all the factors.
fp_gen2alias: given a generator (i.e. the right side of a DR) and an effect name as strings, calculates the resulting alias.

For example:

(am <- fp_alias_matrix(~A*B*C, ~B*C*D))
#> Defining relationships:
#>  I=ABC I=BCD I=AD 
#> 
#>      A B AB C AC BC ABC D AD BD ABD CD ACD BCD ABCD
#> A    0 0  0 0  0  1   0 3  0  0   0  0   0   0    2
#> B    0 0  0 0  1  0   0 0  0  0   3  2   0   0    0
#> AB   0 0  0 1  0  0   0 0  0  3   0  0   2   0    0
#> C    0 0  1 0  0  0   0 0  0  2   0  0   3   0    0
#> AC   0 1  0 0  0  0   0 0  0  0   2  3   0   0    0
#> BC   1 0  0 0  0  0   0 2  0  0   0  0   0   0    3
#> ABC  0 0  0 0  0  0   0 0  2  0   0  0   0   3    0
#> D    3 0  0 0  0  2   0 0  0  0   0  0   0   0    1
#> AD   0 0  0 0  0  0   2 0  0  0   0  0   0   1    0
#> BD   0 0  3 2  0  0   0 0  0  0   0  0   1   0    0
#> ABD  0 3  0 0  2  0   0 0  0  0   0  1   0   0    0
#> CD   0 2  0 0  3  0   0 0  0  0   1  0   0   0    0
#> ACD  0 0  2 3  0  0   0 0  0  1   0  0   0   0    0
#> BCD  0 0  0 0  0  0   3 0  1  0   0  0   0   0    0
#> ABCD 2 0  0 0  0  3   0 1  0  0   0  0   0   0    0

am %>% plot()

The design matrix can be converted to a tibble thanks to the proper as_tibble.design.matrix S3 method:

am %>% as_tibble()
#> # A tibble: 42 × 3
#>    Effect.x Effect.y generator
#>                
#>  1 A        BC       ABC      
#>  2 A        D        AD       
#>  3 A        ABCD     BCD      
#>  4 B        AC       ABC      
#>  5 B        ABD      AD       
#>  6 B        CD       BCD      
#>  7 AB       C        ABC      
#>  8 AB       BD       AD       
#>  9 AB       ACD      BCD      
#> 10 C        AB       ABC      
#> # ℹ 32 more rows

Statistics

The package also provides some useful functions for statistical analysis of data.

Plotting

Normal probability plot

The normal probability plot is provided as an alternative to the quantile-quantile plot:

df <- tibble(
  xn = rnorm(100, mean=20, sd=5),
  xu = runif(100, min=0, max=40)
)

df %>% normplot(xn)

df %>% normplot(xu)

Pareto chart

The Pareto chart is a bar chart that displays the relative importance of problems in a format that is very easy to interpret. The bars are sorted in descending order, and the cumulative percentage of the total is shown by the line.

It can prove useful in the context of factorial plans, to identify the most important factors, or in sensitivity analysis, to identify the most important parameters.

The package provides a generic function, pareto_chart, that can be used with a tibble (or a data frame), or with a linear model (an lm object). In the latter case, the function produces the Pareto chart of the model effects.

For the general case, when you have a tibble with values and names:

set.seed(1)
tibble(
  val=rnorm(10, sd=5),
  cat=LETTERS[1:length(val)]
  ) %>%
  pareto_chart(labels=cat, values=val)
#> Warning in pareto_chart.data.frame(., labels = cat, values = val): 'pareto_chart.data.frame' is deprecated.
#> Use 'geom_pareto' instead.
#> See help("Deprecated")

For the case of a linear model:

filtration %>% 
  lm(Y~A*B*C*D, data=.) %>%
  pareto_chart()

Daniel’s plot

In case of non-replicated factorial plans, the Daniel’s plot can be used to identify the most important factors: a quantile-quantile plot of the factors effects shows the significant factors and interactions off the diagonal.

daniel_plot_qq(lm(Y~A*B*C*D, data=filtration))

If you prefer, you can rather use a half-normal plot:

filtration %>% 
  lm(Y~A*B*C*D, data=.) %>%
  daniel_plot_hn(nlab=6, repel=TRUE)

It shows that none of the effects containing the B factor are significant, so we can reduce the model to Y~A*C*D:

filtration %>% 
  lm(Y~A*C*D, data=.) %>%
  anova()
#> Analysis of Variance Table
#> 
#> Response: Y
#>           Df  Sum Sq Mean Sq F value    Pr(>F)    
#> A          1 1870.56 1870.56 83.3677 1.667e-05 ***
#> C          1  390.06  390.06 17.3844 0.0031244 ** 
#> D          1  855.56  855.56 38.1309 0.0002666 ***
#> A:C        1 1314.06 1314.06 58.5655 6.001e-05 ***
#> A:D        1 1105.56 1105.56 49.2730 0.0001105 ***
#> C:D        1    5.06    5.06  0.2256 0.6474830    
#> A:C:D      1   10.56   10.56  0.4708 0.5120321    
#> Residuals  8  179.50   22.44                      
#> ---
#> Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1

Even better, the model can be further reduced to Y~A*C+A*D. Compare this conclusion with the last Pareto chart above.

Utilities

This package also provides a function for easily loading data files made available on the accompanying course documentation on https://paolobosetti.quarto.pub/data:

examples_url("battery.dat") %>%  read.table(header=TRUE)
#>    RunOrder StandardOrder Temperature Material Repeat Response
#> 1        34             1          15        1      1      130
#> 2        25             2          70        1      1       34
#> 3        16             3         125        1      1       20
#> 4         7             4          15        2      1      150
#> 5         8             5          70        2      1      136
#> 6         1             6         125        2      1       25
#> 7        26             7          15        3      1      138
#> 8        36             8          70        3      1      174
#> 9         6             9         125        3      1       96
#> 10       13            10          15        1      2      155
#> 11        3            11          70        1      2       40
#> 12       31            12         125        1      2       70
#> 13       27            13          15        2      2      188
#> 14       29            14          70        2      2      122
#> 15       12            15         125        2      2       70
#> 16       14            16          15        3      2      110
#> 17       30            17          70        3      2      120
#> 18       24            18         125        3      2      104
#> 19        5            19          15        1      3       74
#> 20       21            20          70        1      3       80
#> 21       10            21         125        1      3       82
#> 22       35            22          15        2      3      159
#> 23       19            23          70        2      3      106
#> 24       28            24         125        2      3       58
#> 25       18            25          15        3      3      168
#> 26       20            26          70        3      3      150
#> 27       33            27         125        3      3       82
#> 28       11            28          15        1      4      180
#> 29       23            29          70        1      4       75
#> 30       17            30         125        1      4       58
#> 31       15            31          15        2      4      126
#> 32       32            32          70        2      4      115
#> 33        4            33         125        2      4       45
#> 34       22            34          15        3      4      160
#> 35        2            35          70        3      4      139
#> 36        9            36         125        3      4       60

That’s all, folks!

Logarithmic scales in GGPlot2

Paolo Bosetti — Sun, 09 Feb 2025 00:00:00 GMT

Rationale

Recently I had to port in R some Matlab code using the tf and bode functions, which are respectively used to calculate the transfer function and create a Bode plot of it.

The R package control luckily provides the analogous functions, although its control::bodeplot function uses the base plot interface. Of course, I wanted to make an analogous plot with GGplot2 tools.

Example: vibration isolation

Let us use the example for a lumped parameters model of a vibration isolation system, that is, a 1-DoF mass-spring-damper system. Briefly, its transfer function can be created as:

M <- 10
K <- 1000
C <- 50

(H <- tf(c(C/K, 1),c(M/K, C/K, 1)))


y1:
        0.05 s^1 + 1 
  - - - - - - - - - - - - -
    0.01 s^2 + 0.05 s + 1 


Transfer Function: Continuous time model

Its Bode representation can be obtained by:

bode(H) %>% str()

List of 3
 $ w    : num [1:10000] 0 0.01 0.02 0.03 0.04 ...
 $ mag  : num [1:10000] 0.00 8.69e-06 3.48e-05 7.82e-05 1.39e-04 ...
 $ phase: num [1:10000] 0.00 -2.87e-08 -2.29e-07 -7.74e-07 -1.83e-06 ...

That is, a list of three vectors reporting frequency (w), magnitude in dB (mag), and phase in degrees (phase). Why the control developers decided to return a list of equally sized vectors rather a data frame is beyond me, but let’s deal with what we have.

bode(H, w=1:1e4) %>% {
  tibble(
    `frequency (rad/s)` = .$w, 
    `magnitude (dB)` = .$mag,
    `phase (deg)` = .$phase
  )
} %>% 
  head() %>% 
  knitr::kable()

frequency (rad/s)	magnitude (dB)	phase (deg)
1	0.0870762	-0.0288644
2	0.3509189	-0.2362699
3	0.7993794	-0.8294252
4	1.4452744	-2.0825653
5	2.3044892	-4.3987054
6	3.3880407	-8.4155907

To make the Bode plot, which reports magnitude vs. frequency on top of phase vs. frequency, we make the tibble tidy and use facet_wrap:

bode(H, w=1:1e4) %>% {
  tibble(
    `frequency (rad/s)` = .$w, 
    `magnitude (dB)` = .$mag,
    `phase (deg)` = .$phase
  )
} %>%
  pivot_longer(-`frequency (rad/s)`) %>% 
  ggplot(aes(x=`frequency (rad/s)`, y=value)) +
  geom_line() +
  geom_point(alpha = 0.2) +
  facet_wrap(~name, nrow=2, scales="free") +
  scale_x_log10()

Note the followings:

in facet_wrap, we use the option scales="free": this allows to independently rescale the axes of each facet;
the horizontal axis is logarithmic, but there is only one secondary grid line, while we usually have secondary gridlines at 2, 3, …, 9;
the point density is not evenly spaced on the logarithmic axis.

So, here we want to tackle the problems in 2. and 3..

Log-tickmarks: Solution

The scale_x_log10() function allows to specify the breaks and the minor_breaks, which we can exploit to fix the gridlines. How can we get a log-spaced sequence? an elegant solution uses the outer product %o% of two vectors: that of the ticks, and that of the orders of magnitude (ooms):

ticks <- 2:9
ooms <- 10^seq(0, 4)

ticks %o% ooms

     [,1] [,2] [,3] [,4]  [,5]
[1,]    2   20  200 2000 20000
[2,]    3   30  300 3000 30000
[3,]    4   40  400 4000 40000
[4,]    5   50  500 5000 50000
[5,]    6   60  600 6000 60000
[6,]    7   70  700 7000 70000
[7,]    8   80  800 8000 80000
[8,]    9   90  900 9000 90000

Looking at the columns , in sequence, we have what we want, so:

(breaks <- as.vector(ticks %o% ooms))

 [1]     2     3     4     5     6     7     8     9    20    30    40    50
[13]    60    70    80    90   200   300   400   500   600   700   800   900
[25]  2000  3000  4000  5000  6000  7000  8000  9000 20000 30000 40000 50000
[37] 60000 70000 80000 90000

bode(H, w=1:1e4) %>% {
  tibble(
    `frequency (rad/s)` = .$w, 
    `magnitude (dB)` = .$mag,
    `phase (deg)` = .$phase
  )
} %>%
  pivot_longer(-`frequency (rad/s)`) %>% 
  ggplot(aes(x=`frequency (rad/s)`, y=value)) +
  geom_line() +
  geom_point(alpha = 0.2) +
  facet_wrap(~name, nrow=2, scales="free") +
  scale_x_log10(minor_breaks=breaks, labels=scales::scientific)

Bingo!

Now, we still have too sparse points to the left, and definitely too many points to the right. We can use a similar approach to define the frequency vector.

Log-spaced points: Solution

We use the same outer product trick, multiplying the vector of orders of magnitude by a vector of positions, exponentially spaced within each magnitude:

pts <- 10^seq(0, 1, 0.1) %>% tail(-1)
(freqs <- as.vector(pts %o% ooms)) %>% head(n=20)

 [1]   1.258925   1.584893   1.995262   2.511886   3.162278   3.981072
 [7]   5.011872   6.309573   7.943282  10.000000  12.589254  15.848932
[13]  19.952623  25.118864  31.622777  39.810717  50.118723  63.095734
[19]  79.432823 100.000000

Finally:

pts <- 10^seq(0, 1, 0.01) %>% tail(-1)
freqs <- as.vector(pts %o% ooms)

bode(H, w=freqs) %>% {
  tibble(
    `frequency (rad/s)` = .$w, 
    `magnitude (dB)` = .$mag,
    `phase (deg)` = .$phase
  )
} %>%
  pivot_longer(-`frequency (rad/s)`) %>% 
  ggplot(aes(x=`frequency (rad/s)`, y=value)) +
  geom_line() +
  geom_point(alpha = 0.2) +
  facet_wrap(~name, nrow=2, scales="free") +
  scale_x_log10(minor_breaks=breaks, labels=scales::scientific)

Putting all together

We can then put everything together and make a useful function (note that we are now converting the frequencies to Hz):

ggbodeplot <- function(tf, fmin=1, fmax=1e4, df=0.01) {
  ticks <- 2:9
  pts <- 10^seq(0, 1, df) %>% tail(-1)
  ooms <- 10^(floor(log10(fmin)):ceiling(log10(fmax)-1))
  breaks <- as.vector(ticks %o% ooms)
  freqs <- as.vector(pts %o% ooms)
  
  bode(tf, freqs*2*pi) %>% {
    tibble(f=.$w/(2*pi), `magnitude (dB)`=.$mag, `phase (deg)`=.$phase)} %>% 
    pivot_longer(-f) %>% 
    ggplot(aes(x=f, y=value)) +
    geom_line() +
    scale_x_log10(minor_breaks=breaks, labels=scales::scientific) +
    facet_wrap(~name, nrow=2, scales="free") +
    labs(x="frequency (Hz)")
}

H %>% ggbodeplot(fmin=0.1, fmax=100)

One last thing…

Well, the above is mostly of academic interest, at least for what pertains the logarithmic grid lines: it shows a nice and useful way for creating regularly spaced vectors, which is really useful to evenly distribute abscissa values when the axis scale is not linear. But thanks to the scales package there is a quick way for having any axis, whichever the scale, with a number of minor breaks different to 1 (the defaiult in GGplot). In fact, we can just use the scales::minor_breaks_n() function to generate minor grid lines at will:

ggbodeplot <- function(tf, fmin=1, fmax=1e4, df=0.01) {
  pts <- 10^seq(0, 1, df) %>% tail(-1)
  ooms <- 10^(floor(log10(fmin)):ceiling(log10(fmax)-1))
  freqs <- as.vector(pts %o% ooms)
  
  bode(tf, freqs*2*pi) %>% {
    tibble(f=.$w/(2*pi), `magnitude (dB)`=.$mag, `phase (deg)`=.$phase)} %>% 
    pivot_longer(-f) %>% 
    ggplot(aes(x=f, y=value)) +
    geom_line() +
    scale_x_log10(
      minor_breaks=scales::minor_breaks_n(10), 
      labels= ~ latex2exp::TeX(paste0("$10^{", log10(.), "}$"))
    ) +
    facet_wrap(~name, nrow=2, scales="free") +
    labs(x="frequency (Hz)")
}

H %>% ggbodeplot(fmin=0.1, fmax=100)

where the trick is to set scale_x_log10(minor_breaks=scales::minor_breaks_n(10)). Note that the argument is the number of intervals rather than the number of grid lines (so, 10 rather than 9). As a final suggestion, try and use the same command with a scale_x_continuous: it works whichever is the axis transformation (including identity). Also, note the labels lambda function used for formatting tick labels.

That’s all, folks!

New Logo

Paolo Bosetti — Thu, 06 Feb 2025 00:00:00 GMT

We have a new group logo

RTUG logo

The new RTUG logo is a hexagon, on the style of Tidyverse packages, it takes the colors of Trentino Province lettering, and of course it incorporates the GNU-R language logo.

First RTUG meeting

Paolo Bosetti — Wed, 05 Feb 2025 00:00:00 GMT

The first R-Trento Users Meeting, code RTUG::25.1, is planned for Friday, May 30, 2025 at 14:30 in the Aula Rossa of the Department of Economics and Management, via Inama 5, Trento.

Participation is free but you need to subscribe to the event via .

Certificates of participation will be available upon request

Topics

Event flyer

Download here the event flyer.

Welcome and Introduction to the R-Trento Users Group (P. Bosetti)
The R Package rmf in Teaching (R. Micciolo and G. Espa)
The adas.utils Package: Design of Experiments, the Tidy Way (P. Bosetti)
Bioinformatics and R: Visualizing Genomic Data (Y. Ciani)
Spatial Sampling in R (M. M. Dickson)
Refreshments and networking!

See below for details on the topics. The slides page lists all the available presentations.

Venue

Aula Rossa of the Department of Economics and Management, via Inama 5, Trento.

Details on Topics

Welcome

Author: P. Bosetti, University of Trento
Style: Presentation

The R Package `rmf` in Teaching

Authors: R. Micciolo and G. Espa, University or Trento
Style: Presentation

We introduce the rmf package, which contains a series of functions developed primarily for use in educational settings, particularly in basic university statistics courses.

The `adas.utils` Package: Design of Experiments, the Tidy Way

Author: P. Bosetti, University or Trento
Style: Presentation

We introduce the adas.utils package, which provides a set of tools for designing factorial designs and analyzing results, in line with the Tidyverse package approach.

Bioinformatics and R: Visualizing Genomic Data

Author: Y. Ciani, University or Trento
Style: Presentation

An overview of the use of R in bioinformatics, particularly its use for visualizing genomic data.

Spatial Sampling in R

Author: M. M. Dickson, University of Padua
Style: Presentation

We introduce the BalancedSampling package, which allows the selection of balanced and spatially balanced samples, widely used in the environmental/forestry field, with recent diffusion also in the economic-social field.

Post Mortem 😜

So RTUG::25.1 is over, and it was a great success! We had a good turnout, with more than 20 participants, and the feedback was very positive. The presentations were well received, and the discussions were lively.

We would like to thank all the speakers for their contributions, and all the participants for their interest and enthusiasm. We hope to see you again at the next RTUG meeting, which will be held in the fall of 2025.

Opening!

Paolo Bosetti — Wed, 05 Feb 2025 00:00:00 GMT

The Users Group is instantiated

An important milestone

On Feb. 5th, 2025, Giuseppe Espa, Maria Michela Dickson, Flavio Santi, Diego Giuliani and Paolo Bosetti had the first founding meeting of the R-Trento Users Group.

RTUG <- list("Giuseppe", "Maria Michela", "Flavio", "Diego", "Paolo")

That’s a milestone.

RTUG

mdbplyr: a tidy interface to MongoDB

Introduction

Usage examples

Installation

Loading data

Basic inspection

Supported verbs

filter()

select()

rename()

mutate()

transmute()

arrange()

group_by()

summarise()

slice_head()

head()

Notes

RTUG::25.2 meeting

Topics

Venue

Details on Topics

Tukey vs. Student

Repeated T-test vs. Tukey’s test

The dataset

Inference on Strength

The Family-Wise Error Rate

R-Consortium supporting RTUG

We’re supported by the R-Consortium

Tukey’s test plot in adas.utils

Rationale

Enter ggTukey

Footnotes

Moving to a new domain

Introducing the adas.utils package

Factorial plans

Full factorial plan

Custom levels

Augment a plan

Full example

Save to/load from a file

Fractional factorial plan

Alias structure

Statistics

Plotting

Normal probability plot

Pareto chart

Daniel’s plot

Utilities

Logarithmic scales in GGPlot2

Rationale

Example: vibration isolation

Log-tickmarks: Solution

Log-spaced points: Solution

Putting all together

One last thing…

New Logo

We have a new group logo

First RTUG meeting

Topics

Venue

Details on Topics

Welcome

The R Package rmf in Teaching

The adas.utils Package: Design of Experiments, the Tidy Way

Bioinformatics and R: Visualizing Genomic Data

Spatial Sampling in R

Post Mortem 😜

Opening!

The Users Group is instantiated

`filter()`

`select()`

`rename()`

`mutate()`

`transmute()`

`arrange()`

`group_by()`

`summarise()`

`slice_head()`

`head()`

Inference on `Strength`

Enter `ggTukey`

The R Package `rmf` in Teaching

The `adas.utils` Package: Design of Experiments, the Tidy Way