Skip to contents

Creating a Makefile

Source: vignettes/articles/a3_makefile.Rmd
a3_makefile.Rmd

Introduction

Functions extract_make() and makefile() help with writing a Makefile to control a data analysis workflow. extract_make() and makefile() do much of the work, but some extra hand coding is also necessary.

An example workflow

To demonstrate, we use a simple workflow where all the code, aside from the shell script, has already been written. The project directory contains the following files:

.
├── data
│   └── raw_data.csv
├── out
├── report.qmd
└── src
    ├── cleaned_data.R
    ├── fig_fitted.R
    ├── model.R
    └── vals_fitted.R
  1. src/cleaned_data.R reads in the raw data, cleans it, and creates out/cleaned_data.rds
  2. src/model.R fits a model to the cleaned data, and creates out/model.rds
  3. src/vals_fitted.R extracts fitted values from the model, and creates out/vals_fitted.rds
  4. src/fig_fitted.R uses out/vals_fitted.rds to create the plot in out/fig_fitted.png
  5. report.qmd uses out/fig_fitted.png to create the document report.html

cleaned_data.R, model.R, vals_fitted.R, and fig_fitted.R all contain calls to cmd_assign(). For instance, src/model.R contains the lines

cmd_assign(.cleaned_data = "out/cleaned_data.rds",
           rlm_method = "M",
           .out = "out/model.rds")

extract_make()

extract_make() extracts the call to cmd_assign() from an R file, and turns it into a Makefile rule. For instance, the call

extract_make("src/model.R")

has return value

out/model.rds: src/model.R \
  out/cleaned_data.rds
    Rscript $^ $@ --rlm_method=M

In the call to cmd_assign() above, the names .cleaned_data and .out start with a dot, and the name rlm_method does not. extract_make() decides which arguments are file names in two steps. If the call includes any name that starts with a dot, those arguments are the file names, and the others are named arguments such as --rlm_method=M. If the call includes no dotted names, a value is treated as a file name when the file exists, or when the value looks like a path, such as when it contains a slash or ends with an extension such as .csv.

extract_make() and makefile() also read calls to cmd_assign_collect(). A character vector of paths becomes one file name per element, in the order written. The last element is the target, and the earlier elements are prerequisites.

The easiest way to use extract_make() is to call it from the R console, and then cut-and-paste the results into Makefile.

makefile()

makefile() creates a draft of the whole Makefile. makefile() loops through all the R files in a directory, extracting cmd_assign() and cmd_assign_collect() calls and converting them into Makefile rules. It then puts the rules into a Makefile. The rules follow the order of the file names, not the order in which they need to run.

For instance, the call 

makefile("src")

creates a new file called Makefile with the lines


.PHONY: all
all:


out/cleaned_data.rds: src/cleaned_data.R \
  data/raw_data.csv
    Rscript $^ $@

out/fig_fitted.png: src/fig_fitted.R \
  out/vals_fitted.rds
    Rscript $^ $@

out/model.rds: src/model.R \
  out/cleaned_data.rds
    Rscript $^ $@ --rlm_method=M

out/vals_fitted.rds: src/vals_fitted.R \
  out/cleaned_data.rds \
  out/model.rds
    Rscript $^ $@


.PHONY: clean
clean:
    rm -rf out
    mkdir out

The output from makefile() needs some editing before it is ready for use. We need to a rule for creating the report, and make that report a prerequisite for all: at the top of the file:


.PHONY: all
all: report.html


out/cleaned_data.rds: src/cleaned_data.R \
  data/raw_data.csv
    Rscript $^ $@

out/fig_fitted.png: src/fig_fitted.R \
  out/vals_fitted.rds
    Rscript $^ $@

out/model.rds: src/model.R \
  out/cleaned_data.rds
    Rscript $^ $@ --rlm_method=M

out/vals_fitted.rds: src/vals_fitted.R \
  out/cleaned_data.rds \
  out/model.rds
    Rscript $^ $@

report.html: report.qmd \
  out/fig_fitted.png
    quarto render $<


.PHONY: clean
clean:
    rm -rf out
    mkdir out

The ordering of the rules in the Makefile reflects the ordering of the files in the src directory, not the order in which the rules should be executed. This does not matter to make, which constructs its own dependency map. But for the benefit of human readers, we rearrange the rules to match the execution order.

.PHONY: all
all: report.html


out/cleaned_data.rds: src/cleaned_data.R \
  data/raw_data.csv
    Rscript $^ $@

out/model.rds: src/model.R \
  out/cleaned_data.rds
    Rscript $^ $@ --rlm_method=M

out/vals_fitted.rds: src/vals_fitted.R \
  out/cleaned_data.rds \
  out/model.rds
    Rscript $^ $@

out/fig_fitted.png: src/fig_fitted.R \
  out/vals_fitted.rds
    Rscript $^ $@


report.html: report.qmd \
  out/fig_fitted.png
    quarto render $<


.PHONY: clean
clean:
    rm -rf out
    mkdir out

Running the Makefile

Our project directory looks like this.

.
├── Makefile
├── data
│   └── raw_data.csv
├── out
├── report.qmd
└── src
    ├── cleaned_data.R
    ├── fig_fitted.R
    ├── model.R
    └── vals_fitted.R

We run the Makefile.

make
Rscript src/cleaned_data.R data/raw_data.csv out/cleaned_data.rds
✔ Assigned object `.raw_data` with value "data/raw_data.csv" and class "character".
✔ Assigned object `.out` with value "out/cleaned_data.rds" and class "character".
Rscript src/model.R out/cleaned_data.rds out/model.rds --rlm_method=M
✔ Assigned object `.cleaned_data` with value "out/cleaned_data.rds" and class "character".
✔ Assigned object `rlm_method` with value "M" and class "character".
✔ Assigned object `.out` with value "out/model.rds" and class "character".
Rscript src/vals_fitted.R out/cleaned_data.rds out/model.rds out/vals_fitted.rds
✔ Assigned object `.cleaned_data` with value "out/cleaned_data.rds" and class "character".
✔ Assigned object `.model` with value "out/model.rds" and class "character".
✔ Assigned object `.out` with value "out/vals_fitted.rds" and class "character".
Rscript src/fig_fitted.R out/vals_fitted.rds out/fig_fitted.png
✔ Assigned object `.vals_fitted` with value "out/vals_fitted.rds" and class "character".
✔ Assigned object `.out` with value "out/fig_fitted.png" and class "character".
null device 
          1 
quarto render report.qmd


processing file: report.qmd
1/3                  
2/3 [unnamed-chunk-1]
3/3                  
output file: report.knit.md

pandoc 
  to: html
  output-file: report.html
  standalone: true
  section-divs: true
  html-math-method: mathjax
  wrap: none
  default-image-extension: png
  
metadata
  document-css: false
  link-citations: true
  date-format: long
  lang: en
  title: Swiss Fertility
  
Output created: report.html

Our project directory now looks like this.

.
├── Makefile
├── data
│   └── raw_data.csv
├── out
│   ├── cleaned_data.rds
│   ├── fig_fitted.png
│   ├── model.rds
│   └── vals_fitted.rds
├── report.html
├── report.qmd
├── report_files
│   └── libs
└── src
    ├── cleaned_data.R
    ├── fig_fitted.R
    ├── model.R
    └── vals_fitted.R

We have created cleaned_data.rds, model.rds, vals_fitted.rds, and fig_fitted.png in the out directory, and report.html in the main directory.

Named vs unnamed arguments

The line passes one filename (myoutput.rds) and one numeric value (100) to the script myfile.R:

Rscript myfile.R myoutput.rds --n_iter=100

The script myfile.R contains

cmd_assign(n_iter = 100,
           .out = "myoutput.rds")

cmd_assign() matches the unnamed argument myoutput.rds with .out and matches the named argument --n_iter=100 with n_iter. Because .out starts with a dot, extract_make() treats it as the file name and writes n_iter into the recipe as --n_iter=100.

Other resources