New features

• The sampling of sex for line list and contacts data can now be weighted instead of assuming a uniform probability via prob_male in config (#255).

• The reporting_delay argument in sim_linelist() and sim_outbreak() now supports <epiparameter> objects, matching the functionality of the other delay distribution arguments (#262).

• A new line list events plot has been added to the vis-linelist.Rmd vignette (#278).

Minor changes

• Added dependabot (#258).

• GitHub actions versions are updated (multiple PRs).

• .lintr is updated to include more linters (#279) and the # nolint flag syntax is fixed (#263).

The changes in v0.7.1 are minor edits to the JOSS paper for publication.

Acknowledgements

Thanks to the whole Epiverse community for their input on the development of the simulist R package.

Thanks to the team at JOSS for rewarding open source software, and the editors and reviewers for their helpful comments that improved the publication (open review found here).

New features

• A new censor_linelist() function is added to the package. This is a new post-processing function for line list data to censor <Date> to daily, weekly and other user-defined time periods. The censor_linelist() also has a weekend-effect option to replicate the low reporting over weekends. This new feature also adds the {grates} package as a dependency (#240).

• messy_linelist() can now accept a vector with multiple missing_values to allow different values to represent missing data (#241).

• A demonstration of how to replicate under-reporting of line list and contact tracing data is added to the wrangling-linelist.Rmd vignette. The demo contains examples for multiple types of under-reporting data found in real-world outbreaks (#244).

Breaking changes

• The $was_case column in contact tracing data simulated by sim_contacts() now contains logical boolean values instead of "Y" and "N" character strings (#243).

• The minimum R version required by the package is incremented to v4.2.0 due to a recursive dependency on {fastymd} (#235).

Minor changes

• Documentation on censoring dates is added to wrangling-linelist.Rmd (#240).

• Documentation on interoperability between {simulist} and {incidence2} for double precision <Date>s and warning for not specifying interval in incidence2::incidence() is added to vis-linelist.Rmd (#232).

• The lifecycle of the package has been updated from experimental to stable (#229).

• @return function documentation for exported functions is improved (#235).

Bug fixes

• truncate_linelist() sets $outcome to NA if they are more recent than truncation time. This matches the way $date_outcome was handled in truncate_linelist() in v0.5.0 (#228).

• Alt-text in vignettes is fixed to render correctly on pkgdown website (#230).

Deprecated and defunct

• None

Acknowledgements

@joshwlambert @avallecam @TimTaylor @sbfnk @seabbs

The R package

The R package in question is {ringbp}. The package has two pieces of functionality: 1) to simulate an infectious disease outbreak using a branching process model with non-pharmaceutical interventions; and 2) to calculate the proportion of simulated outbreaks that are contained (i.e. do not cause a large sustained human-to-human epidemic). The utility of the package’s general model framework has been shown by serving as a template for other epidemiological research such as post-exposure prophylaxis, network effects on control (Firth et al. 2020) and the impact of self-reporting and isolation adherence (Davis et al. 2020).

The problem

It is understandable that because {ringbp} was written in haste to produce insights to inform pandemic response it did not adhere to all software best practices. Usability, documentation, testing, code style, and (computational) performance could be improved. Certain aspects of model code, like parameterisations, were hard-coded, not providing users the full flexibility that the model could allow.

Epiverse contribution

In the recent months Epiverse has collaborated with {ringbp} developers Seb Funk (also a member of Epiverse) and Carl Pearson (external collaborator), based at the London School of Hygiene and Tropical Medicine and University of North Carolina, respectively, to try and improve the R package, both internally and from the user-experience. The following sections will give brief summaries of some of the collaborative developments.

scenario_sim(
  n.sim = 5,
  num.initial.cases = 5,
  cap_max_days = 365,
  cap_cases = 2000,
  r0isolated = 0,
  r0community = 2.5,
  disp.iso = 1,
  disp.com = 0.16,
  k = 0.7,
  delay_shape = 2.5,
  delay_scale = 5,
  prop.asym = 0,
  prop.ascertain = 0
)

scenario_sim(
 n = 5,
 initial_cases = 5,
 offspring = offspring_opts(
   community = \(n) rnbinom(n = n, mu = 2.5, size = 0.16),
   isolated = \(n) rnbinom(n = n, mu = 0, size = 1),
   asymptomatic = \(n) rnbinom(n = n, mu = 2.5, size = 0.16)
  ),
 delays = delay_opts(
   incubation_period = \(n) rweibull(n = n, shape = 2.32, scale = 6.49),
   onset_to_isolation = \(n) rweibull(n = n, shape = 2.5, scale = 5)
  ),
 event_probs = event_prob_opts(
   asymptomatic = 0,
   presymptomatic_transmission = 0.3,
   symptomatic_ascertained = 0
  ),
 interventions = intervention_opts(quarantine = TRUE),
 sim = sim_opts(
   cap_max_days = 365,
   cap_cases = 2000
  )
)

The new API gives the user more control over the model’s parameterisation. The incubation period is now specified by the user instead of being set to an estimate for COVID-19. The way offspring and delay distribution functions are specified also means that any distributional or non-parametric form can be supplied, relaxing the assumption that the onset-to-isolation has to be a Weibull distribution.

Users can now specify the proportion of presymptomatic transmission rather than having to understand the skew normal parameterisation used by the simulation model, making it easier to get started with the package for new users.

Lastly on user-facing changes, the naming and style of function arguments has been standardised for consistent use of snakecase style and abbreviations.

@param sim a `list` with class `<ringbp_sim_opts>`: the simulation control
  options for the \pkg{ringbp} model, returned by [sim_opts()]

Conclusion

The {ringbp} R package implements a simple but informative model for infectious disease transmission and interventions. When originally written it included many well-developed aspects, but the time constraints of real-time outbreak response meant several improvements were possible.

Epiverse-TRACE has the opportunity to not only develop new tooling for pandemic preparedness and response, but to contribute to the ecosystem of open-source software in infectious disease epidemiology. We hope that by covering the collaborative developments of {ringbp}, it can illustrate the benefits of bringing software up to date with best practices, and make tools available, accessible and robust when a new epidemic or pandemic occurs, in turn hopefully removing the need for redeveloping similar software in the future.

These efforts are not limited to {ringbp}. Similar contributions have been made to other packages, including EpiNow2, and open source communities such as the epinowcast community, where documentation, testing, and code enhancements have helped strengthen the forecasting and nowcasting software ecosystem in the last couple of years.

Enhancing the accessibility of software for users and developers by improving its documentation and user interface will hopefully provide a gateway for more external contributors to engage with the project. In the public health landscape of temporal surges in capacity and priorities, better enabling community contributions to open-source software should aid software sustainability.

All of the changes discussed in this blog post can be found in the {ringbp} news. For details of developments see the pull request history of {ringbp} on GitHub.

Acknowledgements

Thanks to Seb Funk, Carl Pearson and James Azam for helpful feedback when drafting this post and for their collaboration on the {ringbp} project.

New features

• The probability_emergence() function has been added to calculate the probability that a pathogen introduced to humans can evolve and emerge to cause a sustained human-to-human outbreak, implementing and extending the framework of Antia et al. (2003) (#124, #133).

• A new vignette, emergence.Rmd, has been added that covers the functionality of probability_emergence() and reproduces the two figures from Antia et al. (2003) as well as a figure using the multiple introductions extension (#124, #133).

• Alt-text has been added to all plots across all vignettes (#129).

Breaking changes

• The percent_transmission argument in proportion_transmission() has been renamed to prop_transmission (#130).

Minor changes

• An .aspell/ folder is added to the package including defaults.R and superspreading.rds to supply a wordlist to the CRAN spell checking to avoid quoting names in the DESCRIPTION (#127).

• Package and function documentation has been updated. Vignette changes include minor reworking of text, updating any information or links that were outdated; function documentation is styled more consistently and follows the Tidyverse style guide (#131, #134).

• Internal code style has been updated to adhere to current best practice (#125).

• The package lifecycle badge has been updated from experimental to stable. CRAN status, CRAN downloads, repo status and Zenodo DOI badges have been added to the README (#119, #132).

• The {pkgdown} development: mode has been set to auto now the package is hosted on CRAN (#118).

Bug fixes

• None

Deprecated and defunct

• None

Acknowledgements

@chartgerink @joshwlambert

safeframe 1.0.0

• First release of safeframe.
• Factors out the core functionality of linelist to make it more viable for alternative domains
• General usage remains the same:

# Create a safeframe object
x <- make_safeframe(cars, mph = "speed", distance = "dist")

# Validate the tagged data are of a specific type
validate_safeframe(x,
  mph = 'numeric',        # speed should be numeric
  # type() is a helper function of related classes
  distance = type('numeric')    # dist should be numeric, integer
)

Breaking changes

• The linelist package is now dependent on R 4.1.0, in line with Epiverse-TRACE policy regarding minimum R version dependency (#159, @Bisaloo).

• Deprecated functions and arguments have been completely removed. The following operations are no longer possible:

  • List of tags now needs to be spliced in make_linelist():

    my_tags <- list(
      id = "case_ID",
      date_onset = "date_of_prodrome",
      age = "age",
      gender = "gender"
    )
    
    # No longer possible
    make_linelist(obj, my_tags)
    
    # Instead slice list in dynamic dots
    make_linelist(obj, !!!my_tags)

  • select_tags():

    # No longer possible
    x %>%
      select_tags("age")
    
    # Instead use
    x %>%
      select(has_tag("age")) %>%
      tags_df()

  • select.linelist(). This change should be invisible to users since the parent method select.data.frame() will be used with the same effect.

  • lost_tags_action() as part of a pipeline is no longer possible:

    # No longer possible
    make_linelist(cars, date_onset = "dist", date_outcome = "speed") |>
      lost_tags_action("none") |>
      dplyr::select(-dist)
    
    # Instead do
    lost_tags_action("none")
    make_linelist(cars, date_onset = "dist", date_outcome = "speed") |>
      dplyr::select(-dist)

Acknowledgements

@Bisaloo @chartgerink @Karim-Mane

Bug fixes

• Fixed a bug in the index for empty rows returned by print_report() after remove_constants(). This function was incorrectly returning the number of rows in the original data frame, instead of the index of empty rows (#173, @Bisaloo).

• Fixed a bug in the date guesser used by the standardize_dates() function. The followings updates were made (#175, @Karim-Mane):

  • allow for quarter and partial dates, and
  • account for odd cases before parsing using the {lubridate} package.

Enhancements

• Rendering the report no longer requires the use of the {withr} package. Files generated during this process will be stored in the R temporary directory (#165, @Bisaloo).

• Simplification and improvement of code performance across several functions (#154, #156, #162, @Bisaloo).

• Make R version dependency explicit (#156, @Bisaloo).

• Replace base R pipe operator with the {tidyverse} package pipe operator (#155, @Karim-Mane).

• Stop automatic conversion of numeric values into Date due to unexpected results from date_guess() function (#151, @Karim-Mane).

• Allow for a vector of character for na_strings argument in replace_missing_values() (#180, Karim-Mane).

• Improved messaging to differentiate between duplicates found in the subject IDs column and duplicates across multiple columns (#180, Karim-Mane).

• Allow for iterative constant data removal (#180, Karim-Mane).

• The remove_constants() now works correctly when the cutoff is different from 1. More importantly, the function no longer depends on the {janitor} package (#177, Karim-Mane).

• The scan_data() now makes use of date_guess() to parse date from characters and lubridate::as_date() to parse dates from numeric. This function now operates only on character columns (#181, Karim-Mane).

• Replace snakecase::to_snake_case() with janitor::clean_names() and make sure variable names specified in keep and rename are preserved after applying janitor::clean_names() (#180, Karim-Mane).

• Use the {cli} package for formatting messages in the package (#196, Karim-Mane).

• Translate messages in the package into French using the {potools} package (#196, Karim-Mane).

• An additional column (row_id) with the row indices where the incorrect date sequences are found is now added to the report made from the check_date_sequence() function (#196, Karim-Mane).

• The date_guess() and related functions now returns a list of two elements: the converted values and a boolean that informs about the presence of numeric values that can also be of type Date (#181, Karim-Mane).

• Update the package vignette to account for the improvements made in the different pull requests (#194, Karim-Mane).

Deprecated and defunct

• Function documentations have been updated. The clean_data() no longer takes the params argument. Instead, it takes many other arguments that are described in the function documentation and the package vignettes (#184, Karim-Mane).

Acknowledgements

@avallecam @Bisaloo @chartgerink @Degoot-AM @joshwlambert @Karim-Mane @LloydChapman

How we setup integration testing in Epiverse

The Epiverse-TRACE collection of packages has a meta-package, {epiverse}, analogous to the tidyverse meta-package (loaded with library(tidyverse)). By default, {epiverse} has dependencies on all released and stable Epiverse-TRACE packages, therefore it is a good home for integration testing. This avoids burdening individual Epiverse packages with taking on potentially extra dependencies purely to test interoperability.

Just as with unit testing within the individual Epiverse packages, we use the {testthat} framework for integration testing (although integration testing can be achieved using other testing frameworks).

Case study of interoperable functionality using {simulist} and {cleanepi}

The aim of {simulist} is to simulate outbreak data, such as line lists or contact tracing data. By default it generates complete and accurate data, but can also augment this data to emulate empirical data via post-processing functionality. One such post-processing function is simulist::messy_linelist(), which introduces a range of irregularities, missingness, and type coercions to simulated line list data. Complementary to this, the {cleanepi} package has a set of cleaning functions that standardised tabular epidemiological data, recording the set of cleaning operations run by compiling a report and appending it to the cleaned data.

set.seed(1)
ll <- simulist::sim_linelist()

test_that("convert_to_numeric corrects prop_int_as_word", {
  # create messy data with 50% of integers converted to words
  messy_ll <- simulist::messy_linelist(
    linelist = ll,
    prop_missing = 0,
    prop_spelling_mistakes = 0,
    inconsistent_sex = FALSE,
    numeric_as_char = FALSE,
    date_as_char = FALSE,
    prop_int_as_word = 0.5,
    prop_duplicate_row = 0
  )
  
  # convert columns with numbers as words into numbers as numeric
  clean_ll <- cleanepi::convert_to_numeric(
    data = messy_ll, 
    target_columns = c("id", "age")
  )
  
  # the below is not TRUE because
  # 1. `clean_ll` has an attribute used to store the report from the performed
  # cleaning operation
  # 2. the converted "id" and "age" columns are numeric not integer
  expect_false(identical(ll, clean_ll))

  # check whether report is created as expected
  report <- attr(clean_ll, "report")
  expect_identical(names(report), "converted_into_numeric")
  expect_identical(report$converted_into_numeric, "id, age")

  # convert the 2 converted numeric columns into integer
  clean_ll[, c("id", "age")] <- apply(
    clean_ll[, c("id", "age")], 
    MARGIN = 2, 
    FUN = as.integer
  )

  # remove report to check identical line list <data.frame>
  attr(clean_ll, "report") <- NULL
  
  expect_identical(ll, clean_ll)
})

Conclusion

When developing multiple software tools that are explicitly designed to work together it is critical that they are routinely tested to ensure interoperability is maximised and maintained. These tests can be implementations of a data standard, or in the case of Epiverse-TRACE a more informal set of design principles. We have showcased integration testing with the compatibility of the {simulist} and {cleanepi} R packages, but there are other integration tests available in the {epiverse} meta-package. We hope that by regularly running these expectations of functioning pipelines, includes those as simple as two steps, like the case study show in this post, that maintainers and contributors will be aware of any interoperability breakages.

If you’ve worked on a suite of tools, R packages or otherwise, and have found useful methods or frameworks for integration tests please share in the comments.

Acknowledgements

Thanks to Karim Mané, Hugo Gruson and Chris Hartgerink for helpful feedback when drafting this post.

New features

Breaking changes

• create_config() has been updated to accepted functions instead of a distribution name and a vector of parameters. This now matches the design of arguments that accept a function in sim_*() functions (#202).

• The structure of the age-structured population <data.frame> input into sim_*() functions has been standardised with the age-stratified risk <data.frame>s by using an $age_limit column instead of an $age_range character column (#200).

• The line list <data.frame> output by sim_linelist() and sim_outbreak() now contains a $date_reporting column (#179).

• Outcome date ($date_outcome) is now conditioned to be after hospitalisation date ($date_admission) using the new internal .sample_outcome_time() function. This is a breaking change as previously hospitalisation times could be before outcome times, sim_linelist() can now through an error if an outcome time cannot be sampled to be after the hospitalisation time (#178).

• The date of first contact is now sampled as the number of days before infection time (equal to symptom onset in the model) rather than days before date of last contact, as this could lead to the infection time before the first contact (#206).

• The minimum required R version for simulist is increased to v4.1.0 from v3.6.0 due to package dependencies (#180).

• The minimum required version of {incidence2} (suggested dependency) is now v2.3.0 (#214).

Minor changes

• Explicitly state function output breaking changes policy in design-principles.Rmd vignette (#208).

Bug fixes

• Date of symptom onset can no longer occur before date of first contact (#206).

• Outcome date can no longer occur before hospitalisation date (#178).

Deprecated and defunct

• None

Acknowledgements

Thanks to those that contributed towards this release, either through opening issues, reviewing pull requests or discussing package developments.

@avallecam @Bisaloo @CarmenTamayo @Degoot-AM @jamesmbaazam @joshwlambert @Karim-Mane @sbfnk

New features

• The proportion_transmission() function has been expanded to incorporate a new method. The new method calculates the proportion of transmission from X% the most infectious individuals, corresponding to the Lloyd-Smith et al. (2005) calculation. The proportion_transmission() has a new method argument to toggle between the two calculations (@dcadam, #99).
• A new vignette explaining the methods in the proportion_transmission() function (#101).
• {bpmodels} is removed as a package dependency and a branching process simulation function, modified from bpmodels::chain_sim(), is added to the {superspreading} package (#103). @sbfnk is added as copyright holder and @jamesmbaazam as a contributor.
• probability_contain() is enhanced to enable to probability of containment within a certain amount of time since the outbreak started. This adds outbreak_time and generation_time arguments to probability_contain(). This addition is backwards compatible as by default the time is unlimited, outbreak_time = Inf, and no generation time is required.

Breaking changes

• The stochastic argument in probability_contain() has been renamed simulate to be consistent with other functions (#103).

Minor changes

• The proportions output of proportion_*() functions are now formatted to significant figures rather than rounding to prevent small values being rounded to zero (#102).
• Improve input checking, error messages and edge case handling for functions (#102).
• Vignettes now use rmarkdown::html_vignette instead of bookdown::html_vignette2 and as_is: true has been removed due to changes to {pkgdown} in v2.1.0. {bookdown} has been removed as a suggested package and code folding is removed from vignettes. KaTeX headers have been added to _pkgdown.yml for correct math rendering (#104 & #109).
• The get_epidist_params() internal function has been renamed get_epiparameter_params() since {epiparameter} renamed the <epidist> class to <epiparameter> (#100).
• Internal constants have been added to the package and used by functions (#111).
• Checking if the user specifies individual parameters (R and k) or provides an <epiparameter> object is now in .check_input_params() (#111).

Bug fixes

• None to {superspreading} functions.
• Update {epiparameter} use in vignette and tests (#106).

Deprecated and defunct

• None

Acknowledgements

@avallecam @Bisaloo @chartgerink @dcadam @joshwlambert

Overview of previous blog post on R package licensing

We previously published a post on the Epiverse-TRACE blog discussing the importance of licensing code and data for open source. It covered the differences between open source licenses for code and those that are required for other forms of information, in our case, data. R packages most commonly distribute code, and therefore require an appropriate open source license. Less frequently, R packages are used to bundle and share data, so-called “data packages”, and require an applicable license for reuse and redistribution. We recommend reading the original blog post for a more in-depth explanation of each of these points.

However, since publishing the original post, the last section, Licensing code and data in one R package, has become outdated given our experience submitting the epiparameter package to CRAN with a dual-license. This blog post provides an updated Licensing code and data in one R package section with learnings from epiparameter package development and CRAN submission.

Licensing code and data in one R package

If you are developing an R package that has both code and data as primary objects of (roughly) equal importance, a software license inadequately covers the data, and a data license inadequately covers the code. Dual licensing can help resolve this issue. This means there is one license for code (for example, MIT license) and another license for the included data (for example, Public Domain Dedication).

After conducting an online search, dual licensing for R packages seems rare. An interesting example of dual licensing is the igraphdata package, which contains several licenses: One for each dataset included in the package. Similar to igraphdata, we dual licensed the epiparameter package, which until versus v0.4.0 contained both code and data. We licensed the code using the DESCRIPTION file and used the LICENSE file to license the data under CC0. Concretely, we included this additional text in LICENSE to clarify the dual license and that we recommend citing the original source regardless:

All data included in the epiparameter R package is licensed under CC0 (https://creativecommons.org/publicdomain/zero/1.0/legalcode.txt). This includes the parameter database (extdata/parameters.json) and data in the data/ folder. Please cite the individual parameter entries in the database when used.

However, upon submission of epiparameter to CRAN, the dual licensing approach was rejected. The CRAN package reviewer stated:

A package can only be licensed as a whole. It can have a single license or a set of alternative licenes. If the data have to be licensed differently then the code, you have to provide the data in a separate data package with the other license.

Therefore, we decided to separate the data, originally stored in epiparameter, into a package called epiparameterDB. This package is solely licensed under CC0, and epiparameter can then also be solely licensed under MIT. Both packages are now hosted on CRAN using a single license dedicated to code or data.

When including data in your R package from other sources it is important to check that the license of your package and the data is compatible¹, or that the individual data license is clearly stated, as in igraphdata. For epiparameterDB, we consider model estimates as facts (that is, not copyrightable).

This blogpost provides an addendum to our original post Dual licensing R packages with code and data, providing our experience and outcomes from submitting epiparameter – and the resulting epiparmeterDB package – to CRAN in a conformant manner. In addition to the requirements and benefits of open source licensing of open software and data in the first post, we hope this follow-up post provides practical information that will be of use to other R package developers, or software developers and open data curators more broadly.

New features

Breaking changes

• The function arguments infect_period and prob_infect have been renamed infectious_period and prob_infection (#143).
• sim_*() functions now use NULL instead of NA to turn off processes (e.g. onset_to_death = NULL for no deaths) (#148).

Bug fixes

• A new function, .sample_infect_period() is added that errors if the infectious period function generates a negative number (#142).
• sim_linelist() no longer errors when hosp_death_risk is NULL and onset_to_death is parameterised as a delay distribution (#144).
• .add_ct() generates the correct number of values and does not duplicate Ct values due to vector recycling (#158).
• Update {epiparameter} usage (#159).

Deprecated and defunct

• None

Acknowledgements

@avallecam @Bisaloo @joshwlambert @Karim-Mane @sbfnk

New features

• The library of epidemiological parameters (parameters.json) has been removed from the {epiparameter} package and moved to the {epiparameterDB} R package which is taken on as a dependency. The {epiparameter} package is licensed solely as MIT with dual licensing with CC0 removed (#415).
• The data dictionary (data_dictionary.json) and JSON validation workflow (validate-json.yaml) have been removed from the package (#415).
• The library of epidemiological parameters is cached (using {cachem}) on loading the package to reduce the runtime of epiparameter_db() (#417).
• The print() and format() methods for the <epiparameter> class have been improved to show the units of parameter estimates, as well as summary statistics, if available, for unparameterised <epiparameter> objects (#419).
• on.exit() is no longer called in plotting functions to allow user-specified graphical parameters to not be overwritten (#420).
• A plot() method for the <multi_epiparameter> class is added. This enables a list of <epiparameter> objects with continuous distributions to be compared on a single plot (#425).
• A lines() method for the <epiparameter> class is added (#425).

Breaking changes

• None

Bug fixes

• A bug in is_continuous() causing it to incorrectly returning FALSE for <epiparameter> objects with normal distributions is fixed (#422).

Deprecated and defunct

• None

Acknowledgements

@joshwlambert

Reflections on COVID-19 Contributions

During COVID-19, academic participants like myself in the UK contributed analysis to the SPI-M-O advisory group, which focused on epidemiology and modelling. This was a subgroup of the Scientific Advisory Group for Emergencies.

These contributions typically fell into two main categories:

• Reports in response to specific questions from the Secretariat (e.g. exploring implications of policy options).
• Reports or preliminary results detailing broader epidemiological insights about COVID-19 my colleagues and I thought were noteworthy (e.g. unusual patterns with novel variants).

This post focuses on the analysis reports that I made a major contribution to as a member of SPI-M-O in the first 18 months after COVID was identified as a threat, i.e. between Jan 2020 and July 2021 (a full list has previously been published). Narrative reports or analysis that did not involve substantial analytics or modelling (e.g. just direct plots of data) are not included here. I also include some major early piece of analysis that did not form SPI-M-O reports, but were published and informed subsequent analysis and modelling.

If another pandemic were to hit, how easily and quickly could we do these analyses again? To document where we currently are, it is important to understand where potential gaps and bottlenecks are. For each report, I therefore review three main criteria:

1. Code availability: is the original analysis code public? (With link if relevant, or context if not.)
2. Package availability: is the analysis code or underlying method currently packaged up for easy reuse?
3. Task readiness time: roughly how long would it take to get the code or package into a rough state where it could re-run an equivalent analysis using the characteristics of a future transmissible H5N1 influenza? And how long to do so while also following robust best practice Epiverse-TRACE development principles, so others can easily build on the analysis? Would it take minutes (i.e. possible to run immediately), hours, days or weeks to get the basic functionality working?

Criteria (1) and (2) are either marked as available (✅), not available (❌), or partially available (⏳). Time taken is also divided into ‘rough’ (rapid, imperfect code to deliver a task) and ‘robust’ (i.e. best practice development for future re-use) to reflect wider discussions about what constitutes ‘good enough’ work in a pandemic. I also suggest some areas for potential further development, or links to ongoing development that will enable easier completion of tasks in future. The focus of the post tends to be packages that sit on the CMMID, Epiverse-TRACE, Epiforecasts or my GitHub repositories, because these were most directly related to the tasks being discussed, but some wider packages are also signposted.

This is not an exclusive list of work performed by myself and colleagues in the Centre for Mathematical Modelling of Infectious Diseases at LSHTM; there is a larger CMMID repository of real-time work, as well as a large volume of published papers in academic journals and reports on the gov.uk website. However, I hope this initial post can provide a useful summary of tasks that were performed, and framework for evaluating wider efforts required for a pandemic.

The overall effectiveness of the UK response, and which areas could be strengthened, are topics currently being examined by the COVID Inquiry, and will not be covered here. If readers are interested, there are some broader reflections from members of the UK modelling community, and recommendations for improvements, in Sherratt et al, 2024, Wellcome Open Research.

Concluding thoughts

Code was generally made available alongside public reports by LSHTM during COVID-19, except for very simple calculations. Several key functions are now available in R packages developed since the emergence of COVID-19, turning tasks that would have taken days into tasks that require only hours, but there are still some remaining bottlenecks to ensure that the methods would be applicable easily to H5N1, meaning some customised tasks for pandemic influenza would take hours when - with some further refinement - they could take minutes.

New features

• The first release contains the parameter database (parameters.json) from the {epiparameter} package v0.3.0.
• Functionality to read the JSON file (parameters.json), render the database as a vignette (database.Rmd), and validate the JSON database with a data dictionary (data_dictionary.json) using a GitHub actions workflow (validate-json.yaml) have been copied from the original {epiparameter} implementation.

Bug fixes

• None

Deprecated and defunct

• None

Acknowledgements

@dependabot[bot] @joshwlambert

What is a full package review?

You may not be surprised to hear that a full package review covers every part of a package ever written. This means the entire current codebase is in scope, from day one of development. It does not necessitate reviewing the entire commit history per se, focussing on the package as it stands at that time. As a maintainer that can be scary, because it can include code you did not write yourself, and may not know how to justify (at least that is what happens for me).

Our regular code reviews take place on GitHub, using Pull Requests. For full package reviews, we also use Pull Requests in Epiverse-TRACE. This means that all the features from GitHub are available to make suggestions, approve the code, or leave precise comments at a specific line in the code.

What is different, is that all the files are considered new, and part of the review. This includes the files already present in our package template, but also all test files, documentation, and more. A full package review is usually triggered by an upcoming release, either on GitHub or CRAN.

How do I set one up?

Git and GitHub do not support full package reviews by default. This is because pull requests are usually merged into an existing branch, which already contains existing code. For a full package review, what we want is to merge onto an empty branch.

In order to create a branch that has no code at all, run the following shell command from within your git directory:

git branch empty `git rev-list --all | tail -1`
git push origin empty

git branch review
git push origin review

These commands are also noted in the blueprints, but I simplified them so that you do not need to copy paste the hash of the first commit. Also, you can use this command for any Git repository and open up the pull request through GitHub (or Gitlab, Forgejo). This means that doing a full package review can be done anywhere, anytime.

The experience of a full package review

Since joining Epiverse-TRACE, I participated in many full package reviews, but did not until quite recently initiate my own. For the new package safeframe, I recently requested a full package review. This highlighted two quirks I did not realize before.

First, it feels like doing something that can easily go wrong. I have been using Git for over a decade¹, but this was completely new to me. Given that git operations can sometimes cause issues, I was hesitant to just run the shell commands. Maybe having a demo video showing it works, will help contributors gain confidence in doing this.

Second, reviewer suggestions are great, but getting them on a full package review involves manual work to include them. I try to leave actionable suggestions directly in the code when I review, so they are easy to accept and integrate into the codebase. With full package reviews, receiving suggestions became slightly annoying. Not because of the reviewer, but because the suggestions have to be manually recreated in a separate branch that will actually be merged. It would be great if we had a way to keep the ease of integrating reviewer suggestions in full package reviews.

Overall, full package reviews are a helpful way to take a step back and take in the big picture. It is a great moment to get feedback from Epiverse-TRACE developers and discuss how the package fits in to the ecosystem. It also comes with some quirks, as I outlined, which I would like to find ways to address.

I would be interested to hear other developer’s experiences with full package reviews in the comments!

Breaking Changes

{vaccineff 1.0.0} refactors the package’s internal structure for better maintainability.

• estimate_vaccineff() replaces effectiveness().

  • It returns an object of class vaccineff.
  • The at parameter must always be provided for accurate results.

• plot.vaccineff_data() replaces plot_coverage().

• cohortdata has been simplified and reduced to improve examples and reduce computation time.

Quick start

To start with {vaccineff} 1.0.0, run the following example. This example estimates VE using the package’s example dataset, adjusted for confounders (age and sex) via the iterative matching strategy.

The summary() method for the vaccineff_data object provides details on cohort balance and composition before and after applying the matching strategy, as well as general cohort features.

When called on the vaccineff object (ve), the summary() method returns the estimation summary, including results from the Schoenfeld test for proportional hazards.

library(vaccineff)

# Create `vaccineff_data`
data(cohortdata)
vaccineff_data <- make_vaccineff_data(
  data_set = cohortdata,
  outcome_date_col = "death_date",
  censoring_date_col = "death_other_causes",
  vacc_date_col = "vaccine_date_2",
  vaccinated_status = "v",
  unvaccinated_status = "u",
  immunization_delay = 15,
  end_cohort = as.Date("2021-12-31"),
  match = TRUE,
  exact = "sex",
  nearest = c(age = 1)
)
summary(vaccineff_data)

Cohort start:  2021-03-26
Cohort end:  2021-12-31
The start date of the cohort was defined as the mininimum immunization date. 
65 registers were removed with outcomes before the start date.

Nearest neighbors matching iteratively performed.
Number of iterations:  4
Balance all:
              u         v         smd
age   63.917069 62.997438 -0.08593156
sex_F  0.520277  0.573474  0.10701746
sex_M  0.479723  0.426526 -0.10701746

Balance matched:
               u          v         smd
age   63.9014396 63.7280362 -0.01587567
sex_F  0.5206718  0.5206718  0.00000000
sex_M  0.4793282  0.4793282  0.00000000

Summary vaccination:
              u     v
All       10973 19905
Matched   10836 10836
Unmatched   137  9069

// tags: outcome_date_col:death_date, censoring_date_col:death_other_causes, vacc_date_col:vaccine_date_2, immunization_date_col:immunization_date, vacc_status_col:vaccine_status 

# Estimate VE
ve <- estimate_vaccineff(vaccineff_data, at = 180)
summary(ve)

Vaccine Effectiveness at 180 days computed as VE = 1 - HR:
     VE lower.95 upper.95
 0.7254   0.5437   0.8347

Schoenfeld test for Proportional Hazards assumption:
p-value = 0.1507

Acknowledgements

@Bisaloo @chartgerink @davidsantiagoquevedo @ErikaCantor @fsabrilb @GeraldineGomez @jpavlich @ManuelVelarde2212 @pratikunterwegs @zmcucunuba

New features

• The library of epidemiological parameters has been updated to include 3 new Chikungunya parameter entries. The Mpox parameters previously missing from the Guzzetta et al. entry have been added (#346 & #374).
• A c() method has been added for <epiparameter> and <multi_epiparameter> objects (#368).
• An aggregate() method has been added for <multi_epiparameter> to enable consensus distributions to be built utilising the mixture distribution class from {distributional} (#388).
• Infrastructure has been added to the package to allow translations of messages/warnings/errors printed to the console. (@Bisaloo, #367).
• convert_summary_stats_to_params() can now convert from median and dispersion for a lognormal distribution (#378).
• The data_dictionary.json has been enhanced to improve validation of the library of epidemiological parameters (parameters.json) (#379).
• An interactive network showing all of the <epiparameter> S3 methods has been added to the design_principles.Rmd vignette (#383).
• The data_from_epireview.Rmd article has been improved and updated with new changes from {epireview} (@CarmenTamayo & @cm401 & @kellymccain28, #305 & #373).
• Parameter units have been added to every entry in the {epiparameter} library (parameters.json) and to the $metadata element in <epiparameter> objects. The create_metadata() function now has a units argument to construct metadata lists (#391).
• Improved database.Rmd vignette by adding a short citation to the reference column (@jamesmbaazam, #348).
• The family() method for <epiparameter> has been improved to allow access to distribution names for transformed (e.g. mixture and truncated distributions) and untransformed (e.g. gamma and lognormal) distributions with a new argument base_dist and a new internal function .distributional_family() (#398).
• as_epiparameter() can now work with the SARS parameters from {epireview} (#407).

Breaking changes

• The <epidist> class has been renamed <epiparameter> to avoid confusion with the similar R package {epidist} (#360).
• Many functions that used epidist in their names have been renamed to use epiparameter due to the renaming of the class (#360).
• The function signatures of epiparameter() and new_epiparameter() functions (previously epidist() and new_epidist()) have been updated to collapse the prob_dist, prob_dist_params, discretise and truncation arguments into prob_distribution, which accepts the output of create_prob_distribution() (#381).
• The epi_dist argument has been renamed epi_name. This is to clarify that {epiparameter} can work with epidemiological parameters that take a variety of forms (e.g. point estimates, ranges, probability distributions, etc.) (#390).
• The <vb_epidist> class and it’s methods have been removed from the package. It was not being used and was increasing the complexity and maintenance load of the package (#359).
• create_prob_dist() has been renamed to create_prob_distribution() (#381).
• validate_epiparameter() (previously validate_epidist()) has been renamed assert_epiparameter(), and test_epiparameter() has been added, with the aim to harmonise design with {contactmatrix} and the messages and errors have been improved (#366 & #402).
• The minimum version of R required by the package is now 4.1.0 due to the use of the base R pipe (|>) in dependencies, and the R-CMD-check workflow on GitHub actions now explicitly runs on the minimum version of R stated in the DESCRIPTION (#384 & #405).

Bug fixes

• Epidemiological parameter entries in the library stored with lognormal distributions, parameterised as median and dispersion are now converted to meanlog and sdlog correctly when creating an <epiparameter> (when auto_calc_params = TRUE) (#381).

Deprecated and defunct

• epidist_db() has been deprecated. It is replaced by epiparameter_db() (#360 & #399).

Acknowledgements

@Bisaloo @CarmenTamayo @adamkucharski @avallecam @chartgerink @jamesmbaazam @joshwlambert

Reviewing transmissibilty training materials as a senior academic

To illustrate the sort of reviews that can be generated with the above approach, the below is the output produced when we imported the Vania senior academic persona alongside an early pre-Alpha version of the training materials for our transmissibility estimation tutorial:

Overall, the tutorial is well-structured and provides a comprehensive guide on how to use the EpiNow2 package to estimate transmission metrics from case data. However, there are several areas that could be improved to enhance clarity and understanding for learners like me.

1. Prerequisites: The prerequisites section could be expanded to include specific resources for learners to review the necessary concepts. For instance, links to resources on Bayesian analysis and the effective reproduction number would be helpful.

2. Jargon and Terminology: The tutorial contains several technical terms and jargon that could be confusing for learners. For example, terms like “Bayesian inference”, “posterior probability”, “likelihood”, and “prior probability” are used without clear explanations. Providing definitions or links to resources for these terms would be beneficial.

3. Code Explanation: While the tutorial does a good job of explaining what the code does, it could provide more context on why certain functions or packages are used. For instance, it would be helpful to explain why the EpiNow2 package is used and how it differs from other packages.

4. Visualizations: Although visualizations are not shown in the Rmd, it would be beneficial to explain what learners should expect to see in the visualizations and how to interpret them.

5. Exercises: The tutorial lacks exercises for learners to practice the concepts and skills they have learned. Including exercises with solutions would greatly enhance the learning experience.

Suggestions for improvements:

• Expand the prerequisites section to include specific resources for learners to review necessary concepts.
• Provide definitions or links to resources for technical terms and jargon.
• Provide more context on why certain functions or packages are used.
• Explain what learners should expect to see in the visualizations and how to interpret them.
• Include exercises with solutions for learners to practice the concepts and skills they have learned.

Reviewing forecast materials as a field epidemiologist

To further illustrate review outputs, the below is what was produced when we imported the Lucia field epidemiologist persona alongside an early pre-Alpha version of the training materials for our forecasting tutorial:

Positive Feedback:

1. The tutorial is well-structured and follows a logical progression, which makes it easier to follow.
2. The use of real-world examples and datasets (COVID-19 and Ebola) makes the tutorial more relatable and practical.
3. The inclusion of questions, objectives, prerequisites, and key points at the beginning and end of the tutorial respectively, helps to guide the learning process.
4. The tutorial provides a good balance between theory and practice, with clear explanations and practical coding examples.
5. The use of callouts to highlight important information or provide additional context is very helpful.

Constructive Feedback:

1. The tutorial assumes a certain level of familiarity with R and statistical concepts, which might be challenging for beginners. It would be helpful to provide more explanations or links to resources for beginners.
2. The code chunks are quite complex and could be broken down into smaller parts with more explanations.
3. The tutorial could benefit from more visualizations to illustrate the concepts and results.
4. The challenge section could provide more guidance or hints to help learners complete the task.

Suggestions for Improvement:

1. Provide more explanations or links to resources for beginners to learn R and statistical concepts.
2. Break down complex code chunks into smaller parts with more explanations.
3. Include more visualizations to illustrate the concepts and results.
4. Provide more guidance or hints in the challenge section to help learners complete the task.
5. Include a glossary of terms or a list of R packages used in the tutorial for quick reference.

Overcoming feedback bottlenecks

A challenge with LLMs trained for general use is finding domain-specific tasks where they can add sufficient value beyond existing human input. Tasks like providing early sense checking and tailored feedback, particularly from differing perspectives, therefore has potential to overcome common bottlenecks in developing training materials (e.g. providing initial comments and flagging obvious issues while waiting for more detailed human feedback).

As Epiverse-TRACE training materials continue to develop, we plan to explore further uses beyond simple first-pass reviews. For example, LLMs are well suited to synthesising qualitative feedback, increasing the range of insights that can be collected and summarised from learners as we move into beta testing. We also hope to identify opportunities where LLMs can help supplement the learner experience, as demonstrated by emerging tools like RTutor for descriptive plotting functionality in R, which combines generation of code in response to user queries with translation into shiny outputs.