This vignette walks you through a variety of different installation options and workflows for analysing data in bulk on a high performance computing (HPC) platform. In addition to detailed comprehensive instructions, we also have created a quick-start guide to help you get set up quickly.
1. Quickstart for TRE support staff
If you have been asked to install the latest version of
nutrientprofiler on a TRE/VLE/other airgapped system, you
will find the package on the GitHub
releases page.
For every release, there is a tar.gz and a
zip file containing the code. Please download the tar file,
either directly from the web interface, e.g. by clicking https://github.com/Leeds-CDRC/nutrientprofiler/archive/refs/tags/v2.0.1.tar.gz
(warning: direct download link), or by using this little R snippet or
any command line download tool of choice:
# download the v2.0.1 release as a .tar.gz archive
# to your current directory
download.file("https://github.com/Leeds-CDRC/nutrientprofiler/archive/refs/tags/v2.0.1.tar.gz",
dest = "./nutrientprofiler-v2.0.1.tar.gz")While this zipped file can be expanded/extracted for review, when it
is approved please move the original tar.gz file to a
user-accessible area of the TRE. This is crucial, as the unzipped
folder cannot be isntalled as an R package.
Additionally, the use of this package generally also requires the
following supporting packages:
dplyr, tidyr, devtools, readxl, writexl, which are all
available via CRANN. We test on R 4.3.0 and the most recent R release
when we update the code (currently tested again R 4.6.1).
Continue reading for more detailed guidance on installing and testing different versions of the code.
2. Quickstart for TRE users
Request that the package be installed as per the above instructions.
Check that you have access to the following supporting packages for your
version of R: dplyr, tidyr, devtools, readxl, writexl,
which are all available via CRANN.
Once the tar.gz file has been moved to your user area,
you can install it using R (ensure you update the file path
appropriately):
# install the package directly from source
install.packages("./nutrientprofiler-v2.0.1.tar.gz", repos = NULL, type="source")Once installed, you can import the package:
Continue reading for more details on installation and workflows on HPC systems.
Loading package onto an “airgapped” HPC system
If working on a secure or “airgapped” system without internet access,
using the R remotes package to install
nutrientprofiler will not work.
Instead, if there is a file/data transfer process available, the zipped package can be transferred to the secure computational platform and installed locally.
This process can be broken down into 3 steps:
- Downloading the package/zipped archive to your local system
- Transferring the files to the secure system (after any required security checks)
- Installing without internet access on the secure system.
The instructions here assume access to some form of archived CRAN mirror so that other commonly used packages are available. Please check through the packages listed in the code snippets and ensure you have access to these on your system.
1. Downloading the package to your local machine
You can download a specific tagged release (see the GitHub releases page) or a development pre-release version of the code from a specific branch of the repository. Find the “Assets” section and download either the tar file or the zip file for any release. See below details for how to do this via the command line.
Tagged releases allow for better reproducibility; however, if a pre-release version needs to be used, reproducibility can still be ensured by recording the download link used and the access date.
1.1 Downloading a tagged release
The recommended method is to download the most recent tagged
release as a .tar.gz archive.
From your local desktop you can run the following R snippet to
download the v2.0.1 release of the package:
# download the v2.0.1 release as a .tar.gz archive
# to your current directory
download.file("https://github.com/Leeds-CDRC/nutrientprofiler/archive/refs/tags/v2.0.1.tar.gz",
dest = "./nutrientprofiler-v2.0.1.tar.gz")Alternatively, this can be downloaded directly from the GitHub releases page.
A different release can be specified by changing the release tag. Make sure to also update the destination filename to prevent confusion:
# replace <tag> with the required version number
download.file("https://github.com/Leeds-CDRC/nutrientprofiler/archive/refs/tags/<tag>.tar.gz",
dest = "./nutrientprofiler-<tag>.tar.gz")If required, zip file archives are also available on the GitHub
releases page or can be downloaded using the following R script
below. Please note that these .zip archives require a
slightly different installation to .tar.gz; please read
through the isntallation steps first.
# download the <tag> release as a zip archive
# to your current directory
download.file("https://github.com/Leeds-CDRC/nutrientprofiler/archive/refs/tags/<tag>.zip",
dest = "./nutrientprofiler-<tag>.zip")1.2 Downloading the current development version or a specific branch
If you want to install a pre-release version of a specific version on
a branch, replace tags/version-number in the url with
heads/branch-name (and rename the destination file
something sensible).
Please record the date of download and most recent commit identifier as the branch may be updated or changed following your download and installation.
For example, the following downloads the current package on the “VarEdits” branch of the repository:
# replace "tags/version-number" with "heads/branch-name":
# in this example, we replaced "tags/v1.0.0" with "heads/VarEdits"
download.file("https://github.com/Leeds-CDRC/nutrientprofiler/archive/refs/heads/VarEdits.tar.gz",
dest = "./nutrientprofiler-VarEdits.tar.gz")Again, a zip file version can be downloaded by replacing
tar.gz in the snippet above with zip. A zip
file can also be downloaded from the project GitHub page by
navigating to the required branch and then using the green “Code” button
to open the “Clone” option menu, and selecting the “Download zip”
option.
For added reproducibility, you can specify the git commit tag to ensure you are using a specific version:
# use the unique commit id after "archive/" to specify a specific commit on a branch
# use the first 8 characters of the commit id to tag your downloaded archive
download.file("https://github.com/Leeds-CDRC/nutrientprofiler/archive/8502784c9e1402505530d87db001fc23fb0fb6df.tar.gz",
dest = "./nutrientprofiler-8502784.tar.gz")Again, make sure to modify the installation command to match the archive name you’ve supplied.
2. Transferring your code to the secure platform
This step will vary depending on the data transfer policies and process enforced by your institution. While the package archive can be unzipped for testing, it should be saved on the secure system in it’s original compressed format.
3. Installation
On the secure computing platform, once the archive has been transferred, you can then install the package. The installation method differs depending on the filetype.
3.1 Install .tar.gz archives
Using an appropriate relative path for the archive, you can install it from source:
# install the package directly from source
install.packages("./nutrientprofiler-v1.0.0.tar.gz", repos = NULL, type="source")Change the suggested filename to suit your specific installation:
# install the package directly from source
install.packages("./nutrientprofiler-VarEdits.tar.gz", repos = NULL, type="source")3.2 Install .zip archives
In order to install the package from a .zip file, you
need to use the devtools package:
# install and load devtools
install.packages("devtools")
library(devtools)
devtools::install_local("./nutrientprofiler-v1.0.0.zip")Again, change the suggested filename to suit your specific installation.
Example workflow for processing data in bulk
Once nutrientprofiler has been installed on your system,
it can be imported and used interactively or run within scripts on bulk
data.
Ensure data is formatted to match the example csv files with same column names.
Assuming the data that you want to analyse is stored in
"data/example_data.csv" and you want to save the results
to"results/example_data_results.csv" with all the original
columns plus the results columns, you can use a script like this:
# load required libraries
library(tidyr)
library(dplyr)
library(nutrientprofiler)
# read in the data
npm_testcases <- read.csv("data/example_data.csv")
# Analyse all entries, including specific gravity conversion, NPM scoring and assessment
npm_testcases_results <- npm_testcases %>%
rowwise() %>%
mutate( sg = SGConverter(pick(everything()))) %>%
mutate(test = NPMScore(pick(everything()), sg_adjusted_label="sg")) %>%
unnest(test) %>%
rowwise() %>%
mutate(assess = NPMAssess(pick(everything()))) %>%
unnest(assess) %>%
select(everything(), energy_score, sugar_score, salt_score, fvn_score,
protein_score, satfat_score, fibre_score, NPM_score, NPM_assessment)
# Save results to a csv file
write.csv(npm_testcases_results, "results/example_data_results.csv", row.names = FALSE)This only requires slight modification if the desired input file is
instead in "data/example_data.xlsx":
# load required libraries
install.packages("readxl") # If this is not already installed in the workspace
library(tidyr)
library(dplyr)
library(readxl)
library(nutrientprofiler)
# read in the data on the first sheet of the spreadsheet (sheet = 1)
npm_testcases <- read_excel("data/example_data.xlsx", sheet = 1)
# The rest of the wokflow is the same...Troubleshooting
The most common errors to arise are likely to be related to incorrect column names in your data, or incorrect datatypes for the values in these columns.
You can check the names of the columns using
names(npm_testcases).
You can replace column names using a script like this:
# load required libraries
library(tidyr)
library(dplyr)
library(nutrientprofiler)
# read in the data
npm_testcases <- read.csv("data/example_data.csv")
# Function to rename variables
replace_var_names <- function(data_frame){
if ("fat_measurement_g" %in% names(data_frame)){
data_frame <- rename(data_frame, satfat_measurement_g = fat_measurement_g)
}
if ("fruit_nut_measurement_percent" %in% names(data_frame)){
data_frame <- rename(data_frame, fvn_measurement_percent = fruit_nut_measurement_percent)
}
}
# Call the function on the example data
replaced_names <- replace_var_names(npm_testcases)The example shown above uses a function so that this can be reused with multiple data sets that have the same column naming issues.