Scratch Disks

Fast I/O (input/output) is facilitated with the use of local scratch disks available on the compute nodes.

note

IMPORTANT: It is essential to make efficient use of Elja and to not slow down network traffic on the cluster. Hence, it is advisable to copy the data and input for your job to the local scratch disk on the compute nodes (/scratch/users/) and launch the program from there. If this step is omitted, the program will run remotely on the compute nodes but will constantly read/write from/to the submit directory on the NFS server. This creates a lot of network traffic, slows down the use of Elja for everybody, and will also slow down the job itself.

Using /scratch/users/ with SBATCH

The example here assumes the user is working with SBATCH scripts in some work directory in their home folder. For example:

[..]$ pwd
[..]$ /users/home/<uname>/myjob

and your SBATCH submit script is in the same directory:

[..]$ /users/home/<uname>/myjob/submit.sh

Example bash lines for using /scratch/users/ are provided below. The following lines should be added to your SBATCH script. An example SBATCH script is provided at the end.

[..]$ cat submit.sh
..
# Location of scratch directory on the compute nodes
scratchlocation=/scratch/users

# Create a user directory if it does not exist
if [ ! -d $scratchlocation/$USER ]; then
    mkdir -p $scratchlocation/$USER
fi

# Create a temporary directory with a unique identifier associated with your jobid
tdir=$(mktemp -d $scratchlocation/$USER/$SLURM_JOB_ID-XXXX)

# Go to the temporary directory
cd $tdir

# Exit if tdir does not exist
if [ ! -d $tdir ]; then
    echo "Temporary scratch directory does not exist ..."
    echo "Something is wrong, contact support."
    exit
fi

# Copy the necessary input files to run your job
scp -l 40000 $SLURM_SUBMIT_DIR/myinput $tdir

# If the program needs many input files you can add a separate line for each file.

# If your job requires a directory of input files
# cp -r $SLURM_SUBMIT_DIR/myinputdir $tdir/

# Now run the job from the temporary directory e.g.
myprogram myinput

# After the job is completed make sure to copy the output to your submit directory.
scp -l 40000 $tdir/myoutputfiles $SLURM_SUBMIT_DIR/

# If the program produces many output files you can add a separate line for each file.
# Please try to only copy the files that you need.

# IMPORTANT: Delete the temporary directory and all of its content
rm -rf $tdir

An example SBATCH script for a typical Python job is provided here. Similar commands can be used when the user is in an interactive session.

Using /scratch/users/ on an interactive session

The example here assumes the user is working in an interactive session and in some work directory in the user's home folder:

[..]$ pwd
[..]$ /users/home/<uname>/myjob

Example bash lines for using /scratch/users/ while on an interactive session are provided below. Information on how to connect to an interactive session can be found here.

$ export scratchlocation=/scratch/users 

# Create a user directory if it does not exist
$ [ ! -d "$scratchlocation/$USER" ] && mkdir "$scratchlocation/$USER"

# Create a temporary directory with a unique identifier associated with your jobid
$ tdir=$(mktemp -d "$scratchlocation/$USER/temp_dir.XXXXXX")

# Go to the temporary directory
$ cd $tdir

# Copy the necessary input files to run your job
$ cp submit_to_scratch_dir $tdir/

# Run the program from your temporary directory
myprogram myinput

# After the job is completed make sure to copy the output to your submit directory.
$ cp $tdir/myoutputfiles submit_to_scratch_dir/

# If the program produces many output files you can add a separate line for each file.
# Please try to only copy the files that you need.

# IMPORTANT: Delete the temporary directory and all of its content
$ rm -rf $tdir

An example SBATCH script for a typical Python job is provided here. Similar commands can be used when the user is in an interactive session.