Install the pacakages:

sudo apt install -y munge libmunge-dev

Create munge keys

sudo /usr/sbin/create-munge-key
sudo chown munge: /etc/munge/munge.key
sudo chmod 400 /etc/munge/munge.key

If the above fails do it manually with

sudo mungekey --create --force --keyfile /etc/munge/munge.key
sudo chown -R munge: /etc/munge /var/log/munge /var/lib/munge /run/munge
sudo chmod 0700 /etc/munge /var/log/munge /var/lib/munge
sudo chmod 0755 /run/munge

Start and check munge

sudo systemctl enable munge
sudo systemctl start munge
sudo systemctl status munge
munge -n | unmunge

Result should be

STATUS:          Success (0)
ENCODE_HOST:     user-G4101 (127.0.1.1)
ENCODE_TIME:     2025-08-25 15:05:03 -0700 (1756159503)
DECODE_TIME:     2025-08-25 15:05:03 -0700 (1756159503)
TTL:             300
CIPHER:          aes128 (4)
MAC:             sha256 (5)
ZIP:             none (0)
UID:             subho (1002)
GID:             subho (1002)
LENGTH:          0

Install wsl slurm of ubuntu with (though it is easier to install, this is old and partial gpu feature often fails, for advance features compile it)

sudo apt install -y slurm-wlm

Commands

• Show fairshare usage for various queue and partitions sacctmgr show qos.
• Show user’s fairshare score sshare -l | grep usernameOrGrpName.
• To obtain basic partition info sinfo -p sulcgpu3 -o "%P %a %l %D %t %N" or sinfo -Nel | grep "sulcgpu3".
• For information on a particular node scontrol show node nodeName
• To obtain queue info use squeue -p parititonName or squeue -u userName
• To submit a empty job sbatch -n 20 --gres=gpu:1 --time=4:00:00 -q sulcgpu1 -p sulcgpu1 --output=/dev/null --error=/dev/null $@ _interactiveScript

where the _interactiveScript can be a very simple command like blank screen.

#!/bin/sh
# Simple batch script that starts SCREEN.

exec screen -Dm -S slurm$SLURM_JOB_ID

• Empty jobs can also be submited using slurm script as given below

#!/bin/sh
#SBATCH -q private
#SBATCH -p general
#SBATCH -G a100:1
#SBATCH -t 5-00:00
#SBATCH -c 20
#SBATCH -o empty.out
#SBATCH -e empty.err
#SBATCH -J PHBdnaD

exec screen -Dm -S slurm$SLURM_JOB_ID

Now to enter into the submitted job, one can use srun --pty --jobid=######## /bin/bash or simply ssh nodeid.

Setting up slurmdbd

• Install mariadb

  sudo apt-get install mariadb-server
  sudo systemctl enable mariadb
  sudo systemctl start mariadb
  

• Generate a database to use with slurm using ```bash sudo mysql -u root -p CREATE DATABASE slurm_acct_db; CREATE USER ‘slurmuser’@’localhost’ IDENTIFIED BY ‘slurmPass’; GRANT ALL ON slurm_acct_db.* TO ‘slurmuser’@’localhost’; FLUSH PRIVILEGES; EXIT;

- Default port fro mariadb is 3306 use `sudo ss -tulpn` to identify that.
- change the following parameters in mydraid `sudo nvim /etc/mysql/mariadb.conf.d/50-server.cnf`

```bash
innodb_buffer_pool_size = 1G
innodb_lock_wait_timeout = 300

• sudo systemctl restart mariadb; sudo systemctl status mariadb

• sudo systemctl restart mariadb; sudo systemctl status mariadb

• A sample slurmdb.conf at /etc/slurm/slurmdbd.conf is given below:

#
# Sample /etc/slurmdbd.conf
#
ArchiveEvents=yes
ArchiveJobs=yes
ArchiveResvs=yes
ArchiveSteps=no
ArchiveSuspend=no
ArchiveTXN=no
ArchiveUsage=no
#ArchiveScript=/usr/sbin/slurm.dbd.archive
AuthInfo=/var/run/munge/munge.socket.2
AuthType=auth/munge
DbdHost=localhost
DbdPort=6819
DebugLevel=info
PurgeEventAfter=1month
PurgeJobAfter=12month
PurgeResvAfter=1month
PurgeStepAfter=1month
PurgeSuspendAfter=1month
PurgeTXNAfter=12month
PurgeUsageAfter=24month
LogFile=/var/log/slurm/slurmdbd.log
PidFile=/var/run/slurm/slurmdbd.pid
SlurmUser=slurm
StoragePass=slurmPass
StorageType=accounting_storage/mysql
StorageUser=slurmuser
StoragePort=3306

• Sample slurm.conf at /etc/slurm/slurm.conf is given below:

ClusterName=ringtail
SlurmctldHost=ringtail

MailProg=/usr/bin/mail

ProctrackType=proctrack/cgroup

ReturnToService=1
SlurmctldPidFile=/var/run/slurmctld.pid
SlurmctldPort=6817
SlurmdPidFile=/var/run/slurmd.pid
SlurmdPort=6818
SlurmdSpoolDir=/var/spool/slurmd
SlurmUser=slurm
AuthType=auth/munge

StateSaveLocation=/var/spool/slurmctld

TaskPlugin=task/affinity,task/cgroup

InactiveLimit=0
KillWait=30

MinJobAge=300

SlurmctldTimeout=120
SlurmdTimeout=300

Waittime=0

SchedulerType=sched/backfill
SelectType=select/cons_tres

AccountingStorageEnforce=limits,qos,safe
AccountingStorageHost=localhost

AccountingStoragePort=6819
AccountingStorageType=accounting_storage/slurmdbd

JobCompLoc=/var/log/slurm/job_completions.log

JobCompType=jobcomp/filetxt

JobAcctGatherFrequency=30

SlurmctldDebug=info
SlurmctldLogFile=/var/log/slurm/slurmctld.log
SlurmdDebug=info
SlurmdLogFile=/var/log/slurm/slurmd.log

# COMPUTE NODES
GresTypes=gpu,mps
NodeName=gpu CPUs=128 Sockets=1 CoresPerSocket=128 ThreadsPerCore=1 Gres=gpu:rtx4060:1,gpu:rtx4080:1,mps:200 NodeHostname=ringtail NodeAddr=ringtail State=idle RealMemory=96092
PartitionName=general Nodes=ALL Default=YES MaxTime=INFINITE State=UP

• sudo sacctmgr add account generic cluster=ringtail description="Generic Account" organization="SulcLab

• sacctmgr list account

• sudo sacctmgr add user subho account=generic cluster=ringtail

Conclusion

SLURM is one of the best open source workload manager and is highly used in modern servers.

References

• https://slurm.schedmd.com/classic_fair_share.html
• https://slurm.schedmd.com/fair_tree.html

To run a restrained simulation it is advisable to run it with hydrogen bond restrains, which can be generated in oxview, using the following steps:

Dynamics -> Forces -> Set stiffness = ( something between 2 to 5) and Relaxed distance 1.2 -> create from base pair -> Yes.

It will take some time to search the base pairs and will generarte restrains for all the base pairs. Save the structure top and dat file and also the force file as well.

These structure could be simulated with MC then MD with restrains and finally production run without any restrains or max backbone force.

MC Simulation 1 - Restrains + Max Backbone Force

A sample input for MC simulation with restraints and max backbone force:

An astrick # indicates a comment

The input section takes all the file input

##############################
####  PROGRAM PARAMETERS  ####
##############################
interaction_type=DNA2
salt_concentration=1.0
sim_type = MC
backend = CPU
#backend_precision = mixed
#mismatch_repulsion = 1
#debug = 1
#seed = 42
#CUDA_list = verlet
#CUDA_sort_every = 0
#use_edge = 1
#edge_n_forces = 1

##############################
####    SIM PARAMETERS    ####
##############################
steps = 1e5
max_backbone_force = 10.
ensemble = nvt
delta_translation = 0.02
delta_rotation = 0.04
use_average_seq = no
seq_dep_file = oxDNA2_sequence_dependent_parameters.txt
thermostat = john
T = 25C
dt = 0.003
verlet_skin = 0.6
maxclust = 63
diff_coeff = 2.5
newtonian_steps = 103
max_density_multiplier=5

##############################
####    INPUT             ####
##############################
topology = output.top
conf_file = output.dat
external_forces = 1
external_forces_file = output_force.txt

##############################
####     OUTPUT           ####
##############################
lastconf_file = last_conf.dat
trajectory_file = trajectoryRelaxCPU.dat
max_io = 5
refresh_vel = 1
log_file = log.dat
no_stdout_energy = 0
restart_step_counter = 0
energy_file = energyRelaxCPU.dat
print_conf_interval = 1e4
print_energy_every = 1e4
time_scale = linear

MD Simulation 2 - Restrains + Max Backbone Force

A input with Restrains and Max Backbone Force, starting from the last configuration:

##############################
####  PROGRAM PARAMETERS  ####
##############################
interaction_type=DNA2
salt_concentration=1.0
sim_type = MD
backend = CUDA
backend_precision = mixed
#mismatch_repulsion = 1
#debug = 1
#seed = 42
CUDA_list = verlet
CUDA_sort_every = 0
use_edge = 1
edge_n_forces = 1

##############################
####    SIM PARAMETERS    ####
##############################
steps = 1e7
max_backbone_force = 10.
ensemble = nvt
delta_translation = 0.02
delta_rotation = 0.04
use_average_seq = no
seq_dep_file = oxDNA2_sequence_dependent_parameters.txt
thermostat = john
T = 25C
dt = 0.003
verlet_skin = 0.6
maxclust = 63
diff_coeff = 2.5
newtonian_steps = 103
max_density_multiplier=5

##############################
####    INPUT             ####
##############################
topology = output.top
conf_file = last_conf.dat
external_forces = 1
external_forces_file = output_force.txt

##############################
####     OUTPUT           ####
##############################
lastconf_file = last_conf.dat
trajectory_file = trajectoryRelaxMD.dat
max_io = 5
refresh_vel = 1
log_file = log.dat
no_stdout_energy = 0
restart_step_counter = 0
energy_file = energyRelaxMD.dat
print_conf_interval = 1e5
print_energy_every = 1e5
time_scale = linear

MD Simulation 3 - High Max Backbone Force

With a fully relaxed structure comment the external forces section and increase the max_backbone_force to 60 or more and run for 1e7 steps. An input file would be something like.

##############################
####  PROGRAM PARAMETERS  ####
##############################
interaction_type=DNA2
salt_concentration=1.0
sim_type = MD
backend = CUDA
backend_precision = mixed
#mismatch_repulsion = 1
#debug = 1
#seed = 42
CUDA_list = verlet
CUDA_sort_every = 0
use_edge = 1
edge_n_forces = 1

##############################
####    SIM PARAMETERS    ####
##############################
steps = 1e7
max_backbone_force = 60.
ensemble = nvt
delta_translation = 0.02
delta_rotation = 0.04
use_average_seq = no
seq_dep_file = oxDNA2_sequence_dependent_parameters.txt
thermostat = john
T = 25C
dt = 0.005
verlet_skin = 0.6
maxclust = 63
diff_coeff = 2.5
newtonian_steps = 103
max_density_multiplier=5

##############################
####    INPUT             ####
##############################
topology = output.top
conf_file = last_conf.dat
#external_forces = 1
#external_forces_file = output_force.txt

##############################
####     OUTPUT           ####
##############################
lastconf_file = last_conf.dat
trajectory_file = trajectory.dat
max_io = 5
refresh_vel = 1
log_file = log.dat
no_stdout_energy = 0
restart_step_counter = 0
energy_file = energy.dat
print_conf_interval = 1e5
print_energy_every = 1e5
time_scale = linear

MD Simulation 4 - Production

After everything is finished run the final production simulation with the following input file. It is recommended to run it for atleast 1e9 steps. One might also want to run multiple replicas to ensure statistical convergence or to explore the conformational space more thoroughly. dt is recommended = 0.3 to 0.5, could be kept at a lower value if needed, but not higher than 0.5. Also the saving frequency could be adjusted accordingly, generally 1e5 to 5e5 produces a good statistical sampling. Adjusting the verlet skip could help with performance and memory usage.

##############################
####  PROGRAM PARAMETERS  ####
##############################
interaction_type=DNA2
salt_concentration=1.0
sim_type = MD
backend = CUDA
backend_precision = mixed
#mismatch_repulsion = 1
#debug = 1
#seed = 42
CUDA_list = verlet
CUDA_sort_every = 0
use_edge = 1
edge_n_forces = 1

##############################
####    SIM PARAMETERS    ####
##############################
steps = 1e9
#max_backbone_force = 10.
ensemble = nvt
delta_translation = 0.02
delta_rotation = 0.04
use_average_seq = no
seq_dep_file = oxDNA2_sequence_dependent_parameters.txt
thermostat = john
T = 25C
dt = 0.003
verlet_skin = 0.6
maxclust = 63
diff_coeff = 2.5
newtonian_steps = 103
max_density_multiplier=5

##############################
####    INPUT             ####
##############################
topology = output.top
conf_file = last_conf.dat
#external_forces = 1
#external_forces_file = output_force.txt

##############################
####     OUTPUT           ####
##############################
lastconf_file = last_conf.dat
trajectory_file = trajectory.dat
max_io = 5
refresh_vel = 1
log_file = log.dat
no_stdout_energy = 0
restart_step_counter = 0
energy_file = energyRelaxMD.dat
print_conf_interval = 1e5
print_energy_every = 1e5
time_scale = linear

To run the simulation at the same time, one could save the above input files as inputMC, inputMDRelax, inputMDEqui, input. Then write a simple job script to submit all the simulations.

#!/bin/sh
#SBATCH -q private
#SBATCH -p general
#SBATCH -t 7-00:00
#SBATCH -c 2
#SBATCH -G 1
#SBATCH -o empty.out
#SBATCH -e empty.err
#SBATCH -J oxDNASimulation

module load cuda-12.1.1-gcc-12.1.0 gcc-12.1.0-gcc-11.2.0 cmake eigen-3.4.0-gcc-11.2.0

export CUDA_MPS_PIPE_DIRECTORY=/tmp/mps-pipe_$SLURM_TASK_PID
export CUDA_MPS_LOG_DIRECTORY=/tmp/mps-log_$SLURM_TASK_PID
mkdir -p $CUDA_MPS_PIPE_DIRECTORY
mkdir -p $CUDA_MPS_LOG_DIRECTORY
nvidia-cuda-mps-control -d

echo "Starting jobs"

oxDNA inputMC > outMC.log
oxDNA inputMDRelax > outMDRelax.log
oxDNA inputMDEqui > outMDEqui.log
oxDNA input > out.log

echo "Done"
exec screen -Dm -S slurm$SLURM_JOB_ID

    sudo pacman -Syu gcc make cmake python3 python-pip git wget nvim zsh cloudflared openssh rsync rclone
    sudo chsh -s $(which zsh) $(whoami)

# Then, ensure visudo has:
# %wheel ALL=(ALL:ALL) ALL
    

# To add 'myuser' to the 'sudo' group
sudo usermod -aG sudo myuser
    

# To add 'myuser' to the 'wheel' group
sudo usermod -aG wheel myuser

# Typically, wheel group is already configured in /etc/sudoers
    

oh-my-zsh installation

## Install oh-my-zsh
sh -c "$(wget https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh -O -)"

## Install plugins
git clone https://github.com/zsh-users/zsh-syntax-highlighting.git ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-syntax-highlighting
git clone https://github.com/zsh-users/zsh-autosuggestions ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-autosuggestions
git clone https://github.com/zsh-users/zsh-history-substring-search ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-history-substring-search

## Replace the default plugins with the following
sed -i.bak '/^plugins=.*/c\
plugins=(\
  git\
  zsh-syntax-highlighting\
  zsh-autosuggestions\
  zsh-history-substring-search\
)' ~/.zshrc

Generate SSH keys

ssh-keygen -t ed25519 -C "pc"
cat ~/.ssh/id_ed25519.pub

Add servers

mkdir -p ~/.ssh
echo "Host gila
        Hostname gila.navraj.me
        User gila
        Port 22
        ProxyCommand cloudflared access ssh --hostname %h
Host sol
        Hostname sol.asu.edu
        User sroy85
        Port 22
        ProxyJump gila" >> ~/.ssh/config
    

Add zshrc functions

mo() {
  # Check if an argument was provided
  if [ -z "$1" ]; then
    echo "Usage: mo <directory_name>"
    return 1 # Indicate an error
  fi

  # Attempt to create the directory.
  # -p: no error if existing, make parent directories as needed
  # --: ensures that $1 isn't misinterpreted as an option if it starts with '-'
  if mkdir -p -- "$1"; then
    # If mkdir was successful, try to cd into it
    if cd -- "$1"; then
      return 0 # Success
    else
      echo "Error: Successfully created directory '$1', but could not cd into it." >&2
      return 1 # Indicate an error
    fi
  else
    # mkdir failed. mkdir itself usually prints an error.
    # We can add a generic one too.
    echo "Error: Could not create directory '$1'." >&2
    return 1 # Indicate an error
  fi
}

• Welcome message is in file /etc/motd, and ASCII art looks great in there.
• OS choices
  1. Debian
  2. Fedora
  3. Arch
  4. Ubuntu (May even damage firmware)

Useful Software

Desktop Environments

• Xfce is one of the best and easy to configure desktop environment.
• To install it sudo apt install xfce4
• to start it startxfce4 from tty terminal.

Fixes

SLURM GPU segregation fix

Slurm uses nvml to detect the number of gpus, segregate them during running, and this api was not working for my case.

• Find the library location sudo find / -name "libnvidia-ml.so" , for my case this was /usr/local/cuda-12.2/targets/sbsa-linux/lib/stubs/libnvidia-ml.so
• Copy this to /usr/lib
• Add header location to /etc/profile , for my case it was

  export C_INCLUDE_PATH="/usr/local/cuda-12.2/targets/sbsa-linux/include:$C_INCLUDE_PATH"
  export CPLUS_INCLUDE_PATH="/usr/local/cuda-12.2/targets/sbsa-linux/include:$CPLUS_INCLUDE_PATH"
  

• By default /usr/lib should be in ldconfig still one can add that in /etc/ld.so.conf.d . sudo ldconfig
• Also here is a c code to check nvml, use gcc nvml.c -lnvidia-ml; ./a.out. If it outputs the correct number of gpu, I guess everything should be configured.

#include <stdio.h>
#include <nvml.h>

int main() {
    nvmlReturn_t result;
    unsigned int device_count = 0;

    // Initialize NVML library
    result = nvmlInit();

    if (NVML_SUCCESS != result) {
        printf("Failed to initialize NVML: %s\n", nvmlErrorString(result));
        return 1;
    }

    // Get the number of GPUs
    result = nvmlDeviceGetCount(&device_count);

    if (NVML_SUCCESS != result) {
        printf("Failed to get device count: %s\n", nvmlErrorString(result));
        nvmlShutdown();
        return 1;
    }
    printf("Number of GPUs detected: %u\n", device_count);

    // Clean up NVML library
    result = nvmlShutdown();

    if (NVML_SUCCESS != result) {
        printf("Failed to shutdown NVML: %s\n", nvmlErrorString(result));
        return 1;

    }
    return 0;
}

Nvidia kernel stuff

• To rebuild the kernel simply sudo akmods --force
• Logs should be in /var/cache/akmods/nvidia/{Kernel Version}
• Check kernel version with uname -srm

Bad ideas

• Never change /usr/bin/gcc to /usr/bin/gcc-13 from default gcc-14 (or whatever your version is). Same for g++. It breaks core system files including nvidia drivers akmod.

cd "SSL path"
mkdir -p Fingerprints
openssl req -x509 -nodes -days 365 -subj /CN=barrier -newkey rsa:4096 -keyout Barrier.pem -out Barrier.pem
openssl x509 -fingerprint -sha256 -noout -in Barrier.pem > Fingerprints/Local.txt
sed -e "s/.*=/v2:sha256:/" -i Fingerprints/Local.txt

• Set the server config by draging in new pc to left or right.
• Start the server.
• Start the client with the server IP. If everything is right client would flash server SSL and in next message would prompt configuration is ready.

• Install oh-my-zsh sh -c "$(wget https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh -O -)"
• Update in ~/.zshrc file

ZSH_THEME="avit"
plugins=(
  git
  zsh-syntax-highlighting
  zsh-autosuggestions
  zsh-history-substring-search
)

• Install syntax-highlight using `

• Install autosuggestion using git clone https://github.com/zsh-users/zsh-autosuggestions ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-autosuggestions

• Install zsh history substring search git clone https://github.com/zsh-users/zsh-history-substring-search ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/git clone https://github.com/zsh-users/zsh-syntax-highlighting.git ${ZSH_CUSTOM:-~/.oh-my-zsh/custom}/plugins/zsh-syntax-highlightingzsh-history-substring-search`

• std::ios::ate - full file is available for writting and the pointer will be at the end of the file in the beginning.
• std::ios::app - starts writting at the end of the file and existing porting is non writable.
• std::ios::trunc - will delete everything in the file and starts writting afresh.
• std::ios::out - used to creat a file if doesn’t exists, writting behaviour is dependant on trunc or ate etc.
• std::ios::binary - most useful mode to generate binary files.

• docker run --rm delets the containers after the run is finished.
• docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi - run using nvidia gpu all can be substituted with device=0 to select a specific gpu.

Container Commands

• docker container ls -a list all the containers