All the action bellow must be performed as a regular user (no root is required). Do not attempt to use sudo.

To install Tensorflow on a specific node with GPU use the following commands:

ssh compute0xx (compute node with GPU)
cd /opt/anaconda3/bin/
conda create -n test_env --clone=/opt/anaconda3
source activate test_env
conda remove conda
conda remove conda-env
conda remove conda-build
conda update --all
conda install python=3.6
conda install -c anaconda tensorflow-gpu
pip install keras==2.2.4(or a compatible version with Tensorflow)

In order to use the configured environment on login, add the following lines into the .bashrc file from your home directory (use your editor of choice):

cd /opt/anaconda3/bin/
source activate test_env
cd ~/

Test for GPU

Keras:

from keras import backend as K
K.tensorflow_backend._get_available_gpus()

TF:

from tensorflow.python.client import device_lib
print(device_lib.list_local_devices())

To check if the program is running on GPU use the following command to see the running programs on GPU:

nvidia-smi -l 1

To check the supported versions for Tensorflow please consult the following link: https://www.tensorflow.org/install/source#linux

Credit: Pop Catalin-Mihai, 2^nd year, HPC master, 2018-2020 class