Discover gists

Multi-node-training on slurm with PyTorch

What's this?

A simple note for how to start multi-node-training on slurm scheduler with PyTorch.
Useful especially when scheduler is too busy that you cannot get multiple GPUs allocated, or you need more than 4 GPUs for a single job.
Requirement: Have to use PyTorch DistributedDataParallel(DDP) for this purpose.
Warning: might need to re-factor your own code.
Warning: might be secretly condemned by your colleagues because using too many GPUs.

*This Dockerfile is intended for SvelteKit applications that use adapter-node. So, the Dockerfile below assumes that you have already installed and configured the adapter.

Dockerfile:

FROM node:18-alpine AS builder
WORKDIR /app
COPY package*.json .
RUN npm ci
COPY . .
RUN npm run build

Deploy Django and React APP in a production VPS (Django + React + PostgreSQL + NGINX + Ubuntu Server)

Assuming You have backend and frontend codes in /home/backend and /home/frontend/ (Use git to upload)

Install required Packages from the Ubuntu Repositories

sudo apt-get update
sudo apt-get install python3-pip python3-dev libpq-dev postgresql postgresql-contrib nginx

Setting up Lando on Fedora 39

Update packages
sudo dnf -y update

Disable cgroups v2 in the kernel
sudo grubby --update-kernel=ALL --args="systemd.unified_cgroup_hierarchy=0"
Reboot.

Uninstall docker if you already did and install Docker CE.

$/
  artifacts/
  build/
  docs/
  lib/
  packages/
  samples/
  src/
 tests/

	Visual Studio 2019 Product Key

	[Please Star this gist]

	Follow My Account --> https://github.com/ch-kashif @ch-kashif

	Lets do a code together
	Join Cloud Disk repository --> https://github.com/ch-kashif/CloudDisk

	BLACK => "\033[30m",
	RED => "\033[31m",
	GREEN => "\033[32m",
	YELLOW => "\033[33m",
	BLUE => "\033[34m",
	PURPLE => "\033[35m",
	CYAN => "\033[36m",
	WHITE => "\033[37m",

	# background color

	/*------------------------------------------
	Responsive Grid Media Queries - 1280, 1024, 768, 480
	1280-1024 - desktop (default grid)
	1024-768 - tablet landscape
	768-480 - tablet
	480-less - phone landscape & smaller
	--------------------------------------------*/
	@media all and (min-width: 1024px) and (max-width: 1280px) { }

	@media all and (min-width: 768px) and (max-width: 1024px) { }

	#!/bin/bash

	# Remove existing "nameserver" lines from /etc/resolv.conf
	sed -i '/nameserver/d' /etc/resolv.conf

	# Run the PowerShell command to generate "nameserver" lines and append to /etc/resolv.conf
	# we use full path here to support boot command with root user
	/mnt/c/Windows/System32/WindowsPowerShell/v1.0/powershell.exe -Command '(Get-DnsClientServerAddress -AddressFamily IPv4).ServerAddresses \| ForEach-Object { "nameserver $_" }' \| tr -d '\r'\| tee -a /etc/resolv.conf > /dev/null