Discover gists

In most of deep learning projects, the training scripts always start with lines to load in data, which can easily take a handful minutes. Only after data ready can start testing my buggy code. It is so frustratingly often that I wait for ten minutes just to find I made a stupid typo, then I have to restart and wait for another ten minutes hoping no other typos are made.

In order to make my life easy, I devote lots of effort to reduce the overhead of I/O loading. Here I list some useful tricks I found and hope they also save you some time.

use Numpy Memmap to load array and say goodbye to HDF5.

I used to relay on HDF5 to read/write data, especially when loading only sub-part of all data. Yet that was before I realized how fast and charming Numpy Memmapfile is. In short, Memmapfile does not load in the whole array at open, and only later "lazily" load in the parts that are required for real operations.

Sometimes I may want to copy the full array to memory at once, as it makes later operations

How to push tagged Docker releases to Google Artifact Registry with a GitHub Action

Here's how I configured a GitHub Action so that a new version issued by GitHub's release interface will build a Dockerfile, tag it with the version number and upload it to Google Artifact Registry.

Before you attempt the steps below, you need the following:

A GitHub repository that contains a working Dockerfile
The Google Cloud SDK tool gcloud installed and authenticated

Create a Workload Identity Federation

I've recently joined Amazon Dublin from India and got opportunities to interview with Meta London, Zalando Berlin & some other companies. I extensively researched about companies hiring internationally which support visa & relocation for Tech roles. So sharing list of companies:

Do consider to STAR, if it helped you.

London

Enable JIT and NOGIL on Python 3.13.0a5 with pyenv

Environment

$ uname -vorm
6.5.0-25-generic #25-Ubuntu SMP PREEMPT_DYNAMIC Wed Feb  7 14:58:39 UTC 2024 x86_64 GNU/Linux

$ lsb_release -a
No LSB modules are available.

	// ==UserScript==
	// @name arte stream url
	// @namespace http://tampermonkey.net/
	// @version 0.1.7
	// @description get arte m3u8 stream url
	// @author You
	// @match https://www.arte.tv/de/videos/*
	// @match https://www.arte.tv/fr/videos/*
	// ==/UserScript==

	// frontend - react

	import { useState } from "react";
	import Alert from "@reach/alert";

	export default function Contact() {
	const [name, setName] = useState("");
	const [email, setEmail] = useState("");
	const [message, setMessage] = useState("");
	const [messageSent, setMessageSent] = useState(false);

	http://www.productkeysfree.com/2016/08/windows-8-product-keys.html

	Windows 8 Product Key ( 32 bit )
	XKY4K-2NRWR-8F6P2-448RF-CRYQH
	TK8TP-9JN6P-7X7WW-RFFTV-B7QPF
	QXV7B-K78W2-QGPR6-9FWH9-KGMM7
	RR3BN-3YY9P-9D7FC-7J4YF-QGJXW
	FB4WR-32NVD-4RW79-XQFWH-CYQG3

	Windows 8 Product Key ( 64 bit )

	; Press Ctrl+Shift+Space to set any currently active window to be always on top.
	; Press Ctrl+Shift+Space again set the window to no longer be always on top.
	; Source: https://www.howtogeek.com/196958/the-3-best-ways-to-make-a-window-always-on-top-on-windows

	^+SPACE::
	WinGetTitle, activeWindow, A
	if IsWindowAlwaysOnTop(activeWindow) {
	notificationMessage := "The window """ . activeWindow . """ is now always on top."
	notificationIcon := 16 + 1 ; No notification sound (16) + Info icon (1)
	}

	# CLI
	sudo apt update -y
	sudo apt install -y \
	git curl btop \
	docker.io docker-buildx \
	build-essential pkg-config autoconf bison rustc cargo clang \
	libssl-dev libreadline-dev zlib1g-dev libyaml-dev libreadline-dev libncurses5-dev libffi-dev libgdbm-dev libjemalloc2 \
	libvips imagemagick libmagickwand-dev mupdf mupdf-tools \
	redis-tools sqlite3 libsqlite3-0 libmysqlclient-dev \
	rbenv apache2-utils

	HANDY ONE-LINE SCRIPTS FOR AWK 30 April 2008
	Compiled by Eric Pement - eric [at] pement.org version 0.27

	Latest version of this file (in English) is usually at:
	http://www.pement.org/awk/awk1line.txt

	This file will also be available in other languages:
	Chinese - http://ximix.org/translation/awk1line_zh-CN.txt

	USAGE: