Braindumps
CPUID
On Intel CPUs, the CPUID can be used to determine which firmware to load:[1]
$ lscpu | grep -e '^CPU family:' -e '^Model:' -e '^Stepping:' CPU family: 6 Model: 142 Stepping: 9
Converting these values to hex:
$ printf "%x " 6 142 9; echo 6 8e 9
Or, with the cpuid utility:
$ cpuid | grep -m3 -e 'family synth' -e 'model synth' -e 'stepping id'
(family synth) = 0x6 (6)
(model synth) = 0x8e (142)
stepping id = 0x9 (9)
Install the correct firmware from your distribution or from upstream repository:
$ ls -go /lib/firmware/intel-ucode/06-8e-09 -rw-r--r--. 1 211968 Aug 10 02:00 /lib/firmware/intel-ucode/06-8e-09
Reload the microcode via reboot or via:
echo 1 > /sys/devices/system/cpu/microcode/reload
BIOS or UEFI
Find out if we booted into BIOS or UEFI mode:[2]
[ -d /sys/firmware/efi ] && echo UEFI || echo BIOS
Faketime
Changing what time a process thinks it is with libfaketime:
$ touch -t $(date -d 1970-01-02\ 00:00:00 +%Y%m%d%H%M.%S) /tmp/foo $ ls -go /tmp/foo -rw-r-----. 1 0 Jan 2 1970 /tmp/foo $ FAKETIME_FMT="%s" faketime -f $(stat -c %Y /tmp/foo) date Fri 2 Jan 00:00:00 CET 1970
Or, in a more permanent fashion:
$ export FAKETIME_FMT="%s" LD_PRELOAD=/usr/lib64/faketime/libfaketime.so.1 FAKETIME=$(stat -c %Y /tmp/foo) $ date Fri 2 Jan 00:00:00 CET 1970
Chromecast
Stream to a Chromecast device [3] via VLC:
vlc --sout-chromecast-ip=10.0.0.3 --sout "#chromecast" --demux-filter=demux_chromecast movie.avi
Via Cast All The Things:
$ cat .config/catt/catt.cfg [options] device = 10.0.0.3 $ catt cast movie.avi $ catt stop # Stop and disconnect from Chromecast
Via chromecast-cli:
$ chromecast --host 10.0.0.3 play movie.avi Playing movie.avi Status IDLE $ chromecast --host chromecast stop # Stop and disconnect from Chromecast
Via Go-Chromecast:
$ go-chromecast load --addr 10.0.0.3 movie.avi $ go-chromecast stop --addr 10.0.0.3 # Stop and disconnect from Chromecast
X11
Find out if Xorg or Wayland is being used:[4]
$ loginctl show-session -p Type $(loginctl | awk '/tty/ {print $1}')
Type=x11
Similarly, one can lock and (remotely) unlock the session:
SESSION=$(loginctl | awk '/tty/ {print $1}')
loginctl lock-session ${SESSION} && sleep 3 && loginctl unlock-session ${SESSION}
HTTP Server
Single-line HTTP server examples:[5]
python -m http.server 8000
With Python2 one needs SimpleHTTPServer[6] for this:
python2 -m SimpleHTTPServer 8000
PHP:
php -S localhost:8000
Clear HSTS policies
With Google Chrome and derivatives:
chrome://net-internals/#hsts
Query, and if needed clear with Delete domain security policies. Note: this is only possible for custom domains, not preloaded domain names.
With Mozilla Firefox, it's a bit more tricky. Either
- Open the browser history (
CTRL+Shift+H) - Search for the specific site and select "Forget about this site". Note: this will delete all history about this site, not only the HSTS information!
Or, as a more specific measure:
- Close Firefox
- Remove the site entry from
SiteSecurityServiceState.txtin the profile directory. The HPKP bits can be cleared as well.
$ grep -i atlas.tor ~/.mozilla/firefox/default/SiteSecurityServiceState.txt atlas.torproject.org:HPKP 1 18165 1574726371209,1,0,XAup3/gyt3mpe4649a5dcd2abe03c6f14b48bfd7f2c=N/0ZhkGAd381599bd99c75fb548f34fbfa6b3ffbg+I= atlas.torproject.org:HSTS 2 18474 1611955423140,1,0,2
Delete files with rsync
How to delete a file with rsync:[7]
rsync -rv --delete --include=foo.txt --exclude="*" . user@remote:/dir/
Or, with --remove-source-files:
rsync -rv --remove-source-files remote:/dir/foo.txt /tmp/ rm /tmp/foo.txt
/proc/net/fib_trie
Display configured IP addresses if no network tools are installed, via the kernel's representation[8] of its FIB trie:
$ awk '/32 host/ {print f} {f=$2}' <<< "$(</proc/net/fib_trie)" | sort -u
10.1.1.123
127.0.0.1
192.168.0.123
192.168.3.123
Or, a less esoteric query:
$ awk '/\|--/ && !/\.0\.0$/ && !/\.255\.255$/ {print $2}' /proc/net/fib_trie | sort -u
10.6.41.171
127.0.0.1
169.254.1.1
initrd
Here's how to generate and boot into a custom initrd.
- Build a kernel with
CONFIG_BLK_DEV_INITRD=y. - Have a busybox installation available in
/sbin/busybox. - Build
initrd.cpiowith the following script:
#!/bin/sh -e outfile=$(pwd)/initrd.cpio cd $(mktemp -d) echo "Preparing initrd in $(pwd)..." mkdir bin dev mnt proc sys cat > init << EOF #!/bin/sh mount -t devtmpfs none /dev mount -t proc none /proc mount -t sysfs none /sys mount -t ext4 LABEL=foo /mnt || true exec /bin/sh EOF chmod +x init cp /sbin/busybox bin/ bin/busybox --install bin find . | cpio -o -H newc > $outfile echo "Written to $outfile."
Optional, create a disk image:
dd if=/dev/zero of=ext4.img bs=1M count=64
mkfs.ext4 -F -L foo ext4.img
Boot the kernel and load the initrd:
qemu-system-x86_64 -kernel $SRC/arch/x86/boot/bzImage -initrd initrd.cpio -m 512M \ -drive file=ext4.img,index=0,format=raw,if=virtio
Use if=ide or other interfaces as needed, depending what the kernel was configured with.[9]
sed
We should probably open up a whole article for sed(1) examples.
Copyright notes
I needed to add a copyright note[10] to some scripts, but since it's only a cosmetic update, I did not want to alter the timestamp of each script. We will do that anyway, but restore it later on:
ls *.sh | while read f; do
if ! grep -qi '(c)' "${f}"; then
echo "FILE: ${f}"
touch -r "${f}" _foo
sed -i "3i # (c)$(stat -c%y "${f}" | cut -d- -f1) John Doe <jdoe@example.net>" "${f}"
touch -r _foo "${f}" && ls -l "${f}"
fi
echo
done
rm _foo
Duplicate characters
Remove duplicate characters:[11]
$ echo 'aabc rüüf eef' | sed 's/\(.\)\1*/\1/g' abc rüf ef
For ASCII characters, we can also use a simpler tr version:[12]
$ echo 'aabc dde' | tr -s 'a-z' abc de
Line numbers
Remove (delete) lines 5 to 10, and line 12 from a file:[13]
sed -e '5,10d;12d' -i.bak file
Use 1 and $ to mark the beginning and the end of the file. Remove the first 5 lines, and also everthing after (and including) line 91:
sed -e '0,5d;91,$d' -i.bak file
Replace on match
$ echo -e "foo 123\nbar 123\nbaz 123" | sed '/^bar/s/2/ /' foo 123 bar 1 3 baz 123
Or, replace after match:
$ echo -e "foo 123\nbar 123\nbaz 123" | sed -E '/^bar/s/.*/&\nadd another line\n/' foo 123 bar 123 add another line baz 123
Replace only n times
Several answers exist for this riddle,[14] but this one I liked best:
$ echo {0..9} | wc -w
10
$ echo {0..9} | rev | sed 's|[[:alnum:]]|x-&|g9' | rev
0-x 1-x 2 3 4 5 6 7 8 9
Things get tricky of course if the substitution character consists of multiple chars, the rev(1) tool scrambles all that.
Remove first n lines
$ seq 0 5 > foo $ cat foo 0 1 2 3 4 5
$ sed '1,3d' foo 3 4 5
Alternative Character Keys
Not to be confused with the compose key:
ä - Alt-R + q ö - Alt-R + p ü - Alt-R + y ß - Alt-R + s © - Alt-R + c ® - Alt-R + r ™ - Alt-R + R
Markdown
The following applications are available for Fedora Linux and seem to be maintained as of 2023:
We can use markdownlint to make sure it's proper Markdown format:
gem install mdl
Update with:
$ gem update $(gem list -d | grep -B5 "${HOME}" | awk '/^[[:alnum:]]/ {print $1}')
Updating installed gems
Updating mdl
Fetching mdl-0.13.0.gem
Successfully installed mdl-0.13.0
[...]
Usage:
$ cat foo.md ## Title * bullet - point `` bla `` $ mdl foo.md foo.md:1: MD002 First header should be a top level header foo.md:3: MD004 Unordered list style foo.md:1: MD022 Headers should be surrounded by blank lines foo.md:2: MD032 Lists should be surrounded by blank lines foo.md:3: MD032 Lists should be surrounded by blank lines foo.md:6: MD047 File should end with a single newline character
lsof
The lsof(8) manpage is far too long to comprehend[15] and sometimes even unclear on its motives. Let's try to find out how to list open file on a single file system only:
+d s causes lsof to search for all open instances of directory s and the files and
directories it contains at its top level. +d does NOT descend the directory tree,
rooted at s. [...] Nor does it search for open files on file system mount points
on subdirectories of s unless the -x or -x f option is also specified.
-d s specifies a list of file descriptors (FDs) to exclude from or include in
the output listing.
+D D causes lsof to search for all open instances of directory D and all the files and
directories it contains to its complete depth. [...] Nor does it search for open
files on file system mount points on subdirectories of D unless the -x or -x f
option is also specified.
-x [fl] may accompany the +d and +D options to direct their processing to cross over
symbolic links and|or file system mount points encountered when scanning the
directory (+d) or directory tree (+D). If -x is specified by itself without a
following parameter, cross-over processing of both symbolic links and file system
mount points is enabled.
Let's try:
$ cat > /tmp/foo &
$ lsof -lnP /tmp/foo `which lsof`
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
cat 4882 0 1w REG 0,46 0 3470802 /tmp/foo # tmpfs
lsof 5868 0 txt REG 259,6 166512 536872858 /usr/bin/lsof # root filesystem
lsof 5869 0 txt REG 259,6 166512 536872858 /usr/bin/lsof
We should be able to see /tmp/foo with -x f +d, right?
$ lsof -lnP -xf +d / | grep /tmp/foo
Hm, no luck. With +D (along with -xf) this works:
$ lsof -lnP -xf +D / | grep /tmp/foo cat 4882 0 1w REG 0,46 0 3470802 /tmp/foo
...but it's a very expensive operation:
> Further note: lsof may process this option slowly and require a large amount of > dynamic memory to do it. This is because it must descend the entire directory > tree, rooted at D, calling stat(2) for each file and directory, building a > list of all the files it finds, and searching that list for a match with every > open file. When directory D is large, these steps can take a long time, so use > this option prudently.
Shell redirection
This chapter should be much bigger, as there are so many redirections to talk about.
Redirect everything (stdout and stderr) to a logfile:
exec 3>&1 1>/tmp/foo.log 2>&1
Another example, write to another file but calculate its checksum on the fly:
pv < file.img | tee copy.img | md5sum
Or, on the receiving side of a network transfer:
ncat -l 5001 | tee copy.img | md5sum
And, on the sending side of the same transfer, with a fifo:
mkfifo -m0600 foo
nc remote.example.net 5001 < foo &
pv < file.img | tee foo | md5sum
Compare stuff from stdout:
$ diff -u <(seq 1 3) <(seq 1 4) --- /dev/fd/63 2023-02-05 21:16:15.783985007 +0100 +++ /dev/fd/62 2023-02-05 21:16:15.774984921 +0100 @@ -1,3 +1,4 @@ 1 2 3 +4
Similarly, but with only one argument:
$ tac <<<$(seq 1 3) 3 2 1
dmesg timestamps
Human-readable timestamps in dmesg:
alias kmsg='dmesg | perl -ne "BEGIN{\$a= time()- qx!cat /proc/uptime!}; s/\[\s*(\d+)\.\d+\]/localtime(\$1 + \$a)/e; print \$_;"'
Let's see how this works:
$ dmesg | tail -1 [541285.101794] r8169 0000:05:00.0 eth0: link down $ kmsg | tail -1 Thu Mar 3 16:22:37 2016 r8169 0000:05:00.0 eth0: link down
Newer versions of dmesg[16] have the ability to show human readable (but not necessarily correct![17]) timestamps:
$ dmesg -T | tail -1 [Thu Mar 3 16:22:37 2016] r8169 0000:05:00.0 eth0: link down
Mail attachments
(echo Hello; uuencode file.jpg file.jpg) | mailx -s test user@example.org
With heirloom-mailx
echo Hello | heirloom-mailx -a file.jpg -s test user@example.org
Without[18] uuencode (we're using openssl instead) and without mailx -a:
recipient=user@example.org
subject="test"
body="Hello, this is a test"
file=foobar.jpg
mime=$(file --brief --mime-type ${file}) # Or use application/octet-stream
( cat <<EOF
To: ${recipient}
Subject: ${subject}
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary=BoundaryString
--BoundaryString
Content-Type: text/plain; charset=utf-8
${body}
--BoundaryString
Content-Type: ${mime}; name=${file}
Content-Transfer-Encoding: base64
EOF
openssl base64 -e -in ${file}) | /usr/sbin/sendmail -t -i
Creating bootable CDROMs
While various "recovery disks" and "live CDs" exist, it's sometimes necessary to create a fully customized disk. BIOS updates are a good example: as the Flash-utitliy often needs a DOS-like operating system, we'll need:
- A bootable DOS image
- A DOS image supporting an ATAPI CDROM
- The Flash-utility and the BIOS ROM file
The mkisofs(1) program helps us doing this:
mkdir -p iso/boot cd iso cp ~/DOS-1440.img boot/ cp ~/flash.exe ~/bios.rom . mkisofs -r -b boot/DOS-1440.img -c boot/boot.catalog -o bootcd.iso .
Now we can burn bootcd.iso and hopefully boot from it as well.
Rename network interfaces
Linux
ip link set eth1 down ip link set eth1 name eth123 ip link set eth123 up
This can be accomplished[19] via an udev rule too:
$ cat /etc/udev/rules.d/99-local.rules
ACTION=="add", SUBSYSTEM=="net", DRIVERS=="?*", ATTR{address}=="00:50:56:00:01:02", NAME="eth123"
Redirecting Websites
HTML
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="refresh" content="3; url='https://www.example.net'" />
</head>
<body>
<p>Redirecting to <a href="https://www.example.net">www.example.net</a>...</p>
</body>
</html>
PHP
<?php
header("Location: https://www.example.net/new/");
exit;
?>
Javascript
Fallback to HTML:
<!DOCTYPE HTML>
<html lang="en-US">
<head>
<meta charset="UTF-8">
<meta http-equiv="refresh" content="1;url=https://www.example.net/new/">
<script type="text/javascript">
window.location.href = "https://www.example.net/new/"
</script>
<title>Page Redirection</title>
</head>
<body>
If you are not redirected automatically, follow the <a href="https://www.example.net/new/">link to example</a>
</body>
</html>
Date conversions
Print 2 days from now:
$ date Fri Jun 5 21:40:00 PDT 2015 $ date -d '2 days ago' # GNU/date Wed Jun 3 21:40:00 PDT 2015 $ date -j -v +2d # BSD/date Sun Jun 7 21:40:00 PDT 2015
Print 2 days from an arbitrary date:
$ date -d "2015-06-05 21:40 PDT +2 days" # GNU/date Sun Jun 7 21:40:00 PDT 2015 $ date -j -v -2d -f "%Y-%m-%d %H:%M" "2015-06-05 21:40" "+%Y-%m-%d %H:%M %Z" # BSD/date 2015-06-03 21:40 PDT
Calculate days until a specific date:
$ echo $(( ($(date +%s) - $(date +%s --date "2015-06-05 12:00:00")) / (60*60*24) )) days 2044 days
Epoch conversion
With GNU or BSD date:
$ date +%s 1415209417
And back again:
$ date -d @1415209417 # GNU/date Wed Nov 5 09:43:37 PST 2014 $ date -r 1415209417 # BSD/date Wed Nov 5 09:43:37 PST 2014
If neither versions of date are available[20], we can use perl (which also honors the TZ environment variable[21]):
$ perl -le "print time"
1415209417
$ TZ=GMT perl -le 'print scalar(localtime(1415209417))'
Wed Nov 5 17:43:37 2014
And again, with awk, but only if strftime is implemented:
$ perl -le "print time" | TZ=GMT awk '{print strftime("%c", $1)}'
Wed Nov 5 17:43:37 2014
While on the topic, sometimes we have to go back in time:
$ date -d "3 days ago" Sun 31 May 18:30:20 EDT 2020
Even further back:[22]
$ date -d "43984 days ago" date: invalid date ‘43984 days ago’
Apparently GNU/date doesn't understand timestamps before January 1, 1970. But its UTC implemenation does:
$ date -u -d "43984 days ago" Sun 31 Dec 22:30:20 UTC 1899
Also, BSD/date understands this just fine, even in a local time zone:
$ date -d "43984 days ago" Sun Dec 31 17:30:20 EST 1899
Number conversion
From Hexadecimal
To decimal:
$ echo "ibase=16; A" | bc 10 $ printf "%d\n" 0xa # Without bc[23] 10
To octal:
$ echo "ibase=16; obase=8; A" | bc 12
To binary:
$ echo "ibase=16; obase=2; A" | bc 1010 $ printf "%..2d\n" 0xA # With ksh only 1010
To ASCII:
$ echo 0x68656c6c6f | xxd -rp hello
From Decimal
To hexadecimal:
$ echo "obase=16; 10" | bc A $ printf "%x\n" 10 # Works with bash, ksh, zsh, csh a
To octal:
$ echo "obase=8; 10" | bc 12 $ printf "%o\n" 10 12
To binary:
$ echo "obase=2; 10" | bc 1010 $ printf "%..2d\n" 10 # With ksh only 1010
From Binary
To hexadecimal:
$ echo "obase=16; ibase=2; 1010" | bc A $ printf '%x\n' $((2#1010)) # Works with bash, ksh, zsh; the printf is a builtin. a
To decimal:
$ echo "ibase=2; 1010" | bc 10 $ echo $((2#1010)) # Works with bash, ksh, zsh 10
IMAP & SMTP on the command line
For SMTP:
$ openssl s_client -starttls smtp -connect localhost:25 # Use nc localhost 25 for non-TLS connections 220 mail.example.com ESMTP Postfix (Debian/GNU) ehlo foobar.com 250-mail.example.com 250-PIPELINING 250-SIZE 250-VRFY 250-ETRN 250-STARTTLS 250-ENHANCEDSTATUSCODES 250-8BITMIME 250 DSN mail from: test@foobar.com 250 2.1.0 Ok rcpt to: test@example.com 250 2.1.5 Ok data 354 End data with <CR><LF>.<CR><LF> subject: this is a test foo bar baz . 250 2.0.0 Ok: queued as 44BD23DBEF quit 221 2.0.0 Bye
For IMAP, we have to specify command tags before the actual command:
$ openssl s_client -connect localhost:993 # Use nc localhost 143 for non-TLS connections * OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS AUTH=PLAIN] Dovecot ready. a01 login bob s3cr3tpa55w0rd a01 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE] Logged in a02 list "" "*" * LIST (\HasChildren) "." INBOX * LIST (\HasChildren) "." INBOX.spam * LIST (\HasNoChildren) "." INBOX.work [...] a03 EXAMINE INBOX * OK [CLOSED] Previous mailbox closed. * FLAGS (\Answered \Flagged \Deleted \Seen \Draft $Forwarded $label1 $MDNSent $label2 $label3 $label4 $label5 NonJunk $NotJunk $Junk) * OK [PERMANENTFLAGS ()] Read-only mailbox. * 3144 EXISTS * 0 RECENT * OK [UNSEEN 3142] First unseen. * OK [UIDVALIDITY 1190930016] UIDs valid * OK [UIDNEXT 44908] Predicted next UID * OK [HIGHESTMODSEQ 78664] Highest a03 OK [READ-ONLY] Examine completed (0.000 secs). a04 FETCH 3144 BODY[] * 3144 FETCH (BODY[] {3807} From test@example.com Thu May 7 09:12:26 2015 [...] Return-Path: <test@example.com> subject: this is a test Date: Thu, 7 May 2015 09:11:08 +0200 (CEST) From: test@example.com foo bar baz ) a04 OK Fetch completed. a05 logout * BYE Logging out a05 OK Logout completed.
Use the PHP built-in web server
As of PHP 5.4, a built-in web server[24] is available for testing purposes:
$ mkdir ~/www $ uname -a | cowsay | sed -e '1s/^/<pre>\n/' -e '$s/$/&\n<\/pre>/' | \ tidy -quiet -asxhtml -indent -utf8 > ~/www/index.html $ php -S 10.0.0.3:8080 -t ~/www/ PHP 5.6.7 Development Server started at Tue Apr 7 13:03:12 2015 Listening on http://10.0.0.3:8080 Document root is /home/dummy/www Press Ctrl-C to quit.
The built-in webserver does not support TLS, but we can use stunnel to emulate it. Again, this is for testing purposes only and not meant to be used in production!
Skip the first or last line
Because I always forget:
$ cat foo 1 2 3 4
Here's how to omit the first line of a file (or stdout):
$ tail -n +2 foo
2
3
4
And how to omit the last line:
$ head -n -1 foo
1
2
3
Compare directory trees
To compare (remote) directories w/o transfering them over the network, we print the checksum of each file, generate the checksum of that (sorted) output and compare that:
$ find . -type f | xargs openssl md5 -r | sort -n e7e04e99d58f244b784ae69b3663db8e *./c/file-c f2b105a7e0d17a453b9832e43590cc26 *./a/b/file-b 2ca3f8cfe4c59569a2c3bfc1d6ec0665 *./a/file-a 11bbccdd8883a732ffb9245ce50c9bfd *./file-0 $ find . -type f | xargs openssl md5 -r | sort -n | sha1sum ec0973f3e629d225f81c25b025a2cce4d3d70a11 -
Note: sorting is important here and may not even help (identical file with different file name and). Also, "./file" is different from "file".
Float comparison
While most shells cannot do float comparison, bc can:
$ echo '1.1 > 1.2' | bc 0 $ echo '1.3 > 1.2' | bc 1
Or even[25]:
$ bc <<< '1.5 > 1.9' 0
Network sockets in Bash
When bash is compiled with --enable-net-redirections[26], the following works:
$ exec 3<>/dev/tcp/example.org/80 $ echo -e "GET /hello.html HTTP/1.1\r\nhost: http://example.org\r\nConnection: close\r\n\r\n" >&3 $ cat <&3 <html>hello, world</html>
Internet speed test
For a lack of a better place, a few speedtest URLs:
https://cachefly.cachefly.net/100mb.test # Chicago, IL https://mirror.leaseweb.com/speedtest/1000mb.bin # LeaseWeb CDN http://speedtest.sjc01.softlayer.com/downloads/test100.zip # San Jose, CA http://speedtest.sea01.softlayer.com/downloads/test100.zip # Seattle, WA http://speedtest.dal01.softlayer.com/downloads/test100.zip # Dallas, TX http://speedtest.wdc01.softlayer.com/downloads/test500.zip # Chantilly, VA
Keyboard layout
In an X11 session:
$ setxkbmap us # See /usr/share/X11/xkb/symbols/
On a console session:
$ loadkeys de # See /usr/share/{,kbd/}keymaps/ or /lib/kbd/keymaps/
Loading /usr/share/kbd/keymaps/i386/qwertz/de.map.gz
assuming iso-8859-15 euro
Note: loadkeys needs the console-data package in Debian/Ubuntu. In SUSE/RedHat, the keymaps are included in the kbd package.
Absolute values in bc
$ echo "val = 5-3; if( val < 0 ) val*= -1; val" | bc 2
$ echo "val = 5-30; if( val < 0 ) val*= -1; val" | bc 25 # abs(-25)
Backup SQLite
From SQLite Backup and Restore:
sqlite3 database.db .dump > database.backup
Restore with:
sqlite3 database.db < database.backup
Remove characters in bash
$ A=hello $ echo $A hello
Remove characters from back of a string:
$ echo ${A%%o}
hell
Remove characters from front of a string:
$ echo ${A##h}
ello
Print characters repeatedly
If seq(1) is available:
$ seq 10 | while read a; do printf "x"; done; echo
xxxxxxxxxx
In Bash, maybe ZSH too:
for a in {0..9}; do printf x; done; echo
Bash-4.1 supports printf -v:
printf -v a "%10s"; printf "%s\n" "${a// /x}"
In Perl:
perl -e 'print "a" x 10, "\n"'
Zero Padding in Bash
$ x=8; while [ $x -lt 12 ]; do echo $x; x=$((x+1)); done 8 9 10 11
To have this zero-padded (in lieu of seq(1)), we could use printf:
$ x=8; while [ $x -lt 12 ]; do printf "%02d\n" $x; x=$((x+1)); done
08
09
10
11
The "2" denotes the digits of the number to pad. I.e. to pad numbers up to 100, one would use "printf "%03d".
Endianness
How to tell if a system is big endian or little endian? For Unix/Linux
$ printf I | od -t o2 | head -1 | cut -d" " -f2 | cut -c6 0 ← Big Endian (mips, parisc, power, powerpc, s390x, sparc) 1 ← Little Endian (alpha, arm, ia64, mipsel, vax, x86, x86-64)
According to the author, this would not work for AIX (and MacOS X) and proposed a solution with awk(1):
echo I | tr -d [:space:] | od -t o2 | head -1 | awk '{print $2}' | cut -c6
The awk(1) version can then be "simplified" to:
printf I | od -t o2 | awk '{print substr($2,6,1); exit}'
And another one, w/o od(1):
printf I | hexdump -o | awk '{print substr($2,6,1); exit}'
dnsdomainname / FQDN
dnsdomainname won't return the domain name?
$ dnsdomainname $ hostname --fqdn alice
Add the FQDN to /etc/hosts:
$ grep `hostname` /etc/hosts
10.0.0.3 alice.example.org alice
$ dnsdomainname example.org $ hostname --fqdn alice.example.org
combination of fields in crontab(5)
Debian #460070 shows it quite clearly:
$ crontab -l #m h dom mon dow command 21 11 5-15 * Thu echo test1 21 11 5-15 * Fri echo test2
Yet both commands were executed, although it was not a Thursday or a Friday. But crontab(5) specifies:
> Finally, if either the month or day of month is specified as an element or list, > and the day of week is also specified as an element or list, then any day matching > either the month and day of month or the day of week, will be matched.
The aforementioned bugreport even has a workaround for this:
#m h dom mon dow command 21 11 5-15 * * [ "`date +%w`" -eq 4 ] && echo test1 21 11 5-15 * * [ "`date +%w`" -eq 5 ] && echo test2
python setup.py
To build & install a Python program:
python setup.py build sudo python setup.py install --prefix=/opt/foo
However, for the new program to work, Python's search path may have to be modified:
$ /opt/foo/bin/foo Traceback (most recent call last): File "/opt/foo/bin/foo", line 6, in <module> import bar ImportError: No module named bar $ PYTHONPATH=/opt/foo/lib/python2.6/site-packages /opt/foo/bin/foo -v 0.1
Python pip
List locally installed packages, i.e. omitting packages installed from the distribution:
$ pip list --local Package Version ------------------ --------- arrow 1.2.3 b2 3.7.1 b2sdk 1.19.0
Updating all locally installed packages can be done manually:
for p in $(pip list --format freeze --local | awk -F'==' '{print $1}'); do pip install --upgrade ${p}; done
...or via pip-review:
pip-review --local --interactive
CR+LF
Applications and operating systems are implementing newlines differently:
LF - Unix, MacOS 10 CR+LF - Windows, DOS LR+CR - RISC OS CR - MacOS 8,9
Here's how to convert between them.
From LF to CR+LF, on Windows:
TYPE unix_file | FIND "" /V > dos_file
On Unix and Linux systems, there's unix2dos
dos2unix dos_file # This will modify dos_file and its original content will be lost! dos2unix -n dos_file unix_file # This will output to unix_file
And vice versa:
unix2dos -n unix_file dos_file
Newer versions of the same program are called tofrodos:
todos unix_file # This will modify unix_file and its original content will be lost! todos -b unix_file # This will leave a backup of the original file in unix_file.bak
And vice versa:
$ fromdos -b dos_file
$ file dos_file{,.bak}
dos_file: ASCII text
dos_file.bak: ASCII text, with CRLF line terminators
The same, with tr(1):
tr -d '\r' < dos_file > unix_file # CR+LF → LF
With sed(1):
sed -e 's/$/\r/' unix_file > dos_file # Unix to DOS (LF → CR+LF) sed -e 's/\r$//' dos_file > unix_file # DOS to UNIX (CR+LF → LF)
With awk(1):
awk 'sub("$", "\r")' unix_file > dos_file # Unix to DOS
awk '{ sub("\r$", ""); print }' dos_file > unix_file # DOS to Unix
With perl(1):
perl -p -e 's/\n/\r\n/' < unix_file > dos_file # Unix to DOS perl -p -e 's/\r$//' < dos_file > unix_file # DOS to Unix
With vi(1):
:1,$s/^M//g # DOS to Unix
For the sake of completeness, remove an entire linebreak[27]
sed ':a;N;$!ba;s/\n/ /g'
Let's try this:
$ printf 'a:\nb\n' a: b $ printf 'a:\nb\n' | sed ':a;N;$!ba;s/:\n/: /g' a: b
Patch strings in binary files with sed
Let's assume a rather simple file, with binary and ASCII parts mixed:
$ openssl rand 10 > file; echo hello >> file; openssl rand 10 >> file
$ cat file
�u�(8V���hello
��O�Id�y��
How to modify that ASCII part of that file?
Let's convert our target string into hex:
$ printf hello | xxd -p 68656c6c6f
Which is indeed part of our file:
$ xxd -p file
c2758b28381656c7fbe068656c6c6f0a848d4f8f49648179bea3
We can replace that string with another string of equal length:
$ printf solar | xxd -p
736f6c6172
Let's replace occurences of hello with solar in our file:
$ xxd -p file | sed 's/68656c6c6f/736f6c6172/g' | xxd -r -p > file_2 $ cat file_2 �u�(8V���solar ��O�Id�y��
- Random notes about using Linux (2012-11-04)
Remove binary data
Remove binary data from a file, with tr:
tr -cd '\11\12\15\40-\176' < file.bin > file.txt
This will preserve:
118: TAB 128: LF 158: CR 408 to 1768: printable characters[28]
With Perl:
perl -pe 's/[^[:ascii:]]//g' file.bin > file.txt
With (XPG4) awk:
awk '{gsub(/[[:cntrl:]]/,x)}1' file.bin > file.txt
.profile
When bash(1) is an interactive login shell, it reads:
/etc/profile, ~/.bash_profile, ~/.bash_login, ~/.profile
When started as an interactive non-login shell:
/etc/bash.bashrc and ~/.bashrc
For zsh(1):
/etc/profile, $HOME/.profile # /etc/zsh/zshrc, $ZDOTDIR/.zshrc # Interactive non-login
To provide a logout script to shells that don't offer one, add the following to a login script:
trap '$HOME/.logout' 0
The .logout file should exist and needs to be executable too.
cpio
List:
cpio -tv < foo.cpio # List, verbose
Extract:
cpio -ivd < foo.cpio # Extract, verbose, create directories
Extract just one file:
cpio -ivd dir/file.ext < foo.cpio
Create:
find . -depth -print | cpio -o > ../foo.cpio # Create
Copy:
find . -depth -print | cpio -pdumv ../new_directory
List contents of an RPM package:
rpm2cpio file.rpm | cpio -tv
Extract (but don't install):
rpm2cpio file.rpm | cpio -ivd
Rescan SCSI/FC bus
$ ls -1d /sys/bus/scsi/devices/target* /sys/bus/scsi/devices/target0:0:0 /sys/bus/scsi/devices/target0:0:1 /sys/bus/scsi/devices/target0:0:2 /sys/bus/scsi/devices/target2:0:0 $ echo "- - -" > /sys/class/scsi_host/host0/scan # SCSI $ echo 1 > /sys/class/fc_host/host0/issue_lip # FC
$ dmesg scsi 0:0:2:0: Direct-Access VMware Virtual disk 1.0 PQ: 0 ANSI: 2 sd 0:0:2:0: [sdc] 20971520 512-byte logical blocks: (10.7 GB/10.0 GiB) sd 0:0:2:0: [sdc] Write Protect is off sd 0:0:2:0: [sdc] Mode Sense: 03 00 00 00 sd 0:0:2:0: [sdc] Cache data unavailable sd 0:0:2:0: [sdc] Assuming drive cache: write through sd 0:0:2:0: [sdc] Cache data unavailable sd 0:0:2:0: [sdc] Assuming drive cache: write through sdc: unknown partition table sd 0:0:2:0: [sdc] Cache data unavailable sd 0:0:2:0: [sdc] Assuming drive cache: write through sd 0:0:2:0: [sdc] Attached SCSI disk
To add a single device (HBTL: host, bus, target, lun) to a SCSI host:
$ echo "scsi add-single-device <H> <B> <T> <L>" > /proc/scsi/scsi
And, while we're at it -- remove a disk from the system:
$ grep . /sys/devices/platform/soc/*/*/*/*/*/host*/target*/*/{vendor,model}
/sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.2/1-1.2:1.0/host0/target0:0:0/0:0:0:0/vendor:ASMT
/sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.2/1-1.2:1.0/host0/target0:0:0/0:0:0:0/model:2115
$ echo 1 > /sys/devices/platform/soc/3f980000.usb/usb1/1-1/1-1.2/1-1.2:1.0/host0/target0:0:0/0:0:0:0/delete
$ dmesg -t
usb 1-1.2: reset high-speed USB device number 6 using dwc_otg
sd 0:0:0:0: [sda] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x07 driverbyte=0x00 cmd_age=0s
sd 0:0:0:0: [sda] tag#0 CDB: opcode=0x1b 1b 00 00 00 02 00
sd 0:0:0:0: [sda] Synchronizing SCSI cache
Power off a USB disk:[29]
udisksctl power-off -b /dev/sdx
tar without resource forks
Setting COPYFILE_DISABLE will omit resource forks from tar-archives:
COPYFILE_DISABLE=1 tar -cvf foo.tar example/
Before MacOS 10.5, COPY_EXTENDED_ATTRIBUTES_DISABLE had to be set. While not documented, it seems as Apple patched gnutar w/o mentioning the variable in the man page.
PAGE_SIZE
The page size can be found out with the getpagesize syscall:
$ getconf PAGE_SIZE 4096 # Solaris 10/i386 8192 # Solaris 10/sun4u 4096 # MacOS 10.7.5/x68-64 4096 # Linux 3.8/powerpc 4096 # Linux 2.6/x86-64
parted: The resulting partition is not properly aligned for best performance
GNU/parted might comlain about inproper alignment:
(parted) mkpart primary ext2 0 -1
Warning: The resulting partition is not properly aligned for best performance.
Ignore/Cancel? c
Let's begin the partition at sector 1, rather than 0:
(parted) mkpart primary ext2 1 -1
(parted) p
Model: VMware Virtual disk (scsi)
Disk /dev/sdb: 10.7GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number Start End Size Type File system Flags
1 1049kB 10.7GB 10.7GB primary
Note: the primary keyword only needs to be given for MSDOS partitions and can be omitted for GPT partition tables.
base64
Via perl:
perl -MMIME::Base64 -ne 'print decode_base64($_)' < file.txt perl -MMIME::Base64 -0777 -ne 'print encode_base64($_)' < file.txt # 0777 will force Perl to read the whole file in one shot.[30]
On MacOS X we also have /usr/bin/base64:
base64 -D -i file.txt -o file.out
Via OpenSSL:
$ echo foo | openssl base64 -e Zm9vCg== $ echo Zm9vCg== | openssl base64 -d foo
iPod import
Ever wanted to import from your iPod to your computer? According to the FAQ, this is not a common usecase:
Synchronization generally occurs only in one direction, from your computer to your iPod. This means you typically cannot transfer music, automatically or manually, from your iPod to a computer, and you cannot use iPod to copy a music library from one computer to another.
To get around this, several tools exist:
- iRip
- CopyTrans
- Q: I have a computer at home and one at work. Can I update my iPod music collection from more than one computer?
ImageMagick for dummies
Sometimes we just want to convert a bunch of files, quickly.
convert file.jpg -resize 50% file.png # For ImageMagick gm convert file.jpg -resize 50% file.png # For GraphicsMagick
convert file.jpg -scale 50% file-scaled.jpg # For ImageMagick gm convert file.jpg -scale 50% file-scaled.jpg # For GraphicsMagick
convert file.jpg -rotate -90 file-left.jpg # For ImageMagick gm convert file.jpg -rotate -90 file-left.jpg # For GraphicsMagick
convert file.jpg -geometry '100x100' file-quadrant.jpg gm convert file.jpg -geometry '100x100!' file-quadrant.jpg # w/o maintaining the aspect ratio
rsync to vfat
Apparently the resolution time of Windows FAT filesystems is two seconds[31] (for writes). We'll use --modify-window=2[32] to not copy already transferred files again:
rsync -rvP --modify-window=2 src/ bob:/mnt/usb/
Unable to Log You on Because of an Account Restriction
When trying to login via RDP, this message appears:
Unable to log you on because of an account restriction.
This happens when there's no password set for the account you're trying to login with. Either set a password (recommended!) or (temporarily) disable this policy:
- Run
gpedit.msc(Group Policy Editor) - Open
Computer Configuration\Windows Settings\Security Settings\Local Policies\Security Options - Double Click on
Accounts: Limit local account use of blank passwords to console logon only - Select
Disabled - Quit
Group Policy Editor
history timestamps in zsh
$ setopt extendedhistory
$ history -i | tail -1
24 2011-04-01 02:41 df
25 2011-04-01 02:41 ls
26 2011-04-01 02:42 du
27 2011-04-01 02:42 top
28 2011-04-01 02:42 dmesg
Note: While the timestamps in HISTFILE are exact to the second, the output of history -i is not :-\
history merging in bash
Read the contents of the old and the current history file and append them to the current (in-memory) history list:
history -r .bash_history.old history -r .bash_history
Write the current history list to the history file, overwriting the history file's contents.
history -w
Inode count in ZFS
Well, let's say file count, for this example:
$ time zdb -dv rpool | grep -Ec 'ZFS (directory|plain file)' 167762 real 0m49.146s user 0m26.281s sys 0m5.348s
Create Windows user accounts
How to create local user accounts in Windows NT:
@echo off
for /F "tokens=1,2,3 delims=," %%a IN (users.txt) DO \
net user %%a Passw0rd /add /fullname:"%%b" /comment:"%%c" /expires:never
MySQL: large BLOB inserts
According to #45236, adding max_allowed_packet helps:
bzip2 -dc sql-backup-with-blobs.sql.bz2 | mysql --max_allowed_packet=128M -D dbname
Fun with subtitles
I had a subtitle file for a movie, which, at one point later in the movie went off by 1 minute or so. Rather than editing the timestamps I found a second subtitle file - which was wrong at the beginning of the movie but got it right at least from the point the first file wandered off. So here's what I did:
$ tr -d '\r' < file-1.srt > file-1.copy $ tr -d '\r' < file-2.srt > file-2.copy $ head -3 file-2.copy 770 01:22:29,860 --> 01:22:32,900 The first was Esteban Vihaio. # somehow we cannot use ^[0-9]*$ $ sed -e 's/^[0-9][0-9][0-9]$/XXX/;s/^[0-9][0-9][0-9][0-9]$/XXX/' -i file-2.copy $ for i in `seq 770 1132`; do echo $i; sed "0,/^XXX$/s//"$i"/" -i file-2.copy; done # now we can merge both files $ cat file-1.copy file-2.copy > file.srt
Yeah, due to its ugliness I did not want to blog about it :-)
Clickable URLs in Terminal.app
Until before Terminal.app came without clickable URL support, one had to use ICeCoffEE to get this working. However, Terminal.app supports this already:
- Move mouspointer over URL, you don't have to mark it
- Press right mouse-button & click "Open URL"
./configure
Just checked out a source tree but the ./configure file is missing? Try, in that order:
autoreconf --install
If that did not work, maybe this will:
libtoolize --copy --install --force cp include/install-sh . # If needed aclocal # Try adding -I config or -I m4 if needed. This should generate "aclocal.m4" autoheader # If needed
The next step will try to create ./configure:
autoconf
The next step will try to create ./Makefile.in:
automake # Try adding --add-missing if needed
With both ./configure and ./Makefile.in in place, running ./configure will (hopefully) generate a Makefile.
Esound
esd -tcp -public -noterminate -port 5001 -b # server
esddsp -s hamma:5001 mpg321 -o esd foo.mp3 # client esddsp -s hamma:5001 ogg123 -d esd foo.ogg esddsp -s hamma:5001 mplayer -ao esd foo.mp[34]
How to install IRIX packages
$ ls fw_pine fw_pine.idb fw_pine.sw $ /usr/sbin/showfiles -f fw_pine -x -- pine
It might complain about missing .man and .src files - ignore them if you don't need them.
tar/ssh
- Get remote directory /data/pr0n and create pr0n.tar.gz on the local host:
ssh remote "cd /data; tar -cf - pr0n" | (gzip -c > /u3/pr0n.tar.gz )
- Get remote directory /data/pr0n and extract underneath /u3 on the local machine. Note how we "cd /data" on the remote host first so that we DON'T create an archive with absolute pathnames:
ssh remote "cd /data; tar -cf - pr0n" | (cd /u3; tar -xf - )
- Grab local /u3/foo and create /bar/foo.tar on the remote host:
(cd /u3; tar -cf - foo) | ssh remote "cat > /bar/foo.tar"
- Grab local /u3/foo and extract it underneath /bar on the remote host:
(cd /u3; tar -cf - foo) | ssh remote "cd /bar; tar -xpf -"
VMware/ethernet0.addressType
Want to have static IP addresses for all your vmware-machines? Change ethernet0.addressType in the .vmx file and put the same IP address in every guest (GNU/sed needed for '-i' to work):
sed -i 's/ethernet0.addressType = "generated"/ethernet0.addressType = "static"/' *.vmx sed -i 's/ethernet0.generatedAddress = ".*"/ethernet0.address = "00:50:56:38:36:8a"/' *.vmx sed -i '/^ethernet0.generatedAddressOffset = /d' *.vmx
See this thread about choosing a valid MAC address. We're limited to addresses starting with 00:50:56.
SO_BSDCOMPAT
Want to get rid of the annoying process ... is using obsolete setsockopt SO_BSDCOMPAT warnings? When compiling
ISC BIND, apply this patch to disable this (stale) option:
--- bind-9.4.0b2/lib/isc/unix/socket.c.ORIG 2006-10-05 15:52:38.575470725 +0100
+++ bind-9.4.0b2/lib/isc/unix/socket.c 2006-10-05 15:53:38.935470726 +0100
@@ -245,6 +245,8 @@ static void build_msghdr_send(isc_socket
struct msghdr *, struct iovec *, size_t *);
static void build_msghdr_recv(isc_socket_t *, isc_socketevent_t *,
struct msghdr *, struct iovec *, size_t *);
+#undef SO_BSDCOMPAT
+
#define SELECT_POKE_SHUTDOWN (-1)
#define SELECT_POKE_NOTHING (-2)
.inputrc
Is your DELETE key not working? END key generating strange characters? Try to create an ~/.inputrc with the following content:
set input-meta on set output-meta on set convert-meta off "\e[1~": beginning-of-line "\e[4~": end-of-line "\e[3~": delete-char "\e[2~": quoted-insert
unpacking .deb files
Normally, one would unpack .deb files with:
$ dpkg-source -x ../file.deb . dpkg-source: error: syntax error in source control file ../file.deb at line 1: \ line with unknown format (not field-colon-value)
Whoops, didn't work. Well, it's an Ubuntu .deb and my dpkg-source is from an aging Debian installation. So, let's try something else:
$ ar vx ../file.deb x - debian-binary x - control.tar.gz x - data.tar.gz $ tar -xzf data.tar.gz
Or, in one step:
$ ar p ../file.deb data.tar.gz | tar -xz
dd under Windows
dd if=\\?\Device\CdRom0 of=c:\temp\disc1.iso bs=1M dd if=\\.\a: of=c:\temp\disk1.img bs=1440k
rdiff-backup
rdiff-backup is a fairly flexible, feature-rich and good-documented backup-utility. Unfortunately it is written in a script-language (python) and so it is prone to occasional br0keness due to library changes and what not. So I really had to compile it by myself because the netbsd-stable version was too old. I installed it in /usr/local, the version-error was gone, but now I got this:
# rdiff-backup
Traceback (most recent call last):
File "/usr/local/bin/rdiff-backup", line 20, in ?
import rdiff_backup.Main
ImportError: No module named rdiff_backup.Main
I was told to fix this by setting PYTHONPATH to the path, where your python site-packages are installed. And it worked:
# export PYTHONPATH=/usr/local/lib/python2.3/site-packages # rdiff-backup -V rdiff-backup 1.1.5
Screen
# Start a new screen session, give it a name: screen -US foo # Detach session: CTRL-a, d # List active sessions: screen -ls # Resume session "foo": screen -r foo # Resume my only session, possibly disconnecting remotely attached terminals: screen -dr # Create new windows within a session, list them: CTRL-a, c CTRL-a, " # Split, switch, unsplit windows CTRL-a S CTRL-a <TAB> CTRL-a Q
While ^e to jump to the end of the line might still work, ^a is now taken. Try ^aa to get to the beginning of the line.
Beep!
Disable the beep! in X11 terminals:
xset b 0
Links
- Writing Safe Shell Scripts
- Tips and Resources for writing better Shell Scripts
- Google Shell Style Guide
References
- ↑ Loading Microcode from the OS
- ↑ How can I tell if my system was booted as EFI/UEFI or BIOS?
- ↑ VLC Chromecast casting via command line?
- ↑ How to know whether Wayland or X11 is being used
- ↑ Big list of http static server one-liners
- ↑ SimpleHTTPServer
- ↑ Using rsync to delete a single file
- ↑ Documentation/networking/fib_trie.txt
- ↑ qemu-system-x86_64(1)
- ↑ Insert a line at specific line number with sed or awk
- ↑ How to remove duplicate characters?
- ↑ How to remove duplicate characters?
- ↑ Delete specific line number(s) from a text file using sed?
- ↑ Sed -- Replace first k instances of a word in the file
- ↑ lsof - restrict output to physical files only - how?
- ↑ dmesg: add --ctime to print human readable timestamps
- ↑ dmesg -T show different time for the same lines
- ↑ Mailing attachments. No UUencode; no mailx -a or mutt
- ↑ CentOS 7 - Rename network interface without rebooting
- ↑ Mit Windows waer das nicht passiert
- ↑ The GNU C Library: 21.4.7 Specifying the Time Zone with TZ
- ↑ Differences between the 1900 and the 1904 date system in Excel
- ↑ printf(3)
- ↑ Built-in web server
- ↑ How to do float comparison in Bash?
- ↑ Bash Reference Manual: Redirections
- ↑ How can I replace a newline (\n) using sed?
- ↑ ASCII#Printable_characters
- ↑ Turn off (spin down) external (USB) HDD on Linux
- ↑ The Command-line Options (Practical Perl Programming, A. D. Marshall 1999-2005)
- ↑ Windows System Information: File Times
- ↑ rsync(1)