Braindumps

From Segfault
Jump to: navigation, search

Contents

sed

We should probably open up a whole article for sed(1) examples.

I needed to add a copyright note[1] to some scripts, but since it's only a cosmetic update, I did not want to alter the timestamp of each script. We will do that anyway, but restore it later on:

ls *.sh | while read f; do
   if ! grep -qi '(c)' $f; then
      echo "FILE: $f"
      touch -r "$f" _foo
      sed -i "3i # (c)$(stat -c%y "$f" | cut -d- -f1) John Doe <jdoe@example.net>" "$f"
      touch -r _foo "$f" && ls -l "$f"
   fi
   echo
done
rm _foo

lsof

The lsof(8) manpage is far too long to comprehend[2] and sometimes even unclear on its motives. Let's try to find out how to list open file on a single file system only:

+d s    causes  lsof to search for all open instances of directory s and the files and directories it contains at its top level.
        +d does NOT descend the directory tree, rooted at s. [...] Nor does it search for open files on file system mount points
        on subdirectories of s unless the -x or -x  f option is also specified.

-d s    specifies  a  list  of  file  descriptors  (FDs) to exclude from or include in the output listing.

+D D    causes lsof to search for all open instances of directory D and all the files and directories it contains to its complete depth.
        [...] Nor does it search for open files on file system mount points on subdirectories of D unless the -x or -x  f option is also
        specified.

-x [fl] may accompany the +d and +D options to direct their processing to cross over symbolic links and|or file system mount points
        encountered when scanning the directory (+d) or directory tree (+D).
        If -x is specified by itself without a following parameter, cross-over processing of both symbolic links and file system mount
        points is enabled.

Let's try:

$ cat > /tmp/foo &
$ lsof -lnP /tmp/foo `which lsof`
COMMAND  PID     USER   FD   TYPE DEVICE SIZE/OFF      NODE NAME
cat     4882        0    1w   REG   0,46        0   3470802 /tmp/foo                             # tmpfs
lsof    5868        0  txt    REG  259,6   166512 536872858 /usr/bin/lsof                        # root filesystem
lsof    5869        0  txt    REG  259,6   166512 536872858 /usr/bin/lsof

We should be able to see /tmp/foo with -x f +d, right?

$ lsof -lnP -xf +d / | grep /tmp/foo

Hm, no luck. With +D (along with -xf) this works:

$ lsof -lnP -xf +D / | grep /tmp/foo
cat     4882        0    1w   REG   0,46        0   3470802 /tmp/foo

...but it's a very expensive operation:

> Further note: lsof may process this option slowly and require a large amount of dynamic memory to do it.  This is because it must descend
> the entire directory tree, rooted at D, calling stat(2) for each file and directory, building a list of all the files it finds, and
> searching that list for a match with every open file.  When directory D is large, these steps can take a long time, so use this option prudently.

Shell redirection

This chapter should be much bigger, as there are so many redirections to talk about.

Redirect everything (stdout and stderr) to a logfile:

exec 3>&1 1>/tmp/foo.log 2>&1

dmesg timestamps

Human-readable timestamps in dmesg:

alias kmsg='dmesg | perl -ne "BEGIN{\$a= time()- qx!cat /proc/uptime!}; s/\[\s*(\d+)\.\d+\]/localtime(\$1 + \$a)/e; print \$_;"'

Let's see how this works:

$ dmesg | tail -1
[541285.101794] r8169 0000:05:00.0 eth0: link down

$ kmsg  | tail -1
Thu Mar  3 16:22:37 2016 r8169 0000:05:00.0 eth0: link down

Newer versions of dmesg[3] have the ability to show human readable (but not necessarily correct![4]) timestamps:

$ dmesg -T | tail -1
[Thu Mar  3 16:22:37 2016] r8169 0000:05:00.0 eth0: link down

Mail attachments

With uuencode and bsd-mailx:

(echo Hello; uuencode file.jpg file.jpg) | mailx -s test user@example.org

With heirloom-mailx

echo Hello | heirloom-mailx -a file.jpg -s test user@example.org

Without[5] uuencode (we're using openssl instead) and without mailx -a:

recipient=user@example.org
subject="test"
body="Hello, this is a test"
file=foobar.jpg
mime=`file --brief --mime-type foobar.jpg`                                     # Or use application/octet-stream

( echo "To: $recipient"
 echo "Subject: test"
 echo "MIME-Version: 1.0"
 echo "Content-Type: multipart/mixed; boundary=BoundaryString"
 echo
 echo "--BoundaryString"
 echo "Content-Type: text/plain; charset=utf-8"
 echo "$body"
 echo "--BoundaryString"
 echo "Content-Type: $mime; name=$file"
 echo "Content-Transfer-Encoding: base64"
 echo
 openssl base64 -e -in $file ) | /usr/sbin/sendmail -t -i

Creating bootable CDROMs

While various "recovery disks" and "live CDs" exist, it's sometimes necessary to create a fully customized disk. BIOS updates are a good example: as the Flash-utitliy often needs a DOS-like operating system, we'll need:

  • A bootable DOS image
  • A DOS image supporting an ATAPI CDROM
  • The Flash-utility and the BIOS ROM file

The mkisofs(1) program helps us doing this:

mkdir -p iso/boot
cd iso
cp ~/DOS-1440.img boot/
cp ~/flash.exe ~/bios.rom .
mkisofs -r -b boot/DOS-1440.img -c boot/boot.catalog -o bootcd.iso .

Now we can burn bootcd.iso and hopefully boot from it as well.

Boot disks:

Rename network interfaces

Linux

ip link set eth1 down
ip link set eth1 name eth123
ip link set eth123 up

This can be accomplished[6] via an udev rule too:

$ cat /etc/udev/rules.d/99-local.rules
ACTION=="add", SUBSYSTEM=="net", DRIVERS=="?*", ATTR{address}=="00:50:56:00:01:02", NAME="eth123"

Redirecting Websites

HTML

<?xml version="1.0"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
  <meta http-equiv="refresh" content="0; url=https://www.example.net/new/" />
<title></title>
</head>
<body>
  Redirecting...
</body>
</html>

PHP

<?php
  header("Location: https://www.example.net/new/");
  exit;
?>

Javascript

Fallback to HTML:

<!DOCTYPE HTML>
<html lang="en-US">
  <head>
    <meta charset="UTF-8">
    <meta http-equiv="refresh" content="1;url=https://www.example.net/new/">
      <script type="text/javascript">
        window.location.href = "https://www.example.net/new/"
      </script>
      <title>Page Redirection</title>
  </head>
  <body>
    If you are not redirected automatically, follow the <a href="https://www.example.net/new/">link to example</a>
  </body>
</html>

Kernel modules

Which package(s) do the used kernel modules belong to?

$ for m in `lsmod | awk '!/^Module/ {print $1}'`; do find /lib/modules/ -type f -name "${m}.ko"; done | xargs dpkg -S
linux-image-3.19.0-21-generic: /lib/modules/3.19.0-21-generic/kernel/crypto/xts.ko
linux-image-extra-3.19.0-21-generic: /lib/modules/3.19.0-21-generic/kernel/drivers/gpu/drm/cirrus/cirrus.ko
linux-image-3.19.0-21-generic: /lib/modules/3.19.0-21-generic/kernel/drivers/video/fbdev/core/syscopyarea.ko
linux-image-3.19.0-21-generic: /lib/modules/3.19.0-21-generic/kernel/drivers/video/fbdev/core/sysfillrect.ko
linux-image-3.19.0-21-generic: /lib/modules/3.19.0-21-generic/kernel/drivers/video/fbdev/core/sysimgblt.ko
linux-image-extra-3.19.0-21-generic: /lib/modules/3.19.0-21-generic/kernel/drivers/gpu/drm/ttm/ttm.ko
[...]

Are any of the used kernel modules in this particular package?

$ for m in `lsmod | awk '!/^Module/ {print $1}'`; do 
     dpkg -L linux-image-extra-3.19.0-21-generic | awk '/.ko/' | grep "/${m}.ko"
done
/lib/modules/3.19.0-21-generic/kernel/drivers/gpu/drm/cirrus/cirrus.ko
/lib/modules/3.19.0-21-generic/kernel/drivers/gpu/drm/ttm/ttm.ko
/lib/modules/3.19.0-21-generic/kernel/drivers/gpu/drm/drm_kms_helper.ko
/lib/modules/3.19.0-21-generic/kernel/drivers/gpu/drm/drm.ko
/lib/modules/3.19.0-21-generic/kernel/drivers/ata/pata_acpi.ko

Date conversions

Print 2 days from now:

$ date
Fri Jun  5 21:40:00 PDT 2015

$ date -d '2 days ago'                                                        # GNU/date
Wed Jun  3 21:40:00 PDT 2015

$ date -j -v +2d                                                              # BSD/date
Sun Jun  7 21:40:00 PDT 2015

Print 2 days from an arbitrary date:

$ date -d "2015-06-05 21:40 PDT +2 days"                                      # GNU/date
Sun Jun  7 21:40:00 PDT 2015

$ date -j -v -2d -f "%Y-%m-%d %H:%M" "2015-06-05 21:40" "+%Y-%m-%d %H:%M %Z"  # BSD/date
2015-06-03 21:40 PDT

Number conversion

From Hexadecimal

To decimal:

$ echo "ibase=16; A" | bc
10

$ printf "%d\n" 0xa                                # Without bc[7]
10

To octal:

$ echo "ibase=16; obase=8; A" | bc
12

To binary:

$ echo "ibase=16; obase=2; A" | bc
1010

$ printf "%..2d\n" 0xA                             # With ksh only
1010

To ASCII:

$ echo 0x68656c6c6f | xxd -rp
hello

From Decimal

To hexadecimal:

$ echo "obase=16; 10" | bc
A

$ printf "%x\n" 10                                 # Works with bash, ksh, zsh, csh
a

To octal:

$ echo "obase=8; 10" | bc
12

$ printf "%o\n" 10
12

To binary:

$ echo "obase=2; 10" | bc
1010

$ printf "%..2d\n" 10                              # With ksh only
1010

From Binary

To hexadecimal:

$ echo "obase=16; ibase=2; 1010" | bc
A

$ printf '%x\n' $((2#1010))                        # Works with bash, ksh, zsh; the printf is a builtin.
a

To decimal:

$ echo "ibase=2; 1010" | bc
10

$ echo $((2#1010))                                 # Works with bash, ksh, zsh
10

IMAP & SMTP on the command line

For SMTP:

$ openssl s_client -starttls smtp -connect localhost:25                # Use nc localhost 25 for non-TLS connections
220 mail.example.com ESMTP Postfix (Debian/GNU)
ehlo foobar.com
250-mail.example.com
250-PIPELINING
250-SIZE
250-VRFY
250-ETRN
250-STARTTLS
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN

mail from: test@foobar.com
250 2.1.0 Ok

rcpt to: test@example.com
250 2.1.5 Ok

data
354 End data with <CR><LF>.<CR><LF>
subject: this is a test
foo bar baz
.
250 2.0.0 Ok: queued as 44BD23DBEF

quit
221 2.0.0 Bye

For IMAP, we have to specify command tags before the actual command:

$ openssl s_client -connect localhost:993                              # Use nc localhost 143 for non-TLS connections
* OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE STARTTLS AUTH=PLAIN] Dovecot ready.
a01 login bob s3cr3tpa55w0rd
a01 OK [CAPABILITY IMAP4rev1 LITERAL+ SASL-IR LOGIN-REFERRALS ID ENABLE IDLE SORT SORT=DISPLAY THREAD=REFERENCES THREAD=REFS THREAD=ORDEREDSUBJECT MULTIAPPEND URL-PARTIAL CATENATE UNSELECT CHILDREN NAMESPACE UIDPLUS LIST-EXTENDED I18NLEVEL=1 CONDSTORE QRESYNC ESEARCH ESORT SEARCHRES WITHIN CONTEXT=SEARCH LIST-STATUS SPECIAL-USE BINARY MOVE] Logged in
a02 list "" "*"
* LIST (\HasChildren) "." INBOX
* LIST (\HasChildren) "." INBOX.spam
* LIST (\HasNoChildren) "." INBOX.work
[...]

a03 EXAMINE INBOX
* OK [CLOSED] Previous mailbox closed.
* FLAGS (\Answered \Flagged \Deleted \Seen \Draft $Forwarded $label1 $MDNSent $label2 $label3 $label4 $label5 NonJunk $NotJunk $Junk)
* OK [PERMANENTFLAGS ()] Read-only mailbox.
* 3144 EXISTS
* 0 RECENT
* OK [UNSEEN 3142] First unseen.
* OK [UIDVALIDITY 1190930016] UIDs valid
* OK [UIDNEXT 44908] Predicted next UID
* OK [HIGHESTMODSEQ 78664] Highest
a03 OK [READ-ONLY] Examine completed (0.000 secs).

a04 FETCH 3144 BODY[]
* 3144 FETCH (BODY[] {3807}
From test@example.com  Thu May  7 09:12:26 2015
[...]
Return-Path: <test@example.com>
subject: this is a test
Date: Thu,  7 May 2015 09:11:08 +0200 (CEST)
From: test@example.com
foo bar baz
)
a04 OK Fetch completed.

a05 logout
* BYE Logging out
a05 OK Logout completed.

Use the PHP built-in web server

As of PHP 5.4, a built-in web server[8] is available for testing purposes:

$ mkdir ~/www
$ uname -a | /usr/games/cowsay | \
   sed -e '1s/^/<pre>\n/' -e '$s/$/&\n<\/pre>/' | \
   tidy -asxhtml -indent -utf8  > ~/www/index.html

$ php -S 10.0.0.3:8080 -t ~/www/
PHP 5.6.7 Development Server started at Tue Apr  7 13:03:12 2015
Listening on http://10.0.0.3:8080
Document root is /home/dummy/www
Press Ctrl-C to quit.

The built-in webserver does not support TLS, but we can use stunnel to emulate it. Again, this is for testing purposes only and not meant to be used in production!

Skip the first or last line

Because I always forget:

$ cat foo
1
2
3
4

Here's how to omit the first line of a file (or stdout):

$ tail -n +2 foo
2
3
4

And how to omit the last line:

$ head -n -1 foo 
1
2
3

Compare directory trees

To compare (remote) directories w/o transfering them over the network, we print the checksum of each file, generate the checksum of that (sorted) output and compare that:

$ find . -type f | xargs openssl md5 -r | sort -n 
e7e04e99d58f244b784ae69b3663db8e *./c/file-c
f2b105a7e0d17a453b9832e43590cc26 *./a/b/file-b
2ca3f8cfe4c59569a2c3bfc1d6ec0665 *./a/file-a
11bbccdd8883a732ffb9245ce50c9bfd *./file-0

$ find . -type f | xargs openssl md5 -r | sort -n | sha1sum 
ec0973f3e629d225f81c25b025a2cce4d3d70a11  -

Note: sorting is important here and may not even help (identical file with different file name and). Also, "./file" is different from "file".

Float comparison

While most shells cannot do float comparison, bc can:

$ echo '1.1 > 1.2' | bc 
0

$ echo '1.3 > 1.2' | bc 
1

Or even[9]:

$ bc <<< '1.5 > 1.9'
0

Epoch conversion

With GNU or BSD date:

$ date +%s
1415209417

And back again:

$ date -d @1415209417                       # GNU/date
Wed Nov  5 09:43:37 PST 2014

$ date -r 1415209417                        # BSD/date
Wed Nov  5 09:43:37 PST 2014

If neither versions of date are available[10], we can use perl (which also honors the TZ environment variable[11]):

$ perl -le "print time"
1415209417

$ TZ=GMT perl -le 'print scalar(localtime(1415209417))'
Wed Nov  5 17:43:37 2014

And again, with awk:

$ perl -le "print time" | TZ=GMT awk '{print strftime("%c", $1)}'
Wed Nov  5 17:43:37 2014

Network sockets in Bash

When bash is compiled with --enable-net-redirections[12], the following works:

$ exec 3<>/dev/tcp/example.org/80
$ echo -e "GET /hello.html HTTP/1.1\r\nhost: http://example.org\r\nConnection: close\r\n\r\n" >&3
$ cat <&3
<html>hello, world</html>

Internet speed test

For a lack of a better place, a few speedtest URLs:

http://cachefly.cachefly.net/100mb.test                               # Chicago, IL
http://speedtest.sjc01.softlayer.com/downloads/test100.zip            # San Jose, CA
http://speedtest.sea01.softlayer.com/downloads/test100.zip            # Seattle, WA
http://speedtest.dal01.softlayer.com/downloads/test100.zip            # Dallas, TX
http://speedtest.wdc01.softlayer.com/downloads/test500.zip            # Chantilly, VA
https://mirror.leaseweb.com/speedtest/1000mb.bin                      # LeaseWeb CDN

Keyboard layout

In an X11 session:

$ setxkbmap us                   # See /usr/share/X11/xkb/symbols/

On a console session:

$ loadkeys de                    # See /usr/share/{,kbd/}keymaps/ or /lib/kbd/keymaps/
Loading /usr/share/kbd/keymaps/i386/qwertz/de.map.gz
assuming iso-8859-15 euro

Note: loadkeys needs the console-data package in Debian/Ubuntu. In SUSE/RedHat, the keymaps are included in the kbd package.

Absolute values in bc

$ echo "val = 5-3;  if( val < 0 ) val*= -1; val" | bc
2
$ echo "val = 5-30; if( val < 0 ) val*= -1; val" | bc
25                                                                       # abs(-25)

Backup SQLite

From SQLite Backup and Restore:

sqlite3 database.db .dump > database.backup

Restore with:

sqlite3 database.db < database.backup

Remove characters in bash

$ A=hello
$ echo $A
hello

Remove characters from back of a string:

$ echo ${A%%o}
hell

Remove characters from front of a string:

$ echo ${A##h}
ello

Print characters repeatedly

If seq(1) is available:

$ seq 10 | while read a; do printf "x"; done; echo
xxxxxxxxxx

In Bash, maybe ZSH too:

for a in {0..9}; do printf x; done; echo

Bash-4.1 supports printf -v:

printf -v a "%10s"; printf "%s\n" "${a// /x}"

In Perl:

perl -e 'print "a" x 10, "\n"'

Zero Padding in Bash

$ x=8; while [ $x -lt 12 ]; do echo $x; x=$((x+1)); done
8
9
10
11

To have this zero-padded (in lieu of seq(1)), we could use printf:

$ x=8; while [ $x -lt 12 ]; do printf "%02d\n" $x; x=$((x+1)); done
08
09
10
11

The "2" denotes the digits of the number to pad. I.e. to pad numbers up to 100, one would use "printf "%03d".

Endianness

How to tell if a system is big endian or little endian? For Unix/Linux

$ printf I | od -t o2 | head -1 | cut -d" " -f2 | cut -c6
0                   ← Big Endian    (mips, parisc, power, powerpc, s390x, sparc)
1                   ← Little Endian (alpha, arm, ia64, mipsel, vax, x86, x86-64)

According to the author, this would not work for AIX (and MacOS X) and proposed a solution with awk(1):

echo I | tr -d [:space:] | od -t o2 | head -1 | awk '{print $2}' | cut -c6

The awk(1) version can then be "simplified" to:

printf I | od -t o2 | awk '{print substr($2,6,1); exit}'

And another one, w/o od(1):

printf I | hexdump -o | awk '{print substr($2,6,1); exit}'

dnsdomainname / FQDN

dnsdomainname won't return the domain name?

$ dnsdomainname
$ hostname --fqdn
alice

Add the FQDN to /etc/hosts:

$ grep `hostname` /etc/hosts
10.0.0.3     alice.example.org  alice
$ dnsdomainname
example.org

$ hostname --fqdn
alice.example.org

Regenerate blowfish_secret

To regenerate blowfish_secret for phpMyAdmin, use one of the following:

perl -le 'print map{("a".."z","A".."Z",0..9) [int(rand(62))]} (1..32)'
openssl rand -base64 32

This can then be included into /etc/phpmyadmin/config.inc.php:

$cfg['blowfish_secret'] = 'SyXP1ZVHwPXatLxIKprrmocZI0lK7M2z';

Note: there's no maximum length for this secret!

combination of fields in crontab(5)

Debian #460070 shows it quite clearly:

$ crontab -l
#m  h   dom  mon dow   command
21 11   5-15 *   Thu   echo test1
21 11   5-15 *   Fri   echo test2

Yet both commands were executed, although it was not a Thursday or a Friday. But crontab(5) specifies:

> Finally, if either the month or day of month is specified as an element or list,
> and the day of week is also specified as an element or list, then any day matching
> either the month and day of month or the day of week, will be matched. 

The aforementioned bugreport even has a workaround for this:

#m  h   dom  mon dow   command
21 11   5-15 *   *     [ "`date +%w`" -eq 4 ] && echo test1
21 11   5-15 *   *     [ "`date +%w`" -eq 5 ] && echo test2

python setup.py

To build & install a Python program:

python setup.py build
sudo python setup.py install --prefix=/opt/foo

However, for the new program to work, Python's search path may have to be modified:

$ /opt/foo/bin/foo
Traceback (most recent call last):
 File "/opt/foo/bin/foo", line 6, in <module>
   import bar
ImportError: No module named bar

$ PYTHONPATH=/opt/foo/lib/python2.6/site-packages /opt/foo/bin/foo -v
0.1

CR+LF

Applications and operating systems are implementing newlines differently:

LF    - Unix, MacOS 10
CR+LF - Windows, DOS
LR+CR - RISC OS
CR    - MacOS 8,9

Here's how to convert between them.

From LF to CR+LF, on Windows:

TYPE unix_file | FIND "" /V > dos_file

On Unix and Linux systems, there's unix2dos

dos2unix dos_file                  # This will modify dos_file and its original content will be lost!
dos2unix -n dos_file unix_file     # This will output to unix_file

And vice versa:

unix2dos -n unix_file dos_file

Newer versions of the same program are called tofrodos:

todos    unix_file                 # This will modify unix_file and its original content will be lost!
todos -b unix_file                 # This will leave a backup of the original file in unix_file.bak

And vice versa:

$ fromdos -b dos_file
$ file dos_file{,.bak}
dos_file:     ASCII text
dos_file.bak: ASCII text, with CRLF line terminators

The same, with tr(1):

tr -d '\r' < dos_file > unix_file                            # CR+LF → LF

With sed(1):

sed -e 's/$/\r/' unix_file > dos_file                        # Unix to DOS  (LF → CR+LF)
sed -e 's/\r$//' dos_file  > unix_file                       #  DOS to UNIX (CR+LF → LF)

With awk(1):

awk 'sub("$", "\r")'            unix_file > dos_file         # Unix to DOS
awk '{ sub("\r$", ""); print }' dos_file  > unix_file        #  DOS to Unix

With perl(1):

perl -p -e 's/\n/\r\n/' < unix_file > dos_file               # Unix to DOS 
perl -p -e 's/\r$//'    < dos_file  > unix_file              #  DOS to Unix

With vi(1):

:1,$s/^M//g                                                  #  DOS to Unix

For the sake of completeness, remove an entire linebreak[13]

sed ':a;N;$!ba;s/\n/ /g'

Let's try this:

$ printf 'a:\nb\n'
a:
b

$ printf 'a:\nb\n' | sed ':a;N;$!ba;s/:\n/: /g'
a: b

Patch strings in binary files with sed

Let's assume a rather simple file, with binary and ASCII parts mixed:

$ openssl rand 10 > file; echo hello >> file; openssl rand 10 >> file
$ cat file
�u�(8V���hello
��O�Id�y��

How to modify that ASCII part of that file?

Let's convert our target string into hex:

$ printf hello | xxd -p
68656c6c6f

Which is indeed part of our file:

$ xxd -p file 
c2758b28381656c7fbe068656c6c6f0a848d4f8f49648179bea3

We can replace that string with another string of equal length:

$ printf solar | xxd -p
736f6c6172

Let's replace occurences of hello with solar in our file:

$ xxd -p file | sed 's/68656c6c6f/736f6c6172/g' | xxd -r -p > file_2
$ cat file_2
�u�(8V���solar
��O�Id�y��

Remove binary data

Remove binary data from a file, with tr:

tr -cd '\11\12\15\40-\176' < file.bin > file.txt

This will preserve:

118: TAB
128: LF
158: CR
408 to 1768: printable characters[14]

With Perl:

perl -pe 's/[^[:ascii:]]//g' file.bin > file.txt

With (XPG4) awk:

awk '{gsub(/[[:cntrl:]]/,x)}1' file.bin > file.txt

.profile

When bash(1) is an interactive login shell, it reads:

/etc/profile, ~/.bash_profile, ~/.bash_login, ~/.profile

When started as an interactive non-login shell:

/etc/bash.bashrc  and  ~/.bashrc

For zsh(1):

/etc/profile, $HOME/.profile                # 
/etc/zsh/zshrc, $ZDOTDIR/.zshrc             # Interactive non-login

To provide a logout script to shells that don't offer one, add the following to a login script:

trap '$HOME/.logout' 0

The .logout file should exist and needs to be executable too.

cpio

List:

cpio -tv < foo.cpio                                   # List, verbose


Extract:

cpio -ivd < foo.cpio                                  # Extract, verbose, create directories

Extract just one file:

cpio -ivd dir/file.ext < foo.cpio

Create:

find . -depth -print | cpio -o > ../foo.cpio          # Create

Copy:

find . -depth -print | cpio -pdumv ../new_directory

List contents of an RPM package:

rpm2cpio file.rpm | cpio -tv

Extract (but don't install):

rpm2cpio file.rpm | cpio -ivd

Rescan SCSI/FC bus

$ ls -1d /sys/bus/scsi/devices/target*
/sys/bus/scsi/devices/target0:0:0
/sys/bus/scsi/devices/target0:0:1
/sys/bus/scsi/devices/target0:0:2
/sys/bus/scsi/devices/target2:0:0

$ echo "- - -" > /sys/class/scsi_host/host0/scan                        # SCSI
$ echo 1 > /sys/class/fc_host/host0/issue_lip                           # FC
$ dmesg
scsi 0:0:2:0: Direct-Access     VMware   Virtual disk     1.0  PQ: 0 ANSI: 2
sd 0:0:2:0: [sdc] 20971520 512-byte logical blocks: (10.7 GB/10.0 GiB)
sd 0:0:2:0: [sdc] Write Protect is off
sd 0:0:2:0: [sdc] Mode Sense: 03 00 00 00
sd 0:0:2:0: [sdc] Cache data unavailable
sd 0:0:2:0: [sdc] Assuming drive cache: write through
sd 0:0:2:0: [sdc] Cache data unavailable
sd 0:0:2:0: [sdc] Assuming drive cache: write through
 sdc: unknown partition table
sd 0:0:2:0: [sdc] Cache data unavailable
sd 0:0:2:0: [sdc] Assuming drive cache: write through
sd 0:0:2:0: [sdc] Attached SCSI disk

To add a single device (HBTL: host, bus, target, lun) to a SCSI host:

$ echo "scsi add-single-device <H> <B> <T> <L>" > /proc/scsi/scsi

tar without resource forks

Setting COPYFILE_DISABLE will omit resource forks from tar-archives:

COPYFILE_DISABLE=1 tar -cvf foo.tar example/

Before MacOS 10.5, COPY_EXTENDED_ATTRIBUTES_DISABLE had to be set. While not documented, it seems as Apple patched gnutar w/o mentioning the variable in the man page.

PAGE_SIZE

The page size can be found out with the getpagesize syscall:

$ getconf PAGE_SIZE
4096                              # Solaris 10/i386
8192                              # Solaris 10/sun4u
4096                              # MacOS 10.7.5/x68-64
4096                              # Linux 3.8/powerpc
4096                              # Linux 2.6/x86-64

parted: The resulting partition is not properly aligned for best performance

GNU/parted might comlain about inproper alignment:

(parted) mkpart primary ext2 0 -1
Warning: The resulting partition is not properly aligned for best performance.
Ignore/Cancel? c

Let's begin the partition at sector 1, rather than 0:

(parted) mkpart primary ext2 1 -1

(parted) p
Model: VMware Virtual disk (scsi)
Disk /dev/sdb: 10.7GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos

Number  Start   End     Size    Type     File system  Flags
1      1049kB  10.7GB  10.7GB  primary

Note: the primary keyword only needs to be given for MSDOS partitions and can be omitted for GPT partition tables.

base64 decode

Via perl:

 perl -MMIME::Base64 -ne 'print decode_base64($_)' < file.txt > file.out

On MacOS X we also have /usr/bin/base64:

 base64 -D -i file.txt -o file.out

iPod import

Ever wanted to import from your iPod to your computer? According to the FAQ, this is not a common usecase:

 Synchronization generally occurs only in one direction, from your computer to your iPod.
 This means you typically cannot transfer music, automatically or manually, from your iPod 
 to a computer, and you cannot use iPod to copy a music library from one computer to another.

To get around this, several tools exist:

ImageMagick for dummies

Sometimes we just want to convert a bunch of files, quickly.

Resize:

   convert file.jpg -resize 50% file.png                       # For ImageMagick
gm convert file.jpg -resize 50% file.png                       # For GraphicsMagick

Scale:

   convert file.jpg -scale 50% file-scaled.jpg                 # For ImageMagick
gm convert file.jpg -scale 50% file-scaled.jpg                 # For GraphicsMagick

Rotate:

   convert file.jpg -rotate -90 file-left.jpg                  # For ImageMagick
gm convert file.jpg -rotate -90 file-left.jpg                  # For GraphicsMagick

Geometry:

   convert file.jpg -geometry '100x100'  file-quadrant.jpg
gm convert file.jpg -geometry '100x100!' file-quadrant.jpg     # w/o maintaining the aspect ratio

Remove all Exif data:

$ exiv2 -d e rm file.jpg 
$ exiv2 pr file.jpg 
File name       : file.jpg
File size       : 59360 Bytes
MIME type       : image/jpeg
Image size      : 576 x 768
file.jpg: No Exif data found in the file

Remove just the geo location from the picture:

$ exiftool -GPSPosition file.jpg
GPS Position                    : 23.0" N, 42.00" W

$ exiftool -gps:all= -xmp:geotag= file.jpg

rsync to vfat

Apparently the resolution time of Windows FAT filesystems is two seconds[15] (for writes). We'll use --modify-window=2[16] to not copy already transferred files again:

rsync -rvP --modify-window=2 src/ bob:/mnt/usb/

Unable to Log You on Because of an Account Restriction

When trying to login via RDP, this message appears:

 Unable to log you on because of an account restriction. 

This happens when there's no password set for the account you're trying to login with. Either set a password (recommended!) or (temporarily) disable this policy:

  1. Run gpedit.msc (Group Policy Editor)
  2. Open Computer Configuration\Windows Settings\Security Settings\Local Policies\Security Options
  3. Double Click on Accounts: Limit local account use of blank passwords to console logon only
  4. Select Disabled
  5. Quit Group Policy Editor


history timestamps in zsh

 $ setopt extendedhistory
 $ history -i | tail -1
  24  2011-04-01 02:41  df
  25  2011-04-01 02:41  ls
  26  2011-04-01 02:42  du
  27  2011-04-01 02:42  top
  28  2011-04-01 02:42  dmesg

Note: While the timestamps in HISTFILE are exact to the second, the output of history -i is not :-\

history merging in bash

Read the contents of the old and the current history file and append them to the current (in-memory) history list:

history -r .bash_history.old
history -r .bash_history

Write the current history list to the history file, overwriting the history file's contents.

history -w

Inode count in ZFS

Well, let's say file count, for this example:

 $ time zdb -dv rpool | egrep -c 'ZFS (directory|plain file)'
 167762
 
 real    0m49.146s
 user    0m26.281s
 sys     0m5.348s

Create Windows user accounts

How to create local user accounts in Windows NT:

  @echo off
  for /F "tokens=1,2,3 delims=," %%a IN (users.txt) DO \
      net user %%a Passw0rd /add /fullname:"%%b" /comment:"%%c" /expires:never

MySQL: large BLOB inserts

According to #45236, adding max_allowed_packet helps:

 bzip2 -dc sql-backup-with-blobs.sql.bz2 | mysql --max_allowed_packet=128M -D dbname

Fun with subtitles

I had a subtitle file for a movie, which, at one point later in the movie went off by 1 minute or so. Rather than editing the timestamps I found a second subtitle file - which was wrong at the beginning of the movie but got it right at least from the point the first file wandered off. So here's what I did:

 $ tr -d '\r' < file-1.srt > file-1.copy
 $ tr -d '\r' < file-2.srt > file-2.copy
 
 $ head -3 file-2.copy
 770
 01:22:29,860 --> 01:22:32,900
 The first was Esteban Vihaio.
 
 # somehow we cannot use ^[0-9]*$
 $ sed -e 's/^[0-9][0-9][0-9]$/XXX/;s/^[0-9][0-9][0-9][0-9]$/XXX/' -i file-2.copy
 $ for i in `seq 770 1132`; do echo $i; sed "0,/^XXX$/s//"$i"/" -i file-2.copy; done
 
 # now we can merge both files
 $ cat file-1.copy file-2.copy > file.srt

Yeah, due to its ugliness I did not want to blog about it :-)

Clickable URLs in Terminal.app

Until before Terminal.app came without clickable URL support, one had to use ICeCoffEE to get this working. However, Terminal.app supports this already:

  1. Move mouspointer over URL, you don't have to mark it
  2. Press right mouse-button & click "Open URL"

Delete all files older than...

Delete all files (and directories?) in C:\backup that are older than 60 days:

 forfiles /P C:\backup /S /M *.* /D -60 /C "cmd /c del /q @path"

This appears to work on Windows7 and Windows Server 2003.

./configure

Just checked out a source tree but the ./configure file is missing? Try, in that order:

autoreconf --install

If that did not work, maybe this will:

libtoolize --copy --install --force

cp include/install-sh .          # If needed
aclocal                          # Try adding -I config or -I m4 if needed. This should generate "aclocal.m4"
autoheader                       # If needed

The next step will try to create ./configure:

autoconf

The next step will try to create ./Makefile.in:

automake                              # Try adding --add-missing if needed

With both ./configure and ./Makefile.in in place, running ./configure will (hopefully) generate a Makefile.

Esound

 esd -tcp -public -noterminate -port 5001 -b        # server
 esddsp -s hamma:5001 mpg321 -o esd foo.mp3         # client
 esddsp -s hamma:5001 ogg123 -d esd foo.ogg
 esddsp -s hamma:5001 mplayer -ao esd foo.mp[34]

This costs you 64 MB of RAM

We're losing 64MB:

 [    0.004000] Your BIOS doesn't leave a aperture memory hole
 [    0.004000] Please enable the IOMMU option in the BIOS setup
 [    0.004000] This costs you 64 MB of RAM

How to install IRIX packages

 wget http://freeware.nekochan.net/cd-3/dist/fw_pine
 wget http://freeware.nekochan.net/cd-3/dist/fw_pine.idb
 wget http://freeware.nekochan.net/cd-3/dist/fw_pine.sw
 /usr/sbin/showfiles -f fw_pine -x -- pine

It might complain about missing .man and .src files - ignore them if you don't need them.

dir_index

Check/enable htree (dir_index) for already existing ext2/ext3 filesystems:

 # tune2fs -l /dev/DEVICE | grep features
 # tune2fs -O dir_index /dev/DEVICE

However, already existing data will not be indexed right away. For this to happen you'll have to either:

  $ mkdir dir.new
  $ mv dir/* dir.new/
  $ rmdir dir
  $ mv dir.new dir

tar/ssh

  • Get remote directory /data/pr0n and create pr0n.tar.gz on the local host:
 ssh remote "cd /data; tar -cf - pr0n" | (gzip -c > /u3/pr0n.tar.gz )
  • Get remote directory /data/pr0n and extract underneath /u3 on the local machine. Note how we "cd /data" on the remote host first so that we DON'T create an archive with absolute pathnames:
 ssh remote "cd /data; tar -cf - pr0n" | (cd /u3; tar -xf - )
  • Grab local /u3/foo and create /bar/foo.tar on the remote host:
 (cd /u3; tar -cf - foo) | ssh remote "cat > /bar/foo.tar"
  • Grab local /u3/foo and extract it underneath /bar on the remote host:
 (cd /u3; tar -cf - foo) | ssh remote "cd /bar; tar -xpf -"

VMware/ethernet0.addressType

Want to have static IP addresses for all your vmware-machines? Change ethernet0.addressType in the .vmx file and put the same IP address in every guest (GNU/sed needed for '-i' to work):

 sed -i 's/ethernet0.addressType = "generated"/ethernet0.addressType = "static"/' *.vmx
 sed -i 's/ethernet0.generatedAddress = ".*"/ethernet0.address = "00:50:56:38:36:8a"/' *.vmx
 sed -i '/^ethernet0.generatedAddressOffset = /d' *.vmx

See this thread about choosing a valid MAC address. We're limited to addresses starting with 00:50:56.

SO_BSDCOMPAT

Want to get rid of the annoying process ... is using obsolete setsockopt SO_BSDCOMPAT warnings? When compiling ISC BIND, apply this patch to disable this (stale) option:

--- bind-9.4.0b2/lib/isc/unix/socket.c.ORIG     2006-10-05 15:52:38.575470725 +0100
+++ bind-9.4.0b2/lib/isc/unix/socket.c  2006-10-05 15:53:38.935470726 +0100
@@ -245,6 +245,8 @@ static void build_msghdr_send(isc_socket
                              struct msghdr *, struct iovec *, size_t *);
 static void build_msghdr_recv(isc_socket_t *, isc_socketevent_t *,
                              struct msghdr *, struct iovec *, size_t *);
+#undef SO_BSDCOMPAT
+

 #define SELECT_POKE_SHUTDOWN           (-1)
 #define SELECT_POKE_NOTHING            (-2)

.inputrc

Is your DELETE key not working? END key generating strange characters? Try to create an ~/.inputrc with the following content:

 set input-meta on
 set output-meta on
 set convert-meta off
 "\e[1~": beginning-of-line
 "\e[4~": end-of-line
 "\e[3~": delete-char
 "\e[2~": quoted-insert

unpacking .deb files

Normally, one would unpack .deb files with:

 $ dpkg-source -x ../file.deb .
 dpkg-source: error: syntax error in source control file ../file.deb at line 1: \
                     line with unknown format (not field-colon-value)

Whoops, didn't work. Well, it's an Ubuntu .deb and my dpkg-source is from an aging Debian installation. So, let's try something else:

 $ ar vx ../file.deb
 x - debian-binary
 x - control.tar.gz
 x - data.tar.gz
 
 $ tar -xzf data.tar.gz

Or, in one step:

 $ ar p ../file.deb data.tar.gz | tar -xz

dd under Windows

 dd if=\\?\Device\CdRom0 of=c:\temp\disc1.iso bs=1M
 dd if=\\.\a: of=c:\temp\disk1.img bs=1440k

apt-get: fatal region error detected; run recovery

 db4.6_dump /var/lib/apt/listchanges.db | db4.5_load a.db
 mv a.db /var/lib/apt/listchanges.db

rdiff-backup

rdiff-backup is a fairly flexible, feature-rich and good-documented backup-utility. Unfortunately it is written in a script-language (python) and so it is prone to occasional br0keness due to library changes and what not. So I really had to compile it by myself because the netbsd-stable version was too old. I installed it in /usr/local, the version-error was gone, but now I got this:

 # rdiff-backup
 Traceback (most recent call last):
   File "/usr/local/bin/rdiff-backup", line 20, in ?
     import rdiff_backup.Main
 ImportError: No module named rdiff_backup.Main

I was told to fix this by setting PYTHONPATH to the path, where your python site-packages are installed. And it worked:

 # export PYTHONPATH=/usr/local/lib/python2.3/site-packages
 # rdiff-backup -V
 rdiff-backup 1.1.5

Screen

 # Start a new screen session, give it a name:
 screen -US foo
 
 # Detach session:
 CTRL-a, d
 
 # List active sessions:
 screen -ls
 
 # Resume session "foo":
 screen -r foo
 
 # Resume my only session, possibly disconnecting remotely attached terminals:
 screen -dr
 
 # Create new windows within a session, list them:
 CTRL-a, c
 CTRL-a, "
 
 # Split, switch, unsplit windows
 CTRL-a S
 CTRL-a <TAB>
 CTRL-a Q

While ^e to jump to the end of the line might still work, ^a is now taken. Try ^aa to get to the beginning of the line.

Beep!

Disable the beep! in X11 terminals:

 xset b 0

References