How to auto-run a script on boot with Linux / Raspberry PI

If you want something to happen when you login, then there are several ways to do that, but that is not what this post is about.

With a Raspberry PI it can be very nice to configure the device to have a service running or to do some operation which you want to happen at boot and without user interaction, to do this use:

sudo vi /etc/rc.local

If you have a problem with this script or you need to break out of it running (similar to CTRL+C in a shell) then use:

alt + print screen + K

Have a nice day!

Advertisements

How can I see where libraries are being loaded from on Linux?

To figure out where libraries are being loaded from, if you have your environment already setup in the same way for which you want to test, then you can run:

/sbin/ldconfig -N -v

However, this will not search LD_LIBRARY_PATH so you must also include that manually:

/sbin/ldconfig -N -v $(sed ‘s/:/ /g’ <<< $LD_LIBRARY_PATH)

If you would like to see which libraries are actually being loaded when running an executable, then use:

strace myprog

This will show you a lot more than you care to see (all system calls), but if you grep the results for “^open.*\.so”, then you will see all of the *.so files which are being opened from that process.

I also see several processes which fork child processes and strace will not report system calls for these by default. However, you can add the ‘-f’ switch to strace and then all child processes will be reported:

strace -f myprog

This will produce a fair amount of noise, but you can filter that with:

strace -e trace=open -f myprog

Using rpm to find an installed program (RedHat, CentOS, Fedora)

To see what programs are installed use:

rpm -qa

If you just installed a program and want to know where it went then grep for it:

$ rpm -qa | grep vim-X11
vim-X11-7.4.160-2.el7.x86_64

Once you have the name of the package, then you can list the files it installed:

$ rpm -ql vim-X11-7.4.160-2.el7.x86_64
/usr/bin/evim
/usr/bin/gex
/usr/bin/gview
/usr/bin/gvim
/usr/bin/gvimdiff
/usr/bin/gvimtutor
/usr/bin/vimx
/usr/share/applications/gvim.desktop
/usr/share/icons/hicolor/16x16/apps/gvim.png
/usr/share/icons/hicolor/32x32/apps/gvim.png
/usr/share/icons/hicolor/48x48/apps/gvim.png
/usr/share/icons/hicolor/64x64/apps/gvim.png
/usr/share/man/man1/evim.1.gz

There is my gvim I was looking for.

P.S. If these newly installed executables aren’t working from PATH, then try opening a fresh shell.

Searching large source trees in an efficient way on Linux

TL;DR

Here is the alias:

alias search 'find \!:1 -noleaf -type f -not -path "*/boost/*" -not -path "*/extensions/*" -print0 | xargs -0 -n 100 -P 8 grep -I --color -H -n \!:2*'

 

How do I use it?

Here is how I use it:

search [dir] [term] [grep_options]
e.g.
search ./src/ the\ search\ term
search ./src/ keyTerm -A5 -B5

How does it work?

find

This search alias uses find as follows to locate all files under the provided directory (i.e. first argument) while excluding directories that we don’t care about:

find \!:1 -noleaf -type f -not -path "*/boost/*" -not -path "*/extensions/*" -print0

For aliases remember this:

!* is all but the first
!:0 is only the first, the command itself
!:1 is only the first argument
!:2* is all but the first argument
!$ is only the last argument
!:1- is all but the last argument
!! is all
$0 is the shell
$# is the number of args
$$ is the process id (PID)
$! is the PID of the previous command
$? is the return code from the previous command

Thus, the “\!:1” means only the first argument, and the bang (!) has to be escaped.

\!:1

The “-noleaf” is used because I am normally working on Windows/NTFS mounts and it is not safe to assume that directories containing 2 fewer subdirectories than their hard link count only contain files.

-noleaf

We only want to gather files for searching so I use the “-type f”.

-type f

I normally have very large directories which I do not care to search in, so I specify:

-not -path "*/boost/*" -not -path "*/extensions/*"

Finally for the find command I pass “-print0” which returns null (instead of new line) terminated strings. This adds support for paths with spaces in them:

-print0

xargs

The xargs command controls how many files are being passed into grep and it is handling running them in parallel.

xargs -0 -n 100 -P 8 grep -I --color -H -n \!:2*

The “-0” option is used here to tell xargs that the strings coming in are null terminated (this adds support for files with spaces):

-0

The “-n 100” and the “-P 8” options are where the speed and power of this alias come from. The “-n 100” is telling xargs to pass 100 files from find into grep at a time. The “-P 8” is telling xargs to run 8 grep commands in parallel.

This means that if we have a source tree of 1600 files, then grep will be called 16 times and each will be passed 100 files. The best part is that 8 of those grep commands will be running in parallel each on 100 files, so the command finishes as if there were only two (2) grep invocations – very fast even on large source trees:

-n 100 -P 8

grep

The grep command is used to do the actual searching in files.

grep -I --color -H -n \!:2*

The “-I” option ignores binary files:

-I

Colored results make it much easier to see hits:

--color

Because we are passing in the files to grep it may not show the file name where the hit occurred so we add “-H” to print the file name:

-H

The line number is also important, so we add “-n”:

-n

The ability to control grep is handled with an arguments wildcard. Here the “\!:2*” means the second and all subsequent arguments passed into the search alias. Thus the grep search term and all other grep options can be specified after the directory to search:

\!:2*

The final piece is that the xargs command will add the files from the find command to the grep command. It will add 100 (or less if there are less than 100) files to every grep command and each of those will be run in parallel with up to 8 running at any given time.

Enjoy your searching.

How can I easily access my Linux command history? Is there a hot key?

One of the fastest ways to search your previous commands is to use CTRL+R and start typing, once you’ve entered enough text you can use CTRL+R again and again to search your history for matches.

Let’s assume we execute the following commands:

$ echo dog
dog
$ echo cat
cat
$ echo hotdog
hotdog

Now press CTRL+R and you will see a new “bck:” prompt at the bottom:

$
bck:

Now if you type “dog” you will see the last command that had that string anywhere in it, populated on the previous command prompt:

$ echo hotdog
bck:dog

Pressing CTRL+R again will cycle through the history:

$ echo dog
bck:dog

You can press ENTER to execute the command, or CTRL+E to go to the end of the command without executing it.

I also hear you can use CTRL+S to go backwords through the search results, but that never works for me – I believe my terminal or window manager is swallowing the CTRL+S.

 

How do you SSH without a password and what permissions are needed on the .ssh files?

SSH stands for Secure Shell and it is a protocol which enables secure network service connections over unsecured networks. For most users they think of SSH as a way to remotely connect to and control another machine. Every time you establish a new connection you will need to authenticate. This is generally done using a password, but a secure key can be used instead. This article talks about using a key so you don’t have to enter a password and the default .ssh permissions are listed out here too – these are the permissions you should use on these files for security reasons.

 

Home Directory Group

I have encountered problems where I switched groups in my company and the machines I would connect to were expecting a different user group permissions on my home drive.  Because I didn’t have the correct group on my home directory SSH daemons on these machines that were owned by the new group were unable to read my ~/.ssh/ files.

You can test this by changing your home directory permissions to 755. Note: Most IT departments will want your home directory locked down (700), so this might not be something you want to do.

If you know someone else who is able to SSH without a password, then see what group is set on their home. You can change this with chown, e.g.

$ ls -ld ~
drwx------ 109 me old 196608 Jan 24 17:38 .
$ chown me:new ~
$ ls -ld ~
drwx------ 109 me new 196608 Jan 24 17:38 .

Preserve Old .ssh Folder

If you don’t have a ~/.ssh folder, then you don’t need to worry about this step.

Preserve your old .ssh folder by moving it to a new name. This prevents us from losing any important keys that we might find out later we were using.

mv ~/.ssh{,.old}

This will move ~/.ssh to ~/.ssh.old.

Create the Key

Now let’s create our key. The SSH tools, provided on most Linux machines, include a command to generate keys: ssh-keygen

You can tell this command what type of key to generate with -t, e.g. ssh-keygen -t rsa

For our use case, we want an RSA key which is the default, so I won’t specify one here. We also don’t want a password, but if you did you could enter one.

$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/me/.ssh/id_rsa): [ENTER]
Created directory '/home/me/.ssh'.
Enter passphrase (empty for no passphrase): [ENTER]
Enter same passphrase again: [ENTER]
Your identification has been saved in /home/me/.ssh/id_rsa.
Your public key has been saved in /home/me/.ssh/id_rsa.pub.
The key fingerprint is:
9a:f0:d2:de:dd:1a:69:b1:ee:c1:a3:f1:4c:7f:07:51 me@myserver
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|                E|
|               . |
|              .  |
|   . S  .      . |
|  +  o  .  +   . |
|       . = . X . |
| o  .  .  @  =  o|
|      . . o.B ...|
+-----------------+

Let’s see what was created and also take note of the current permissions:

$ ls -la ~/.ssh
total 208
drwx------   2 me grp   4096 Jan 24 10:31 .
drwxr-xr-x 110 me grp 196608 Jan 24 10:31 ..
-rw-------   1 me grp   1675 Jan 24 10:31 id_rsa
-rw-r-----   1 me grp    401 Jan 24 10:31 id_rsa.pub

Looks like we’d expect, right. The ~/.ssh/id_rsa (private key) is only readable by me, and the ~/.ssh/id_rsa.pub (public key) is readable by my group.

Share Public Key

In order to SSH into a machine you will need that machine to provide your key with access. This is accomplished by adding your public key to the ~/.ssh/authorized_keys file.

Notice: The ~/.ssh/authorized_keys file I’m referring to here is on the machine you want to connect to – not the machine you are currently on. This could even be under a different user.

This can even work for root at /root/.ssh/authorized_keys.

Regardless of the target machine configuration, if it has SSH installed and a daemon running (which most will), then you can send your key to this machine with the ssh-copy-id command:

$ ssh-copy-id mymachine
The authenticity of host 'mymachine(1.2.3.4)' can't be established.
RSA key fingerprint is 17:62:db:8d:c6:84:fa:6a:84:8d:c6:19:31:75:1a:fd.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'mymachine,1.2.3.4' (RSA) to the list of known hosts.
me@mymachine's password:
Now try logging into the machine, with "ssh 'mymachine'", and check in:

.ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.

Let’s see what was created and also take note of the current permissions:

$ ls -la ~/.ssh
total 208
drwx------   2 me grp   4096 Jan 24 10:31 .
drwxr-xr-x 110 me grp 196608 Jan 24 10:31 ..
-rw-------   1 me grp    432 Jan 24 11:06 authorized_keys
-rw-------   1 me grp   1675 Jan 24 10:31 id_rsa
-rw-r-----   1 me grp    401 Jan 24 10:31 id_rsa.pub
-rw-r--r--   1 me grp    408 Jan 24 11:02 known_hosts

Notice that we now have two more files. The authorized_keys file which we mentioned and expected from earlier, but the known_hosts we have not talked about and that is also new – we will talk more about that later.

If your home directory is the same on all machines

This becomes really nice if you connect to several machine which all share your same home directory. This means that you can put your own public key into your own authorized_keys file and then every machine you connect to on the network will let you connect without a password.

If you have the same home drive on the machine(s) you want to connect to, then instead of using ssh-copy-id, you can just add the public key to the authorized_keys file:

cat ~/.ssh/id_rsa.pub &gt;&gt; ~/.ssh/authorized_keys

The known_hosts file

The known_hosts hosts file keeps a list of finger prints from machines you connect to. The first time you connect you will see this:

The authenticity of host 'mymachine(1.2.3.4)' can't be established.
RSA key fingerprint is 17:62:db:8d:c6:84:fa:6a:84:8d:c6:19:31:75:1a:fd.
Are you sure you want to continue connecting (yes/no)?

If you type yes, then every time after this first connection you will not see this message. This mechanism is in place to protect you in case an attacker added a machine with this host name to the network you could connect to the correct host name, but get the wrong machine.

If this happens then you will see a message like this:

$ ssh mymachine
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
87:62:db:fd:c6:82:f9:6a:84:8d:c6:29:31:75:1a:f9.
Please contact your system administrator.
Add correct host key in /home/me/.ssh/known_hosts to get rid of this message.
Offending key in /home/me/.ssh/known_hosts:1
RSA host key for mymachine has changed and you have requested strict checking.
Host key verification failed.

Have fun sshing!

How do you chain commands in BASH or CSH?

I often need to run several commands which can take an hour or more and I won’t necessarily be present the entire time. It is nice to be able to run a command, and if it is successful, then run another; However, if one command fails, then stop the flow.  This article will tell you how to do this on the command line.

Chaining On Success

Perhaps you want to build, and then test. It would be annoying if the tests ran even if the build failed.

To chain the commands so subsequent commands will only run if the preceding command was successful (returned a zero):

make && run_test && echo "SUCCESS"

Chaining On Failure

To run a command followed by another command, only if the first command failed, then use:

make || echo "BUILD FAILED!"

You can also combine them:

(sh -c "exit 0" && sh -c "exit 1") && echo "SUCCESS" || echo "FAIL"

With Email

This becomes very powerful if you send yourself an email with the results so you know when a run completes and the result, e.g.

(sh -c "exit 0" && sh -c "exit 1") && mailx -s "Build: SUCCESS" $USER < /dev/null || mailx -s "Build: FAIL" $USER < /dev/null

Chaining Regardless

To run a command followed by another command, regardless of the return code, then use:

make; echo "BUILD DONE - make returned code: $?"

How does this work?

It is easiest to think of this all simply as Boolean logic. Boolean comparisons will only evaluate until the result can be determined.

On Linux a return code from a command of “0” means success. All other return codes are regarded as failures. This can be confusing because normal Boolean logic uses a 1 for true, e.g.

1 && 1 == 1, Evaluates the first and second expression
1 && 0 == 0, Evaluates the first and second expression
0 && 1 == 0, Only evaluates the first expression
0 && 0 == 0, Only evaluates the first expression

However, for our return codes this looks like:

exit 0 && exit 0 == Success, Executes the first and second command
exit 0 && exit 1 == Fail, Executes the first and second command
exit 1 && exit 0 == Fail, Only executes the first command
exit 1 && exit 1 == Fail, Only executes the first command

We can use a similar example for OR:

1 || 1 == 1, Only evaluates the first expression
1 || 0 == 1, Only evaluates the first expression
0 || 1 == 1, Evaluates the first and second expression
0 || 0 == 0, Evaluates the first and second expression

With return codes this looks like:

exit 0 || exit 0 == Success, Only executes the first command
exit 0 || exit 1 == Success, Only executes the first command
exit 1 || exit 0 == Success, Executes the first and second command
exit 1 || exit 1 == Fail, Executes the first and second command