How do you SSH without a password and what permissions are needed on the .ssh files?

SSH stands for Secure Shell and it is a protocol which enables secure network service connections over unsecured networks. For most users they think of SSH as a way to remotely connect to and control another machine. Every time you establish a new connection you will need to authenticate. This is generally done using a password, but a secure key can be used instead. This article talks about using a key so you don’t have to enter a password and the default .ssh permissions are listed out here too – these are the permissions you should use on these files for security reasons.

 

Home Directory Group

I have encountered problems where I switched groups in my company and the machines I would connect to were expecting a different user group permissions on my home drive.  Because I didn’t have the correct group on my home directory SSH daemons on these machines that were owned by the new group were unable to read my ~/.ssh/ files.

You can test this by changing your home directory permissions to 755. Note: Most IT departments will want your home directory locked down (700), so this might not be something you want to do.

If you know someone else who is able to SSH without a password, then see what group is set on their home. You can change this with chown, e.g.

$ ls -ld ~
drwx------ 109 me old 196608 Jan 24 17:38 .
$ chown me:new ~
$ ls -ld ~
drwx------ 109 me new 196608 Jan 24 17:38 .

Preserve Old .ssh Folder

If you don’t have a ~/.ssh folder, then you don’t need to worry about this step.

Preserve your old .ssh folder by moving it to a new name. This prevents us from losing any important keys that we might find out later we were using.

mv ~/.ssh{,.old}

This will move ~/.ssh to ~/.ssh.old.

Create the Key

Now let’s create our key. The SSH tools, provided on most Linux machines, include a command to generate keys: ssh-keygen

You can tell this command what type of key to generate with -t, e.g. ssh-keygen -t rsa

For our use case, we want an RSA key which is the default, so I won’t specify one here. We also don’t want a password, but if you did you could enter one.

$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/me/.ssh/id_rsa): [ENTER]
Created directory '/home/me/.ssh'.
Enter passphrase (empty for no passphrase): [ENTER]
Enter same passphrase again: [ENTER]
Your identification has been saved in /home/me/.ssh/id_rsa.
Your public key has been saved in /home/me/.ssh/id_rsa.pub.
The key fingerprint is:
9a:f0:d2:de:dd:1a:69:b1:ee:c1:a3:f1:4c:7f:07:51 me@myserver
The key's randomart image is:
+--[ RSA 2048]----+
|                 |
|                E|
|               . |
|              .  |
|   . S  .      . |
|  +  o  .  +   . |
|       . = . X . |
| o  .  .  @  =  o|
|      . . o.B ...|
+-----------------+

Let’s see what was created and also take note of the current permissions:

$ ls -la ~/.ssh
total 208
drwx------   2 me grp   4096 Jan 24 10:31 .
drwxr-xr-x 110 me grp 196608 Jan 24 10:31 ..
-rw-------   1 me grp   1675 Jan 24 10:31 id_rsa
-rw-r-----   1 me grp    401 Jan 24 10:31 id_rsa.pub

Looks like we’d expect, right. The ~/.ssh/id_rsa (private key) is only readable by me, and the ~/.ssh/id_rsa.pub (public key) is readable by my group.

Share Public Key

In order to SSH into a machine you will need that machine to provide your key with access. This is accomplished by adding your public key to the ~/.ssh/authorized_keys file.

Notice: The ~/.ssh/authorized_keys file I’m referring to here is on the machine you want to connect to – not the machine you are currently on. This could even be under a different user.

This can even work for root at /root/.ssh/authorized_keys.

Regardless of the target machine configuration, if it has SSH installed and a daemon running (which most will), then you can send your key to this machine with the ssh-copy-id command:

$ ssh-copy-id mymachine
The authenticity of host 'mymachine(1.2.3.4)' can't be established.
RSA key fingerprint is 17:62:db:8d:c6:84:fa:6a:84:8d:c6:19:31:75:1a:fd.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'mymachine,1.2.3.4' (RSA) to the list of known hosts.
me@mymachine's password:
Now try logging into the machine, with "ssh 'mymachine'", and check in:

.ssh/authorized_keys

to make sure we haven't added extra keys that you weren't expecting.

Let’s see what was created and also take note of the current permissions:

$ ls -la ~/.ssh
total 208
drwx------   2 me grp   4096 Jan 24 10:31 .
drwxr-xr-x 110 me grp 196608 Jan 24 10:31 ..
-rw-------   1 me grp    432 Jan 24 11:06 authorized_keys
-rw-------   1 me grp   1675 Jan 24 10:31 id_rsa
-rw-r-----   1 me grp    401 Jan 24 10:31 id_rsa.pub
-rw-r--r--   1 me grp    408 Jan 24 11:02 known_hosts

Notice that we now have two more files. The authorized_keys file which we mentioned and expected from earlier, but the known_hosts we have not talked about and that is also new – we will talk more about that later.

If your home directory is the same on all machines

This becomes really nice if you connect to several machine which all share your same home directory. This means that you can put your own public key into your own authorized_keys file and then every machine you connect to on the network will let you connect without a password.

If you have the same home drive on the machine(s) you want to connect to, then instead of using ssh-copy-id, you can just add the public key to the authorized_keys file:

cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys

The known_hosts file

The known_hosts hosts file keeps a list of finger prints from machines you connect to. The first time you connect you will see this:

The authenticity of host 'mymachine(1.2.3.4)' can't be established.
RSA key fingerprint is 17:62:db:8d:c6:84:fa:6a:84:8d:c6:19:31:75:1a:fd.
Are you sure you want to continue connecting (yes/no)?

If you type yes, then every time after this first connection you will not see this message. This mechanism is in place to protect you in case an attacker added a machine with this host name to the network you could connect to the correct host name, but get the wrong machine.

If this happens then you will see a message like this:

$ ssh mymachine
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that the RSA host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
87:62:db:fd:c6:82:f9:6a:84:8d:c6:29:31:75:1a:f9.
Please contact your system administrator.
Add correct host key in /home/me/.ssh/known_hosts to get rid of this message.
Offending key in /home/me/.ssh/known_hosts:1
RSA host key for mymachine has changed and you have requested strict checking.
Host key verification failed.

Have fun sshing!

Advertisements

How do you chain commands in BASH or CSH?

I often need to run several commands which can take an hour or more and I won’t necessarily be present the entire time. It is nice to be able to run a command, and if it is successful, then run another; However, if one command fails, then stop the flow.  This article will tell you how to do this on the command line.

Chaining On Success

Perhaps you want to build, and then test. It would be annoying if the tests ran even if the build failed.

To chain the commands so subsequent commands will only run if the preceding command was successful (returned a zero):

make && run_test && echo "SUCCESS"

Chaining On Failure

To run a command followed by another command, only if the first command failed, then use:

make || echo "BUILD FAILED!"

You can also combine them:

(sh -c "exit 0" && sh -c "exit 1") && echo "SUCCESS" || echo "FAIL"

With Email

This becomes very powerful if you send yourself an email with the results so you know when a run completes and the result, e.g.

(sh -c "exit 0" && sh -c "exit 1") && mailx -s "Build: SUCCESS" $USER < /dev/null || mailx -s "Build: FAIL" $USER < /dev/null

Chaining Regardless

To run a command followed by another command, regardless of the return code, then use:

make; echo "BUILD DONE - make returned code: $?"

How does this work?

It is easiest to think of this all simply as Boolean logic. Boolean comparisons will only evaluate until the result can be determined.

On Linux a return code from a command of “0” means success. All other return codes are regarded as failures. This can be confusing because normal Boolean logic uses a 1 for true, e.g.

1 && 1 == 1, Evaluates the first and second expression
1 && 0 == 0, Evaluates the first and second expression
0 && 1 == 0, Only evaluates the first expression
0 && 0 == 0, Only evaluates the first expression

However, for our return codes this looks like:

exit 0 && exit 0 == Success, Executes the first and second command
exit 0 && exit 1 == Fail, Executes the first and second command
exit 1 && exit 0 == Fail, Only executes the first command
exit 1 && exit 1 == Fail, Only executes the first command

We can use a similar example for OR:

1 || 1 == 1, Only evaluates the first expression
1 || 0 == 1, Only evaluates the first expression
0 || 1 == 1, Evaluates the first and second expression
0 || 0 == 0, Evaluates the first and second expression

With return codes this looks like:

exit 0 || exit 0 == Success, Only executes the first command
exit 0 || exit 1 == Success, Only executes the first command
exit 1 || exit 0 == Success, Executes the first and second command
exit 1 || exit 1 == Fail, Executes the first and second command

 

How to easily manipulate long paths in Linux

I found this little gem a while ago and a coworker identified it as useful too, so I’m posting it:

Let’s say you want to copy a file from a different directory to a new file in that same directory but don’t want to change to that directory, and the path is really long…

cp /my/very/long/path/to/{old_file,new_file}.cpp

This will copy /my/very/long/path/to/old_file.cpp to /my/very/long/path/to/new_file.cpp

The differences between Linux shells (namely BASH and TCSH)

Overview

The two most common shells in Linux are CSH and BASH. Shells are programs used to allow a user to enter and execute commands as well as to interpret entire files of commands called scripts. You can work with Graphical User Interfaces (GUIs) in Linux but the real power of Linux is that it was designed around Command Line Interfaces (CLIs) and thus you realize significant benefits when working with CLIs from a shell.

I would recommend you pick one of these shells and learn it, and that you learn it well.

The Korn shell (KSH) is another fairly popular shell which extends from the Bourne shell (sh), just as BASH does, but I won’t spend a lot of time talking about KSH here.  It is worth learning a little about KSH, you may like the fact that it has backward compatibility with sh and it has an interactive style similar to CSH.

C shell – CSH/TCSH

The C shell shares a lot of the flow control constructs as the C programming language and thus this shell is dubbed the “C shell”. The Tee-shell, Tee-cee-sheel, or Tee-cee-ess-aitch (TCSH) is really just CSH with command-line completion and other features, most people today who are using CSH are really using TCSH, yet only refer to their shell as CSH. For the remained of this post CSH will be used but it will be referring to TCSH as a whole.

Bourne again shell – BASH

The Bourne shell (SH) was the standard shell in older versions of UNIX (starting with the Seventh Edition). As part of the GNU project, the Bourne shell was re-implemented to provide interactive features and this newer version of the Bourne shell is called the Bourne again shell (BASH). Almost all UNIX based systems today are still delivered with SH or what is really BASH emulating SH. Of the popular shells, BASH/SH is the oldest and most widely used.

Because every UNIX based system is delivered with BASH you will undoubtedly have to use it if you are working on Linux for an extended period of time.

Regardless of your preferred shell, I would also recommend you learn the basics of BASH.

Interactive features of CSH and BASH

CSH

One benefit of CSH is that it has a truly interactive feature not found in BASH. For instance, the following lines are evaluated interactively (i.e. line-by-line) in CSH:

> if ( 1 ) then
>  echo true
true
> endif

BASH

While SH (Bourne Shell) does not support interactive mode supposedly BASH does. However, this might behave differently than you would expect, as BASH commands are evaluated in logical blocks.

There are some subtle differences between CSH and BASH interactive modes. For example, BASH will evaluate blocks at the end whereas CSH will evaluate blocks as they are entered. Here is the same example as provided for CSH, but now provided for BASH:

$ if [ 1 ]; then
>  echo true
> fi
true

Both shells behave very similar, with one difference and that is: BASH evaluates in blocks of logic in the interactive mode while CSH does the evaluation line-by-line.

Variables

Shell Variables

BASH

In BASH you can set a shell variable using:

$ abc=123

BASH does not handle spaces very well:

$ abc = 123
BASH: abc: command not found

When we say a shell variable we mean that child processes will not inherit the variable.  This enables you to set variables and work with them locally, and to be sure that what you did won’t have some perverse effect on child processes that you start.

To demonstrate what a shell variable means, let’s create a run.sh script:

#!/bin/bash
echo switch:$switch

To execute this script, we would use (make sure it has executable permissions [chmod +x ./run.sh]):

$ ./run.sh
switch:

From here you will see that nothing is printed for the switch variable.

CSH

In CSH you can set a shell variable using:

> set abc=123

CSH trims spaces:

> set abc =   123
> echo "'${abc}'"
'123'

Let’s create a script to demonstrate some of the use cases.  Let’s call it run.csh:

#!/bin/csh
echo switch:$switch

To invoke this script, we would use (make sure it has executable permissions [chmod +x ./run.csh]):

> ./run.csh
switch: Undefined variable.

As you can see, for CSH an unset variable is an error condition.

Referencing Undefined Variables

BASH

Unset variables in BASH are allowed by default.  Just as you saw above with the run.csh script this is an error condition for CSH. Earlier we ran the run.sh script and that didn’t error even though it had very similar code. Here is that example again:

$ ./run.sh 
switch:

We can actually force BASH to report an error for undefined variables.  This is done using BASH set variables, and to make our scripts behave the same way we would do this:

#!/bin/bash
set -u
echo switch:$switch

Running this script again will produce an error now:

$ ./run.sh 
./run.sh: line 3: switch: unbound variable

CSH

As we already pointed out CSH will error, by default, if an undefined variable is referenced. Here is that example provided again:

> ./run.csh
switch: Undefined variable.

Environment Variables

BASH

From here you can run export to make this variable accessible to the child processes:

$ switch=on
$ export switch
$ ./run.sh
switch:on

$ echo test:$switch
test:on

You can simplify this by declaring, defining, and exporting the variable all in the same line:

$ export switch=off
$ ./run.sh 
switch:off

CSH

In CSH we use the setenv command to make environment variables visible to child processes. Here is an example of that syntax:

> setenv switch=on
> ./run.csh
switch:on

> echo test:$switch
test:on

Per Command Variables

BASH

With BASH we can actually set this variable on a per command line basis:

$ switch=on ./run.sh
switch:on

$ echo test:$switch
test:

Setting a variable on the same line as a command, in BASH, will affect the child process. However, the local shell or script will not be impacted (as shown above with ‘test:’).

Furthermore, we can also run this on multiple lines and see different results than when set on the same line as a command:

$ switch=on
$ ./run.sh
switch:

$ echo test:$switch
test:on

In the script you can see that the variable shows as unset, even though in the parent shell we have set it. This is what is meant by a shell variable.

CSH

CSH doesn’t have as convenient of a way as BASH does for setting a variable on a per command line basis, but you can still do this using the concept of subshells:

> (setenv switch on; ./run.csh)
switch:on

> echo test:$switch
test:

Setting a variable in a subshell will not impact the parent shell or script.

However, we can also run this on multiple lines with different results:

> switch=on
> ./run.sh
switch:

> echo test:$switch
test:on

In the script you can see that the variable shows as unset, even though in the parent shell we have set it. This is what is meant by a ‘local’ variable.

Unset a Variable

BASH

Once a variable is set you can unset and unexport it with the unset command:

$ export switch=off
$ unset switch
$ ./run.sh 
switch:

Even if you set the variable again, it is no longer exported after an unset:

$ export switch=off
$ unset switch
$ switch=on
$ ./run.sh 
switch:

However, it is important to note that once a variable is exported changes to it’s value will impact new child processes:

$ export switch=off
$ ./run.sh 
switch:off
$ switch=on
$ ./run.sh 
switch:on

CSH

Before we talk about how to unset a variable in CSH we should talk about how variables are managed in CSH. The shell variables and the environment variables are handled in completely different ways and these different mechanisms can be said to be independent of each-other. Furthermore, it is only the referencing of these variables that is the same.

What is meant by independent mechanisms? The shell variables can be set and unset independent of environment variables. Once a variable is set you can unset it with the unset or unsetenv command.  Here’s an example:

> setenv switch on
> echo envvar:$switch
envvar:on
> set switch=off
> echo shell:$switch
shell:off
> unset switch
> echo envvar:$switch
envvar:on
> unsetenv switch
> echo $switch
switch: Undefined variable.

appending $(path 123)

direct access

Variable interpretation

Both shells will evaluate non-quoted variable expressions

Here is an example in CSH:

> set test = "value"
> echo $test
value

Here is an equivalent example is BASH:

$ test="value"
$ echo $test
value

Both shells will not evaluate single-quoted variable expressions

Here is an example in CSH:

> set test = "value"
> echo '$test'
$test

Here is an equivalent example is BASH:

$ test="value"
$ echo '$test'
$test

Both shells will evaluate double-quoted variable expressions

Here is an example in CSH:

> set test = "value"
> echo "$test"
value

Here is an equivalent example is BASH:

$ test="value"
$ echo "$test"
value

Things start to differ when escaping the dollar sign ($) in a double quotes use case. Here is an example script, test.sh, script showing how to escape variable evaluation in BASH:

#!/bin/bash
#test
check="test"
grep "${check}$" $0

This script should search (using grep) this file ($0 is this file) for lines that end with “test” (in regex $ signifies the end-of-line), and this does work as expected. Running this script finds the ‘#test’ line as expected:

$ ./test.sh
#test

However, there are some nuances with CSH. For example, this script (test.csh), looks like it should work:

#!/bin/csh
#test
set check = "test"
grep "${check}$" $0

This actually produces an error:

> ./test.csh
Variable name must contain alphanumeric characters.

In CSH this should actually be executed as (quotes are removed from grep):

#!/bin/csh
#test
set check = "test"
grep ${check}$ $0

This will work as expected:

> ./test.csh
#test

It is nearly impossible to escape a dollar sign ($) inside of double quotes when using CSH; However, single quotes will prevent expansion, or just using variables without quotes enables escape sequences to solve the problem.

Syntax checking in BASH

Syntax can be checked in BASH with:

bash -n script.sh

This syntax checking process is nice as it does not execute any of the commands, and just checks the entire script for syntax.  This is a quick and easy way to validate all logical branches in a script, for syntax. If you forget to add a semicolon after a square bracket in an if statement, such as this test.sh file:

#/bin/bash
one=1
if [ $one ] then
  echo test
fi

Then the following syntax error will be reported:

./test.sh: line 5: syntax error near unexpected token `fi'
./test.sh: line 5: `fi'

However, a simple syntax error such as adding spaces around the equal sign in a variable assignment will not be caught:

#/bin/bash
one = 1
if [ $one ]; then
  echo test
fi

This is because the syntax checker doesn’t considered this a syntax error. The BASH syntax checker will think that one is a command.  This script will error at run-time with the following error:

./test.sh: line 2: one: command not found

Bottom-line: The syntax checker in BASH is nice and should be used to check syntax of BASH scripts, but be aware that the syntax checker will not catch everything.  Unfortunately, CSH doesn’t have an equivalent.

Startup scripts

BASH

The following scripts are used in BASH…

Login

  • /etc/profile (system)
  • /etc/bash.bashrc (system)
  • ~/.bash_profile
  • ~/.bash_login
  • ~/.profile
  • ~/.bashrc

Opening a shell

  • ~/.bashrc

Running a script

  • /etc/bash.bashrc (system)

CSH

The following scripts are used in CSH…

Login

  • /etc/CSH.cshrc (system)
  • /etc/CSH.login (system)
  • ~/.tcshrc
  • ~/.cshrc
  • ~/.history
  • ~/.login
  • ~/.cshdirs

Opening a shell

  • /etc/CSH.cshrc (system)
  • ~/.tcshrc
  • ~/.cshrc

Running a script

  • ~/.cshrc

Prevent loading the .cshrc file

The .cshrc file is a file where you can specify CSH commands to be executed every-time you load CSH. However, sometimes you want default CSH (i.e. that does not have your custom commands run) and this can be done with the -f switch.  This can, and should, be set in all CSH scripts as it helps to ensure that you are not depending on settings in your current environment:

#!/bin/csh -f
env

This script will show you what your environment variables looks like without your .cshrc file being sourced. You can also use the -f switch when opening a new shell:

> csh -f

Prevent loading the .bashrc and .bash_profile files

The .bashrc and .bash_profile files are places where you can specify custom commands to be executed every-time you start a BASH shell or login, respectively. However, sometimes you want default BASH (i.e. that does not have your custom commands run) and this can be done with the –norc and –noprofile switches.  These arguments can only be specified on the command line:

$ bash --norc --noprofile

Running with a clean environment

Have you ever been stuck seeing strange behavior that nobody else on your project is seeing? Have a sneaking suspicion that your environment is messed up? In swoops the ‘env’ command to save the day!

Running with ‘env -i <command>’ will prevent your environment variables from being inherited from child processes. When used in conjunction with switches to prevent shell scripts from running, you can be pretty confident that you have a clean environment.

BASH

To run clean:

$ env -i bash --norc --noprofile

CSH

To run clean:

> env -i csh -f

PATH and path

Both BASH and CSH shells have and use the $PATH environment variable. However, CSH has a variable called $path which can be manipulated in a slightly different way. Generally the use case of the PATH environment variable is so that applications can find executables and this is done by adding executable directories to the PATH environment variable. CSH gives us a direct and convenient way to do this:

set path = ($path /usr/local/bin)

The lower case path variable enables direct access indexes:

> echo $path[1]
/home/pi/bin

Redirects catching just STDERR, STDOUT, or both

BASH

Let’s start with a script that prints a message to both STDERR and STDOUT:

#!/bin/bash
echo "message intended for stdout"
(&amp;amp;amp;amp;amp;amp;gt;&amp;amp;amp;amp;amp;amp;amp;2 echo "message intended for stderr")

Now if we call this script this is what we’ll see:

$ ./std.sh 
message intended for stdout
message intended for stderr

In BASH to redirect STDOUT only:

$ ./std.sh > out.log
message intended for stderr
$ cat out.log
message intended for stdout

In BASH to redirect STDERR only:

$ ./std.sh 2> err.log
message intended for stdout
$ cat err.log
message intended for stderr

In BASH to redirect both:

$ ./std.sh >& std.log
$ cat std.log
message intended for stdout
message intended for stderr

In BASH to redirect STDOUT to one file and STDERR to another:

$ ( ./std.sh 2> err.log ) > out.log
$ cat err.log
message intended for stderr
$ cat out.log
message intended for stdout

CSH

We will use the same script provided above (BASH) that prints a message to both STDERR and STDOUT, but we will invoke this script from CSH and manipulate the STDERR and STDOUT streams:

#!/bin/csh
echo "message intended for stdout"
(>&2 echo "message intended for stderr")

Now if we call this script this is what we’ll see (remember we are in CSH):

> ./std.sh
message intended for stdout
message intended for stderr

In CSH to redirect STDOUT only:

> ./std.sh > out.log
message intended for stderr
> cat out.log
message intended for stdout

In CSH to redirect STDERR only:

$ ( ./std.sh > /dev/tty ) > & err.log
message intended for stdout
$ cat err.log
message intended for stderr

In CSH to redirect both:

$ ./std.sh > & std.log
$ cat std.log
message intended for stdout
message intended for stderr

In CSH to redirect STDOUT to one file and STDERR to another:

$ ( ./std.sh > out.log ) > & err.log
$ cat err.log
message intended for stderr
$ cat out.log
message intended for stdout