The Answer Guy

Embedding Newlines in Shell and Environment Values

From Mark Chitty on Mon, 08 May 2000

Hi Jim,

Thanks for the solution. I had gone down a different path but this has cleared up that little conundrum !! I seems obvious now that you point it out.

Your reply is much appreciated.

Oh yes, if you ever write a book let me know. I'll buy it !!

cheers, mark

Actually I have written one book --- it's _Linux_System_Administration_ (New Riders Publishing, 1999, with M Carling and Stephen Degler). (http://www.linuxsa.com).

However, I might have to write a new one on shell scripting. Oddly enough it seems to be a topic of growing interest despite the ubiquity of PERL, Python, and many other scripting languages.

In fact, one thing I'd love to do is learn enough Python to write a book that covers all three (comparatively). Python seems to be a very good language for learning/teaching programming. I've heard several people refer to Python as "executable psuedo-code."

Despite the availability of other scripting languages, the basic shell, AWK, and related tools are compelling. They are what we use when we work at the command line. Often enough we just want our scripts to "do what we would do manually" --- and then to add just a bit of logic and error checking around that.

Extra tidbit:

I recently found a quirky difference between Korn shell ('93) and bash. Consider the following:

echo foo | read bar; echo $bar

... whenever you see a "|" operator in a shell command sequence you should understand that there is implicitly a subshell (new process) that is created (forked) on one side of it or the other.

Of course other processes (including subshells) cannot affect the values of your shell variables. So the sequence above consists of three commands (echo the string "foo", read something and assign it to a shell variable named "bar", and echo the value of (read the $ dereferencing operator as "the value of") the shell named "bar"). It consists of two processes. One on one side of the pipe, and the other on the other side of the pipe. At the semicolon the shell waits for the completion of any programs and commands that precede it, and then continues with a new command sequence in the current shell.

The question becomes whether the subshell was created on the left or the right of the | in this command. In bash it is clearly created on the right. The 'read' command executes in a subshell. That then exits (thus "forgetting" its variable and environment heaps). Thus $bar is unaffected after the semicolon.

In ksh '93 and in zsh the subshell seems to be created to the left of the pipe. The 'read' command is executed in the current shell and thus the local value of "bar" is affected. Then the subsequent access to that shell variable does reflect the new value.

As far as I know the POSIX spec is silent on this point. It may even be that ksh '93 and zsh are in violation of the spec. If so, the spec is wrong!

It is very useful to be able to parse a set of command outputs into a local list of shell variables. Note that for a single variable this is easy:

bar=$(echo foo)

or:

bar=`echo foo`

... are equivalent expressions and they work just fine.

However, when we want to read the outputs into several values, and especially when we want to do so using the IFS environment value to parse these values then we have to resort of inordinate amounts of fussing in bash while ksh '93 and newer versions of zsh allow us to do something like:

grep ^joe /etc/passwd | IFS=":" read login pw uid gid gecos home sh

(Note the form: 'VAR=val cmd' as shown here is also a bit obscure but handy. The value of VAR is only affected for the duration of the following command --- thus saving us the trouble of saving the old IFS value, executing our 'read' command and restoring the IFS).

BTW: If you do need to save/restore something like IFS you must using proper quoting. For example:

OLDIFS="$IFS"
# MUST have double/soft quotes here!
IFS=:,
# do stuff parsing words on colons and commas
IFS="$OLDIFS"
# MUST also have double/soft quotes here!

Anyway, I would like to do some more teaching in the field of shell scripting. I also plan to get as good with C and Python as I currently am with 'sh'. That'll take at least another year or so, and a lot more practice!

Copyright © 2000, James T. Dennis
Published in The Linux Gazette Issue 54 June 2000

HTML transformation by Heather Stern of Tuxtops, Inc., http://www.tuxtops.com/

  1   2   3   4   5   6   7   8   9   10   11   12   13   14   15   16   17   18