Environment Variables and some bash commands

To use librat, we need to have a passing awareness of some computer system settings called environment variables. We do this in this chapter, alongside a few other basic linux/unix commands that may be useful to know.

In practical terms, the important thing here is that you can generate the file `test/test_examples/init.sh <test/test_examples/init.sh>`__ and modify it to your needs. The rest you can skip for now, if you really want to. But you may well find yourself returning to this chapter when you want to ask more of your computer and of this tool.

Our focus will be on `bash <https://opensource.com/article/19/8/what-are-environment-variables>`__ environment variables.

This chapter is not generally critical for understanding librat but may help if you go into any details on your setup, or have problems.

The chapter covers:

  • Introduction to shell and environment variables

  • Some important environment variables and related

  • Important environment variables for librat

Introduction to shell and environment variables

export

An environment variable is one that is passed through from a shell to any child processes.

We can recognise these as they are usually defined in upper case (capital letters), and (in bash) defined with a export command: e.g.:

export MATLIB=test/test_examples

In this case, this would set the environment variable called MATLIB to test/test_examples. The syntax is:

export NAME=value

White space and single quotes '

If value has white space (gaps in the name), it will need quotes to contain the string, e.g.:

export SOMEHWERE='C:/Program Files'

Here, we contain the string C:/Program Files, which has white space, in single quotes ('). Its a good idea to avoid white space in filenames as they can cause problems. Use dash - or underscore _ instead.

echo

We can see the value a variable is set to with the command echo, and refer to the value of a variable with a $ symbol e.g.:

[1]:

%%bash

export MATLIB=test/test_examples
echo "MATLIB is set to $MATLIB"

MATLIB is set to test/test_examples

Note that there must be no gaps in NAME=value part of the statement. That is a typical thing for new users to get wrong and which can cause problems.

Double quotes " and backslash escape /

If you want to replace the value of a variable in a string, then you should generally use double quotes (") instead of single quotes ' as above:

[2]:

%%bash

export MATLIB=test/test_examples

echo '1. MATLIB is set to $MATLIB in single quotes'
echo "2. MATLIB is set to $MATLIB in double quotes"
echo "2. MATLIB is set to \$MATLIB in double quotes but with \ escaping the \$"

1. MATLIB is set to $MATLIB in single quotes
2. MATLIB is set to test/test_examples in double quotes
2. MATLIB is set to $MATLIB in double quotes but with \ escaping the $

However, we can also ‘escape’ the interpretation of the $ symbol in the double quoted string, with the backslash escape symbol \, as in example 3.

env, grep, pipe |

To see the values of all environment variables, type env (or printenv). Because this list can be quite long, we might want to select only certain lines from the list. One way to do this is to use the command `grep <https://en.wikipedia.org/wiki/Grep>`__, which searches for patterns in the each line:

[3]:

%%bash

export MATLIB=test/test_examples

env | grep M

TERM_PROGRAM=Apple_Terminal
TERM=xterm-color
TMPDIR=/var/folders/mp/9cxd5s793bjd4q3zng6dv_cw0000gn/T/
CONDA_PROMPT_MODIFIER=(librat)
TERM_PROGRAM_VERSION=433
GSETTINGS_SCHEMA_DIR_CONDA_BACKUP=
TERM_SESSION_ID=7FC61198-BCF5-4CED-8B68-076E150723FD
KERNEL_LAUNCH_TIMEOUT=40
MATLIB=test/test_examples
PATH=/Users/plewis/opt/anaconda3/envs/librat/bin:/Users/plewis/opt/anaconda3/condabin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Applications/VMware Fusion.app/Contents/Public:/opt/X11/bin:/Library/Apple/usr/bin
GSETTINGS_SCHEMA_DIR=/Users/plewis/opt/anaconda3/envs/librat/share/glib-2.0/schemas
MPLBACKEND=module://ipykernel.pylab.backend_inline
_CE_M=
XPC_SERVICE_NAME=0
HOME=/Users/plewis
LOGNAME=plewis

Here, we ‘pipe’ the output of the command env into the command grep with the pipe symbol |. grep M will filter only lines containing the character M. We see that this includes the variable MATLIB that we have set.

EXERCISE

1. Try removing the '| grep M' above to see the full list of environment variables.
2. Try some other 'grep' filters, such as filtering lines containing the string 'PATH'

Shell variable

A shell variable is one that is not passed through from a shell to any child processes. It is only relevant to the shell it is run in.

These are sometimes set as lower case variables (to distinguish from environment variables). The syntax is similar to that of the environment variable, but without the export. The syntax is:

name=value

for example:

[4]:

%%bash

hello="hello world $USER"
echo $hello

hello world plewis

set, head , tail

We can see the values of shell variables with the set command.

Like env, this is likely to produce a long list. We could filter as above, with grep, or here, we use tail to take the last N lines produced or head for the first N lines. The syntax is:

head -N
tail -N

[5]:

%%bash

echo '--------------------'
echo "1. The first 5 shell variables ..."
echo '--------------------'
set | head -5
echo

echo '--------------------'
echo "2. The last 5 shell variables ..."
echo '--------------------'
set | tail -5

--------------------
1. The first 5 shell variables ...
--------------------
BASH=/bin/bash
BASH_ARGC=()
BASH_ARGV=()
BASH_LINENO=()
BASH_SOURCE=()

--------------------
2. The last 5 shell variables ...
--------------------
XPC_SERVICE_NAME=0
_=--------------------
_CE_CONDA=
_CE_M=
__CF_USER_TEXT_ENCODING=0x1F5:0:2

Important environment variables for librat

cat <<EOF > output ... EOF

We can conveniently create files in bash from text in the bash shell. This is done using cat and defining a marker (often EOF, meaning End Of File), such as:

[27]:

%%bash

# change directory from docs/source up to root
cd ../..
BPMS=${BPMS-$(pwd)}

cat <<EOF > $BPMS/test/test_examples/first.obj
# My first object file
mtllib plants.matlib
usemtl white
v 0 0 0
v 0 0 1
plane -1 -2
!{
usemtl white
!{
v 0 0 1000
ell -1 30000 30000 1000
!}
!}
EOF

Let’s look at the file we have just created:

[28]:

%%bash

# change directory from docs/source up to root
cd ../..
BPMS=${BPMS-$(pwd)}

cat $BPMS/test/test_examples/first.obj

# My first object file
mtllib plants.matlib
usemtl white
v 0 0 0
v 0 0 1
plane -1 -2
!{
usemtl white
!{
v 0 0 1000
ell -1 30000 30000 1000
!}
!}

EXERCISE

Use the approach above (`cat <<EOF > output ... EOF`) to create your own text file, then check the contents are as you expected.

MATLIB, RSRLIB etc.

In librat, there is a considerable set of data that we need to describe world data for any particular simulation. For example, we need to have one or more object files giving the geometry, material files describing the spectral scattering properties of materials, sensor spectral response functions etc.

To try to make models and simulation scenarios portable, we want to avoid ‘hardwiring’ these file locations. One way to do that is to simply use relative file names throughout the description, so that we can then determine the full filenames from some core base directory.

If we happen to run the simulation from this directory, then clearly the relative filenames we use would directly describe all file locations.

However, if we run the simulation from elsewhere on the system, we need a mechanism to describe the base of the scene description files. More generally, we might want to store spectral response files in one part of the file system, and spectral scattering properties elsewhere. In that case, we need a set of base descriptors for these different types of file.

That is the file system philosophy used in librat. These base locations are defined by environment variables which we will describe below. Whilst you do not have to use these, it makes sense to set them up, even if they are all set to the same value (i.e. the base of the model files is the same for all file types).

The following environmental variables can be used:

Name

File types

MATLIB

material library e.g. `plants.matlib <test/test_examples/plants.matlib>`__, all materials defined in a material library e.g. `refl/white.dat <test/test_examples/refl/white.dat>`__

ARARAT_OBJECT

(extended) wavefront object files e.g. `first.obj <test/test_examples/first.obj>`__

DIRECT_ILLUMINATION

spectral files for direct illumination: those defined in -RATdirect command line option

RSRLIB

sensor waveband files: those defined in -RATsensor_wavebands command line option

BPMS_FILES

Not used

As noted, you can set all of these to the same value, in which case the database of files is all defined relative to that point. This is the most typical use of librat. We illustrate this setup below for the librat distribution, where a set of examples use files from the directory test/test_example.

Additionally, the following environment variables can be set to extend the size of some aspects of the model. You would only need to use these in some extreme case.

Name

Purpose

MAX_GROUPS

Maximum number of groups allowed (100000)

PRAT_MAX_MATERIALS

Maximum number of materials allowed (DEFAULT_PRAT_MAX_MATERIALS=1024 in mtllib.h)

 
[29]:

%%bash

# change directory from docs/source up to root
cd ../..
BPMS=${BPMS-$(pwd)}

# create the init.sh file we want
outfile=$BPMS/test/test_examples/init.sh

cat <<EOF > $outfile
#!/bin/bash
#
# preamble
#
BPMS=\${BPMS-\$(pwd)}
# set shell variables lib, bin, verbose
# with defaults in case not set
lib=\${lib-"\$BPMS/src"}
bin=\${bin-"\$BPMS/src"}
VERBOSE=\${VERBOSE-1}

# set up required environment variables for bash
export LD_LIBRARY_PATH="\${lib}:\${LD_LIBRARY_PATH}"
export DYLD_LIBRARY_PATH="\${lib}:\${DYLD_LIBRARY_PATH}"
export PATH="\${bin}:\${PATH}"

# set up required environment variables for librat
export TEST=\${BPMS}/test/test_example
export MATLIB=\$TEST
export RSRLIB=\$TEST
export ARARAT_OBJECT=\$TEST
export DIRECT_ILLUMINATION=\$TEST
export BPMS_FILES=\$TEST

if [ "\$(which start)" == "\${bin}/start" ]
then
  if [ "\$VERBOSE" ]; then
      echo "start found ok"
  fi
else
  # we should create them
  make clean all
fi
EOF

# set executable mode
chmod +x $outfile
# test run
$outfile

start found ok

Summary

In this chapter, we have covered a range of basic unix/linux and bash commands, so you should be able to navigate you way around a unix file system, and find your way back safely. Being familiar with these tools takes somne time of course, so you might now want to go on and take some other unix/linux course to see if you can deepen your understanding in that way. Alternatively, just spend some time exploring your system, looking to see what files are where, reading on the internet or help pages what they do, and so on.

Maybe thats wishful thinking on my part though. You may not feel you have time for basic unix at the moment … and we did say at the top of this chapter that it was not compulsory … I’d reccomend you do spend some time on unix … you’ll develop skills that willlast you a lifetime! ;-)

In practical terms, as we have said, the important thing here is that you can generate the file `test/test_examples/init.sh <test/test_examples/init.sh>`__ and modify it to your needs.