Adding Help & Option in Bash Script

Abhishek Ghosh

By Abhishek Ghosh August 18, 2015 8:48 pm Updated on August 18, 2015

Adding Help & Option in Bash Script

Many New Hobbyist Programmers and Students Write Bash Scripts. Adding Help & Option in Bash Script is Easy. Here is a Basic Guide With Examples. Advanced guide will need dedicated website or books. Previously, we have talked about bash script and simple example of such script can be bash script for counting the number of balls easily for a pyramid made of balls.

Adding Help & Option in Bash Script : Basic Theory

There is a shell tool getopt. The GNU implementation of getopt(3) (used by the command-line getopt(1) on GNU/Linux) supports parsing long options. But the BSD implementation of getopt (e.g. on Mac OS X) does not support parsing long options.

getopt and getopts are different. getopts is a built-in command to bash to process command-line options in a loop and assign each found option and value in turn to built-in variables, so that we can further process them. getopt is an external utility program and it does not actually process our options for us the way that getopts does, as well as the Perl Getopt module or the Python optparse/argparse modules do. getopt canonicalize the options that are passed in -- that, is convert them to a more standard form, so that it becomes easier for a shell script to process them.

So for execution of commands in this way :

./shell.sh -c abc.pl /tmp/

1	./shell.sh -c abc.pl /tmp/

it is easier and avoiding long words like --copyfile instead of -c shown above. Now, suppose this is our script’s main content :

#!/bin/bash
# main starts
echo -n "Hello! $USER, today is "
date +%A
# main ends

1

2

3

4

5

#!/bin/bash

# main starts

echo -n "Hello! $USER, today is "

date +%A

# main ends

if we save it as bash-script.sh, to execute it, we will normally do these :

chmod +x bash-script.sh && ./bash-script.sh
# or
chmod +x bash-script.sh && sh bash-script.sh

1

2

3

chmod +x bash-script.sh && ./bash-script.sh

# or

chmod +x bash-script.sh && sh bash-script.sh

Adding Help & Option in Bash Script : Add Option

But, we want to supply the name as option, then print the date with this format of command :

---

./bash-script.sh -a Abhishek

1	./bash-script.sh -a Abhishek

so that, we will get this output :

Hello! Abhishek, today is Wednesday

1	Hello! Abhishek, today is Wednesday

Quite practical example. In this case, we need to write the script in this way (tested on OS X as working, report us if you face error) :

#!/bin/bash

# A string with command options
options=$@

# An array with all the arguments
arguments=($options)

# Loop index
index=0

for argument in $options
  do
    # Incrementing index
    index=`expr $index + 1`

    # The conditions
    case $argument in
      -a) echo "Hello! ${arguments[index]}, today is `date +%A`";;
      -abc) echo "Hello! ${arguments[index]}, today is $(date)" ;;
    esac
  done

exit;

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

#!/bin/bash

# A string with command options

options=$@

# An array with all the arguments

arguments=($options)

# Loop index

index=0

for argument in $options

do

# Incrementing index

index=`expr $index + 1`

# The conditions

case $argument in

-a) echo "Hello! ${arguments[index]}, today is `date +%A`";;

-abc) echo "Hello! ${arguments[index]}, today is $(date)" ;;

esac

done

exit;

very basic example.

Adding Help & Option in Bash Script : Add Help

Easiest example is :

#!/bin/bash

if [ "$1" == "-h" ] ; then
    echo "Usage 1: ./ `basename $0` -a <anyname>"
    echo "Usage 2: sh `basename $0` -a <anyname>"
    echo "This Help File: sh `basename $0` -h"
    exit 0
fi

# A string with command options
options=$@

# An array with all the arguments
arguments=($options)

# Loop index
index=0

for argument in $options
  do
    # Incrementing index
    index=`expr $index + 1`

    # The conditions
    case $argument in
      -a) echo "Hello! ${arguments[index]}, today is `date +%A`";;
      -abc) echo "Hello! ${arguments[index]}, today is $(date)" ;;
    esac
  done

exit;

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

#!/bin/bash

if [ "$1" == "-h" ] ; then

echo "Usage 1: ./ `basename $0` -a <anyname>"

echo "Usage 2: sh `basename $0` -a <anyname>"

echo "This Help File: sh `basename $0` -h"

exit 0

fi

# A string with command options

options=$@

# An array with all the arguments

arguments=($options)

# Loop index

index=0

for argument in $options

do

# Incrementing index

index=`expr $index + 1`

# The conditions

case $argument in

-a) echo "Hello! ${arguments[index]}, today is `date +%A`";;

-abc) echo "Hello! ${arguments[index]}, today is $(date)" ;;

esac

done

exit;

If we run :

./bash-script.sh -h

1	./bash-script.sh -h

then we will get the help :

Usage 1: ./ bash-script.sh -a <anyname>
Usage 2: sh bash-script.sh -a <anyname>
This Help File: sh bash-script.sh -h

1

2

3

Usage 1: ./ bash-script.sh -a <anyname>

Usage 2: sh bash-script.sh -a <anyname>

This Help File: sh bash-script.sh -h

But it has problem. It can not show anything if only ./bash-script.sh is executed. That is why we need to re-write in this way :

#!/bin/bash

usage="$(basename "$0") -- Program to demonstrate the help to life or bash, she bang, she is Kate, she fck, she is sh. She indeed is GNU/Linux.

where:
    -h  show this help text
    -s  set the seed value (default: 42)"
        
seed=42
while getopts ':hs:' option; do
  case "$option" in
    h) echo "$usage"
       exit
       ;;
    s) seed=$OPTARG
       ;;
    :) printf "missing argument for -%s\n" "$OPTARG" >&2
       echo "$usage" >&2
       exit 1
       ;;
   \?) printf "illegal option: -%s\n" "$OPTARG" >&2
       echo "$usage" >&2
       exit 1
       ;;
  esac
done
shift $((OPTIND - 1))

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

#!/bin/bash

usage="$(basename "$0") -- Program to demonstrate the help to life or bash, she bang, she is Kate, she fck, she is sh. She indeed is GNU/Linux.

where:

-h show this help text

-s set the seed value (default: 42)"

seed=42

while getopts ':hs:' option; do

case "$option" in

h) echo "$usage"

exit

;;

s) seed=$OPTARG

;;

:) printf "missing argument for -%s\n" "$OPTARG" >&2

echo "$usage" >&2

exit 1

;;

\?) printf "illegal option: -%s\n" "$OPTARG" >&2

echo "$usage" >&2

exit 1

;;

esac

done

shift $((OPTIND - 1))

Real life example is not going to be easy. It is difficult because in real we need to authenticate, then define path and option , set errors, on error again the help will show. It is difficult as there will be minimum 2 arguments, logical flow for failure. If we do not set the flow rightly, we will not get any output! Easiest logical flow can be studied by saving this fully working example as supload.sh and executing it. We need to run command in this way :

/supload.sh -u <username> -k <password> <dest_dir> <src_path>

1	/supload.sh -u <username> -k <password> <dest_dir> <src_path>

This is designed to work on OpenStack Swift or rather OpenStack Object Storage where API based authentication is allowed. With valid parameter on HP Cloud, it will print out the HTTP error. For dry run, it will throw the help options :

#!/bin/bash
#
########### Cloud Storage Uploader #################
#
# Script for upload files to cloud storage supported
# Cloud Files API (such as OpenStack Swift).
# License: GPL-3
#
####################################################
set -o noglob


usage() {
    cat <<EOF
Usage:
    supload.sh [-a AUTH_URL] -u <USER> -k <KEY> [-r] [[-e PATTERN]...] [options] <dest_dir> <src_path>

Options:
    -a AUTH_URL    authentication url (default: https://region-a.geo-1.identity.hpcloudsvc.com:35357/v2.0/)
    -u USER        user name
    -k KEY         user password
    -r             recursive upload
    -M             force upload without check by md5 sum
    -e PATTERN     exclude files by pattern (shell pattern syntax, ex. .git/*)
    -d NUM<m:h:d>  auto delete file in storage after NUM minutes or hours or days (ex. 7d)
    -s NUM<K:M:G>  specify the maximum transfer rate you want use to upload (ex. 1M)
    -m FILTER      add MTIME filter. Usefull to upload only new files in large directory (find -mtime syntax, ex. -1)
    -z FORMAT      Treat file as archive of a given type and extract it after upload. Supported formats: tar, tar.gz, tar.bz2
    -c             enable detect mime type for file and set content-type for uploading file (usually the storage can do it self)
    -q             quiet mode (error output only)

Params:
     <dest_dir>    destination directory or container in storage (ex. container/dir1/), not a file name
     <src_path>    source file or directory
EOF
}


# Defaults
AUTH_URL="https://region-a.geo-1.identity.hpcloudsvc.com:35357/v2.0/"
RECURSIVEMODE=""
USER=""
KEY=""
DEST_DIR=""
SRC_PATH=""
MD5CHECK="1"
EXPIRE=""
_ttlsec=""
QUIETMODE="0"
DETECT_MIMETYPE="0"
MTIME=""
SPEED=""
EXTRACT_ARCHIVE=""
declare -a EXCLUDE_LIST

# Utils
CURL="`which curl`"
CURLOPTS="--http1.0 --insecure"

FILEEX=`which file`
MD5SUM=`which md5sum`
if [ -z "$MD5SUM" ]; then
    MD5SUM=`which md5`
    if [ -n "$MD5SUM" ]; then
        MD5SUM="$MD5SUM -r"
    fi
fi

# check utils
if [ -z "$CURL" ]; then
    echo "[!] To use this script you need to install util 'curl'"
    exit 1
fi
if [ -z "$FILEEX" ]; then
    echo "[~] Util 'file' not found, detection mime type will be skipped"
    DETECT_MIMETYPE="0"
fi
if [ -z "$MD5SUM" ]; then
    echo "[!] To use this script you need to install util 'md5sum' or 'md5'"
    exit 1
fi

i=0
_agrs=()
for arg in "$@"; do
    _agrs[$i]="$arg"
    i=$((i + 1))
done

while getopts ":ra:u:k:d:Mqe:c:m:s:z:" Option; do
    case $Option in
            r ) RECURSIVEMODE="1";;
            a ) AUTH_URL="$OPTARG";;
            u ) USER="$OPTARG";;
            k ) KEY="$OPTARG";;
            M ) MD5CHECK="0";;
            d ) EXPIRE="$OPTARG";;
            q ) QUIETMODE="1";;
            e ) EXCLUDE_LIST=( "${EXCLUDE_LIST[@]}" "$OPTARG" );;
            c ) DETECT_MIMETYPE="1";;
            m ) MTIME="$OPTARG";;
            s ) SPEED="$OPTARG";;
            z ) EXTRACT_ARCHIVE="$OPTARG";;
            * ) echo "[!] Invalid option" && usage && exit 1;;
    esac
done
shift $(($OPTIND - 1))

# Hide password key
if [ -n "$SELECTEL_STORAGE_PWD" ]; then
    KEY="$SELECTEL_STORAGE_PWD"
    export -n SELECTEL_STORAGE_PWD  # unset
elif [ -n "$KEY" ]; then
    export SELECTEL_STORAGE_PWD="$KEY"
    # reexec and hide password key
    exec $0 "${_agrs[@]/$KEY/*****}"
fi

if [[ -z "$USER" || -z "$KEY" || -z "$1"  || -z "$2" ]]; then
    usage
    exit 1
fi

if [ -n "$EXTRACT_ARCHIVE" ]; then
    case "$EXTRACT_ARCHIVE" in
        "tar") ;;
        "tar.gz") ;;
        "tar.bz2") ;;
        *) echo "[!] Invalid format for option -z" && exit 1;;
    esac

    if [ -n "$RECURSIVEMODE" ]; then
        echo "[!] Option -z doen't support recursive upload (-r)"
        exit 1
    fi

    DETECT_MIMETYPE=""
    MD5CHECK=""
fi

_expire_invalid() {
    echo "[!] Invalid value for option -d. Examples: 7d, 24h, 30m"
    usage
    exit 1
}

if [ -n "$EXPIRE" ]; then
    _e_val="${EXPIRE:0:${#EXPIRE}-1}"
    _e_spec="${EXPIRE: -1}"

    [ -z "$_e_val" ] && _expire_invalid
    (("$_e_val" >= 1)) || _expire_invalid

    case "$_e_spec" in
        "d")    let "_ttlsec = _e_val * 86400" ;;
        "h")    let "_ttlsec = _e_val * 3600" ;;
        "m")    let "_ttlsec = _e_val * 60" ;;
         *)     _expire_invalid ;;
    esac
fi

## helper for get abspath
canonical_readlink() {
  local filename

  cd `dirname "$1"`;
  filename=`basename "$1"`;
  if [ -h "$filename" ]; then
    canonical_readlink `readlink "$filename"`;
  else
    echo "`pwd -P`/$filename";
  fi
}

DEST_DIR="${1%%/}/" # ensure / in end
SRC_PATH=`canonical_readlink "$2"`
# remove /. and / in end
SRC_PATH="${SRC_PATH%/.}"
SRC_PATH="${SRC_PATH%/}"

## Print message
# params:
# * $1 - level: 0 - info, 1 - error, 2 - debug info
# * $2 - message
msg() {
    if [ "$1" == "0" ]; then
        if [ "$QUIETMODE" == "0" ]; then
            echo "$2"
            return
        fi
        return
    fi

    if [ "$1" == "1" ]; then
        echo "$2"
        return
    fi

    if [ "$1" == "2" ]; then
        echo "[DEBUG]:"
        echo "$2"
        echo "[^^^^^]"
        return
    fi
}


## Authentication request
#
# params:
# * $1 - auth url
# * $2 - user name
# * $3 - user password
#
# If authentication is successful the function sets environment variables:
# * STOR_URL - storage url (always with / in end)
# * AUTH_TOKEN - authentication token
# ret_codes:
# * 0 - successfully
# * 1 - failed
auth() {
    local temp_file
    local url
    local user
    local key
    local resp_status

    url="$1"
    user="$2"
    key="$3"

    temp_file=`mktemp /tmp/.supload.XXXXXX`
    ${CURL} ${CURLOPTS} -H "X-Auth-User: ${user}" -H "X-Auth-Key: ${key}" "${url}" -s -D "${temp_file}" 1> /dev/null

    resp_status=`cat "${temp_file}" | head -n1 | tr -d '\r'`
    resp_status="${resp_status#* }"
    if [ "$resp_status" == "403 Forbidden" ]; then
        echo "[!] Deny access, auth failed!"
        rm -f "${temp_file}"
        return 1
    fi

    STOR_URL=`cat "${temp_file}" | tr -d '\r' | awk -F': ' 'tolower($1) ~ /^x-storage-url$/ { print $2 }'`
    AUTH_TOKEN=`cat "${temp_file}" | tr -d '\r' | awk -F': ' 'tolower($1) ~ /^x-auth-token$/ { print $2 }'`

    if [[ -z "${STOR_URL}" || -z "${AUTH_TOKEN}" ]]; then
        echo "[!] Auth failed"
        cat "${temp_file}"
        rm -f "${temp_file}"
        return 1
    fi

    STOR_URL="${STOR_URL%%/}/"

    rm -f "${temp_file}"
}


## Url quoting
#
# params:
# * $1 - input string
#
# return: quote string
url_encode() {
    local encodedurl
    encodedurl="$1";

    encodedurl=`
        echo "$encodedurl" | hexdump -v -e '1/1 "%02x\t"' -e '1/1 "%_c\n"' |
        LANG=C awk '
            $1 == "20"                    { printf("%s",   "%20"); next }
            $1 ~  /0[adAD]/               {                      next } # strip newlines
            $2 ~  /^[a-zA-Z0-9.*()\/-]$/  { printf("%s",   $2);  next } # pass through what we can
                                          { printf("%%%s", $1)        } # take hex value of everything else
    '`

    echo "${encodedurl}"
}


## Request ETAG for file from storage
#
# params:
# * $1 - file url
#
# return: etag string or nothing
head_etag() {
    local temp_file
    local url
    local etag
    local resp_status

    temp_file=`mktemp /tmp/.supload.XXXXXX`
    url="$1"

    $CURL ${CURLOPTS} -H "X-Auth-Token: ${AUTH_TOKEN}" "${url}" -s -I -D "${temp_file}" 1> /dev/null

    resp_status=`cat "${temp_file}" | head -n1 | tr -d '\r'`
    resp_status="${resp_status#* }"
    if [ "$resp_status" == "403 Forbidden" ]; then
        rm -f "${temp_file}"
        echo ""
        return 2
    fi

    etag=`cat "${temp_file}" | egrep -i -w -o "etag: .+" | tr -d '\r' | tr '[:upper:]' '[:lower:]' | sed 's/etag: //g'`

    rm -f "${temp_file}"

    echo "$etag"
}


## Detect mime-type for local file
#
# params:
# * $1 - path to local file
#
# return: mime-type string or nothing
content_type() {
    if [[ x"$DETECT_MIMETYPE" == x"0" ]]; then
        echo ""
        return 0
    fi

    local file
    file=$1

    if [ -z "$FILEEX" ]; then
        echo ""
        return
    fi

    echo "`$FILEEX -b --mime "$file" | awk -F\; '{ print $1 }'`"
}


## Check for container existence
#
# params:
# * $1 - container name or path
#
# return: "ok" if container existence or error
check_container() {
    local url
    local temp_file
    local cont
    local status
    cont=`url_encode "${1%%/*}"`
    temp_file=`mktemp /tmp/.supload.XXXXXX`

    url="${STOR_URL}/${cont}"
    $CURL ${CURLOPTS} -H "X-Auth-Token:${AUTH_TOKEN}" "${url}" -s -I -D "${temp_file}" 1> /dev/null

    status=`cat "${temp_file}" | grep "204 No Content"`
    rm -f "${temp_file}"

    if [ -z "$status" ]; then
        echo "not exist"
    fi

    echo "ok"
}


## Upload file
#
# params:
# * $1 - destination path in stotage
# * $2 - local file path
# ret_codes:
# * 0 - successfully uploaded
# * 1 - upload failed
# * 2 - access denied
# * 3 - source file doesn't exist
# * 4 - can't calc file hash
# * 5 - file already uploaded
# * 6 - hash doesn't match
# * 7 - invalid request
# return: some info about uploaded file or error messages
_upload() {
    local temp_file
    local dest
    local dest_url
    local dest_file_url
    local src
    local filehash
    local etag
    local cont_type
    local header_etage
    local header_auto_delete
    local header_content_type
    local resp_status
    local rc
    local response

    dest="$1"
    src="$2"

    dest_url="${STOR_URL}`url_encode "$dest"`"
    dest_file_url="${STOR_URL}`url_encode "$dest${src##*/}"`"

    # check for local file existence
    if [[ ! -e "$src" || -d "$src" ]]; then
        return 3
    fi

    # check for file hash
    if [ "$MD5CHECK" == "1" ]; then
        # local file hash
        filehash=`${MD5SUM} "$src" | sed 's/ .*//g'`
        if [ -z "$filehash" ]; then
            return 5
        fi

        # compare file hash
        etag=`head_etag "$dest_file_url"`
        rc=$?
        if [ $rc -eq 2 ]; then
            return 2 # denied get ETAG from HEAD request
        fi

        if [ "z${filehash}" == "z${etag}" ] ; then
            return 5
        fi
    fi

    # mime-type
    cont_type=`content_type "$src"`
    if [ -n "$cont_type" ]; then
        header_content_type="-H Content-Type:$cont_type"
    fi

    # md5
    if [ "$MD5CHECK" == "1" ]; then
        header_etage="-H ETag:$filehash"
    fi
    # auto delete
    if [[ -n "$_ttlsec" ]]; then
        header_auto_delete="-H X-Delete-After:$_ttlsec"
    fi

    if [[ -n "$EXTRACT_ARCHIVE" ]]; then
        dest_url="${dest_url}?extract-archive=${EXTRACT_ARCHIVE}"
        header_content_type="-Hx-detect-content-type:true"
    fi

    opts="${CURLOPTS}"
    if [[ -n "$SPEED" ]]; then
        opts="${CURLOPTS} --limit-rate ${SPEED}"
    fi

    # uploading
    temp_file=`mktemp /tmp/.supload.XXXXXX`
    $CURL ${opts} -X PUT -H "X-Auth-Token: ${AUTH_TOKEN}" $header_content_type $header_etage $header_auto_delete "$dest_url" -g -T "$src" -s -D "$temp_file" 1> /dev/null
    response=`cat "${temp_file}"`
    rm -f "${temp_file}"

    resp_status=`echo "$response" | head -n1 | tr -d '\r'`
    resp_status="${resp_status#* }"

    # -- successful upload
    if [[ "$resp_status" == "201 Created" || "$resp_status" == "200 OK" ]]; then
        # get hash for uploaded file (from response)
        etag=`echo "$response" | egrep -i -w -o "etag: .+" | tr -d '\r' | tr '[:upper:]' '[:lower:]' | sed 's/etag: //g'`

        if [[ -n "$EXTRACT_ARCHIVE" ]]; then
            echo "Archive unpacked"
            return
        fi

        if [ -z "$etag" ]; then
            echo "$response"
            return 1
        fi

        if [ "$MD5CHECK" == "1" ]; then
            if [ "z$etag" != "z$filehash" ]; then
                echo "$response"
                return 6
            fi
        fi

        echo "ETag: $etag"
        return
    fi

    # -- handler error responses
    echo "$response"
    if [ "$resp_status" == "403 Forbidden" ]; then
        return 2
    fi
    if [ "$resp_status" == "401 Unauthorized" ]; then
        return 2
    fi
    if [ "$resp_status" == "400 Bad Request" ]; then
        return 7
    fi

    return 1
}


## Upload file (with attempt again if failed)
#
# params:
# * $1 - destination path in stotage
# * $2 - local file path
# ret_codes:
# * 0 - successfully
# * 1 - fail
upload() {
    local count
    local src
    local dst
    local need_reauth
    local out

    dst="$1"
    src="$2"
    need_reauth="0"

    count=0
    while [ 1 ]; do
            ((++count))
            if [ $count -gt 5 ]; then
                msg 1 "[!] Failed upload $src after $((count - 1)) attempts."
                return 1
            fi

            if [ "x$need_reauth" == "x1" ]; then
                auth "${AUTH_URL}" "${USER}" "${KEY}"
                rc=$?
                if [ $rc -eq 0 ]; then
                    need_reauth="0"
                else
                    sleep "$count"
                    continue
                fi
            fi

            msg 0 "[.] Uploading $src..."
            out=$(_upload "$dst" "$src")
            rc=$?

            if [ $rc -eq 0 ]; then
                msg 0 "[*] Uploaded OK! $out"
                return
            fi

            if [ $rc -eq 1 ]; then
                msg 1 "[!] Attempt failed, try uploading again"
                sleep "$count"
                continue
            fi

            if [ $rc -eq 2 ]; then
                msg 1 "[!] Access denied, try reauth and uploading again"
                sleep "$count"
                need_reauth="1"
                continue
            fi

            if [ $rc -eq 3 ]; then
                msg 1 "[!] Source file $src doesn't exist!"
                return 1
            fi

            if [ $rc -eq 4 ]; then
                msg 1 "[!] Error with calculate file hash, skip uploading $src"
                return 1
            fi

            if [ $rc -eq 5 ]; then
                msg 0 "[.] File already uploaded"
                return
            fi

            if [ $rc -eq 6 ]; then
                msg 1 "[!] Hash doesn't match after uploading"
                msg 2 "$out"
                sleep "$count"
                continue
            fi

            if [ $rc -eq 7 ]; then
                msg 1 "[!] Something is wrong with the upload request:"
                msg 2 "$out"
                return 1
            fi

            msg 1 "[!] Unknown error, failed upload $src"
            msg 2 "$out"
            return 1.
    done
}


## Main
main() {
    local rc
    local exc_opts

    auth "${AUTH_URL}" "${USER}" "${KEY}"
    rc=$?
    if [ $rc -ne 0 ]; then
        exit 1
    fi

    if [ "`check_container "${DEST_DIR}"`" != "ok" ]; then
        echo "[!] Container not exist"
        exit 1
    fi

    ## Single file uploading
    if [ "z${RECURSIVEMODE}" != "z1" ]; then
        upload "${DEST_DIR}" "${SRC_PATH}"
        rc=$?

        exit $rc
    fi

    ## Recursive uploading
    if [ ! -d "${SRC_PATH}" ]; then
        echo "[!] ${SRC_PATH} is not dir"
        exit 1
    fi

    for i in "${EXCLUDE_LIST[@]}"; do
        exc_opts="$exc_opts -not -wholename $SRC_PATH/$i"
    done

    opts=""
    if [[ -n "$MTIME" ]]; then
        opts="-mtime ${MTIME}"
    fi

    find "${SRC_PATH}" $opts -type f $exc_opts -print0 | while read -d $'\0' f
    do
        src=$f

        a="${f#$SRC_PATH}"
        a="${a%/*}"
        dest="${DEST_DIR}${a#/}"
        dest="${dest%%/}/"

        upload "$dest" "$src"
    done

    rc=$?
    if [ $rc -eq 0 ]; then
        echo "[*] All files uploaded"
        exit 0
    fi
}

main

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

30

31

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

50

51

52

53

54

55

56

57

58

59

60

61

62

63

64

65

66

67

68

69

70

71

72

73

74

75

76

77

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

155

156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

191

192

193

194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

215

216

217

218

219

220

221

222

223

224

225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

330

331

332

333

334

335

336

337

338

339

340

341

342

343

344

345

346

347

348

349

350

351

352

353

354

355

356

357

358

359

360

361

362

363

364

365

366

367

368

369

370

371

372

373

374

375

376

377

378

379

380

381

382

383

384

385

386

387

388

389

390

391

392

393

394

395

396

397

398

399

400

401

402

403

404

405

406

407

408

409

410

411

412

413

414

415

416

417

418

419

420

421

422

423

424

425

426

427

428

429

430

431

432

433

434

435

436

437

438

439

440

441

442

443

444

445

446

447

448

449

450

451

452

453

454

455

456

457

458

459

460

461

462

463

464

465

466

467

468

469

470

471

472

473

474

475

476

477

478

479

480

481

482

483

484

485

486

487

488

489

490

491

492

493

494

495

496

497

498

499

500

501

502

503

504

505

506

507

508

509

510

511

512

513

514

515

516

517

518

519

520

521

522

523

524

525

526

527

528

529

530

531

532

533

534

535

536

537

538

539

540

541

542

543

544

545

546

547

548

549

550

551

552

553

554

555

556

557

558

559

560

561

562

563

564

565

566

567

568

569

570

571

572

573

574

575

576

577

578

579

580

581

582

583

584

585

586

587

588

589

590

591

592

593

594

595

596

597

598

599

600

601

602

603

604

605

606

607

608

609

610

611

612

613

614

615

616

617

618

619

620

621

622

623

624

625

626

627

628

629

630

631

632

633

634

635

636

637

638

639

640

641

642

643

644

645

646

647

648

649

650

651

652

653

654

655

656

657

658

#!/bin/bash

#

########### Cloud Storage Uploader #################

#

# Script for upload files to cloud storage supported

# Cloud Files API (such as OpenStack Swift).

# License: GPL-3

#

####################################################

set -o noglob

usage() {

cat <<EOF

Usage:

supload.sh [-a AUTH_URL] -u <USER> -k <KEY> [-r] [[-e PATTERN]...] [options] <dest_dir> <src_path>

Options:

-a AUTH_URL authentication url (default: https://region-a.geo-1.identity.hpcloudsvc.com:35357/v2.0/)

-u USER user name

-k KEY user password

-r recursive upload

-M force upload without check by md5 sum

-e PATTERN exclude files by pattern (shell pattern syntax, ex. .git/*)

-d NUM<m:h:d> auto delete file in storage after NUM minutes or hours or days (ex. 7d)

-s NUM<K:M:G> specify the maximum transfer rate you want use to upload (ex. 1M)

-m FILTER add MTIME filter. Usefull to upload only new files in large directory (find -mtime syntax, ex. -1)

-z FORMAT Treat file as archive of a given type and extract it after upload. Supported formats: tar, tar.gz, tar.bz2

-c enable detect mime type for file and set content-type for uploading file (usually the storage can do it self)

-q quiet mode (error output only)

Params:

<dest_dir> destination directory or container in storage (ex. container/dir1/), not a file name

<src_path> source file or directory

EOF

}

# Defaults

AUTH_URL="https://region-a.geo-1.identity.hpcloudsvc.com:35357/v2.0/"

RECURSIVEMODE=""

USER=""

KEY=""

DEST_DIR=""

SRC_PATH=""

MD5CHECK="1"

EXPIRE=""

_ttlsec=""

QUIETMODE="0"

DETECT_MIMETYPE="0"

MTIME=""

SPEED=""

EXTRACT_ARCHIVE=""

declare -a EXCLUDE_LIST

# Utils

CURL="`which curl`"

CURLOPTS="--http1.0 --insecure"

FILEEX=`which file`

MD5SUM=`which md5sum`

if [ -z "$MD5SUM" ]; then

MD5SUM=`which md5`

if [ -n "$MD5SUM" ]; then

MD5SUM="$MD5SUM -r"

fi

# check utils

if [ -z "$CURL" ]; then

echo "[!] To use this script you need to install util 'curl'"

exit 1

fi

if [ -z "$FILEEX" ]; then

echo "[~] Util 'file' not found, detection mime type will be skipped"

DETECT_MIMETYPE="0"

fi

if [ -z "$MD5SUM" ]; then

echo "[!] To use this script you need to install util 'md5sum' or 'md5'"

exit 1

fi

i=0

_agrs=()

for arg in "$@"; do

_agrs[$i]="$arg"

i=$((i + 1))

done

while getopts ":ra:u:k:d:Mqe:c:m:s:z:" Option; do

case $Option in

r ) RECURSIVEMODE="1";;

a ) AUTH_URL="$OPTARG";;

u ) USER="$OPTARG";;

k ) KEY="$OPTARG";;

M ) MD5CHECK="0";;

d ) EXPIRE="$OPTARG";;

q ) QUIETMODE="1";;

e ) EXCLUDE_LIST=( "${EXCLUDE_LIST[@]}" "$OPTARG" );;

c ) DETECT_MIMETYPE="1";;

m ) MTIME="$OPTARG";;

s ) SPEED="$OPTARG";;

z ) EXTRACT_ARCHIVE="$OPTARG";;

* ) echo "[!] Invalid option" && usage && exit 1;;

esac

done

shift $(($OPTIND - 1))

# Hide password key

if [ -n "$SELECTEL_STORAGE_PWD" ]; then

KEY="$SELECTEL_STORAGE_PWD"

export -n SELECTEL_STORAGE_PWD # unset

elif [ -n "$KEY" ]; then

export SELECTEL_STORAGE_PWD="$KEY"

# reexec and hide password key

exec $0 "${_agrs[@]/$KEY/*****}"

fi

if [[ -z "$USER" || -z "$KEY" || -z "$1" || -z "$2" ]]; then

usage

exit 1

fi

if [ -n "$EXTRACT_ARCHIVE" ]; then

case "$EXTRACT_ARCHIVE" in

"tar") ;;

"tar.gz") ;;

"tar.bz2") ;;

*) echo "[!] Invalid format for option -z" && exit 1;;

esac

if [ -n "$RECURSIVEMODE" ]; then

echo "[!] Option -z doen't support recursive upload (-r)"

exit 1

fi

DETECT_MIMETYPE=""

MD5CHECK=""

fi

_expire_invalid() {

echo "[!] Invalid value for option -d. Examples: 7d, 24h, 30m"

usage

exit 1

}

if [ -n "$EXPIRE" ]; then

_e_val="${EXPIRE:0:${#EXPIRE}-1}"

_e_spec="${EXPIRE: -1}"

[ -z "$_e_val" ] && _expire_invalid

(("$_e_val" >= 1)) || _expire_invalid

case "$_e_spec" in

"d") let "_ttlsec = _e_val * 86400" ;;

"h") let "_ttlsec = _e_val * 3600" ;;

"m") let "_ttlsec = _e_val * 60" ;;

*) _expire_invalid ;;

esac

fi

## helper for get abspath

canonical_readlink() {

local filename

cd `dirname "$1"`;

filename=`basename "$1"`;

if [ -h "$filename" ]; then

canonical_readlink `readlink "$filename"`;

else

echo "`pwd -P`/$filename";

fi

}

DEST_DIR="${1%%/}/" # ensure / in end

SRC_PATH=`canonical_readlink "$2"`

# remove /. and / in end

SRC_PATH="${SRC_PATH%/.}"

SRC_PATH="${SRC_PATH%/}"

## Print message

# params:

# * $1 - level: 0 - info, 1 - error, 2 - debug info

# * $2 - message

msg() {

if [ "$1" == "0" ]; then

if [ "$QUIETMODE" == "0" ]; then

echo "$2"

return

fi

return

fi

if [ "$1" == "1" ]; then

echo "$2"

return

fi

if [ "$1" == "2" ]; then

echo "[DEBUG]:"

echo "$2"

echo "[^^^^^]"

return

fi

}

## Authentication request

#

# params:

# * $1 - auth url

# * $2 - user name

# * $3 - user password

#

# If authentication is successful the function sets environment variables:

# * STOR_URL - storage url (always with / in end)

# * AUTH_TOKEN - authentication token

# ret_codes:

# * 0 - successfully

# * 1 - failed

auth() {

local temp_file

local url

local user

local key

local resp_status

url="$1"

user="$2"

key="$3"

temp_file=`mktemp /tmp/.supload.XXXXXX`

${CURL} ${CURLOPTS} -H "X-Auth-User: ${user}" -H "X-Auth-Key: ${key}" "${url}" -s -D "${temp_file}" 1> /dev/null

resp_status=`cat "${temp_file}" | head -n1 | tr -d '\r'`

resp_status="${resp_status#* }"

if [ "$resp_status" == "403 Forbidden" ]; then

echo "[!] Deny access, auth failed!"

rm -f "${temp_file}"

return 1

fi

STOR_URL=`cat "${temp_file}" | tr -d '\r' | awk -F': ' 'tolower($1) ~ /^x-storage-url$/ { print $2 }'`

AUTH_TOKEN=`cat "${temp_file}" | tr -d '\r' | awk -F': ' 'tolower($1) ~ /^x-auth-token$/ { print $2 }'`

if [[ -z "${STOR_URL}" || -z "${AUTH_TOKEN}" ]]; then

echo "[!] Auth failed"

cat "${temp_file}"

rm -f "${temp_file}"

return 1

fi

STOR_URL="${STOR_URL%%/}/"

rm -f "${temp_file}"

}

## Url quoting

#

# params:

# * $1 - input string

#

# return: quote string

url_encode() {

local encodedurl

encodedurl="$1";

encodedurl=`

echo "$encodedurl" | hexdump -v -e '1/1 "%02x\t"' -e '1/1 "%_c\n"' |

LANG=C awk '

$1 == "20" { printf("%s", "%20"); next }

$1 ~ /0[adAD]/ { next } # strip newlines

$2 ~ /^[a-zA-Z0-9.*()\/-]$/ { printf("%s", $2); next } # pass through what we can

{ printf("%%%s", $1) } # take hex value of everything else

'`

echo "${encodedurl}"

}

## Request ETAG for file from storage

#

# params:

# * $1 - file url

#

# return: etag string or nothing

head_etag() {

local temp_file

local url

local etag

local resp_status

temp_file=`mktemp /tmp/.supload.XXXXXX`

url="$1"

$CURL ${CURLOPTS} -H "X-Auth-Token: ${AUTH_TOKEN}" "${url}" -s -I -D "${temp_file}" 1> /dev/null

resp_status=`cat "${temp_file}" | head -n1 | tr -d '\r'`

resp_status="${resp_status#* }"

if [ "$resp_status" == "403 Forbidden" ]; then

rm -f "${temp_file}"

echo ""

return 2

fi

etag=`cat "${temp_file}" | egrep -i -w -o "etag: .+" | tr -d '\r' | tr '[:upper:]' '[:lower:]' | sed 's/etag: //g'`

rm -f "${temp_file}"

echo "$etag"

}

## Detect mime-type for local file

#

# params:

# * $1 - path to local file

#

# return: mime-type string or nothing

content_type() {

if [[ x"$DETECT_MIMETYPE" == x"0" ]]; then

echo ""

return 0

fi

local file

file=$1

if [ -z "$FILEEX" ]; then

echo ""

return

fi

echo "`$FILEEX -b --mime "$file" | awk -F\; '{ print $1 }'`"

}

## Check for container existence

#

# params:

# * $1 - container name or path

#

# return: "ok" if container existence or error

check_container() {

local url

local temp_file

local cont

local status

cont=`url_encode "${1%%/*}"`

temp_file=`mktemp /tmp/.supload.XXXXXX`

url="${STOR_URL}/${cont}"

$CURL ${CURLOPTS} -H "X-Auth-Token:${AUTH_TOKEN}" "${url}" -s -I -D "${temp_file}" 1> /dev/null

status=`cat "${temp_file}" | grep "204 No Content"`

rm -f "${temp_file}"

if [ -z "$status" ]; then

echo "not exist"

fi

echo "ok"

}

## Upload file

#

# params:

# * $1 - destination path in stotage

# * $2 - local file path

# ret_codes:

# * 0 - successfully uploaded

# * 1 - upload failed

# * 2 - access denied

# * 3 - source file doesn't exist

# * 4 - can't calc file hash

# * 5 - file already uploaded

# * 6 - hash doesn't match

# * 7 - invalid request

# return: some info about uploaded file or error messages

_upload() {

local temp_file

local dest

local dest_url

local dest_file_url

local src

local filehash

local etag

local cont_type

local header_etage

local header_auto_delete

local header_content_type

local resp_status

local rc

local response

dest="$1"

src="$2"

dest_url="${STOR_URL}`url_encode "$dest"`"

dest_file_url="${STOR_URL}`url_encode "$dest${src##*/}"`"

# check for local file existence

if [[ ! -e "$src" || -d "$src" ]]; then

return 3

fi

# check for file hash

if [ "$MD5CHECK" == "1" ]; then

# local file hash

filehash=`${MD5SUM} "$src" | sed 's/ .*//g'`

if [ -z "$filehash" ]; then

return 5

fi

# compare file hash

etag=`head_etag "$dest_file_url"`

rc=$?

if [ $rc -eq 2 ]; then

return 2 # denied get ETAG from HEAD request

fi

if [ "z${filehash}" == "z${etag}" ] ; then

return 5

fi

# mime-type

cont_type=`content_type "$src"`

if [ -n "$cont_type" ]; then

header_content_type="-H Content-Type:$cont_type"

fi

# md5

if [ "$MD5CHECK" == "1" ]; then

header_etage="-H ETag:$filehash"

fi

# auto delete

if [[ -n "$_ttlsec" ]]; then

header_auto_delete="-H X-Delete-After:$_ttlsec"

fi

if [[ -n "$EXTRACT_ARCHIVE" ]]; then

dest_url="${dest_url}?extract-archive=${EXTRACT_ARCHIVE}"

header_content_type="-Hx-detect-content-type:true"

fi

opts="${CURLOPTS}"

if [[ -n "$SPEED" ]]; then

opts="${CURLOPTS} --limit-rate ${SPEED}"

fi

# uploading

temp_file=`mktemp /tmp/.supload.XXXXXX`

$CURL ${opts} -X PUT -H "X-Auth-Token: ${AUTH_TOKEN}" $header_content_type $header_etage $header_auto_delete "$dest_url" -g -T "$src" -s -D "$temp_file" 1> /dev/null

response=`cat "${temp_file}"`

rm -f "${temp_file}"

resp_status=`echo "$response" | head -n1 | tr -d '\r'`

resp_status="${resp_status#* }"

# -- successful upload

if [[ "$resp_status" == "201 Created" || "$resp_status" == "200 OK" ]]; then

# get hash for uploaded file (from response)

etag=`echo "$response" | egrep -i -w -o "etag: .+" | tr -d '\r' | tr '[:upper:]' '[:lower:]' | sed 's/etag: //g'`

if [[ -n "$EXTRACT_ARCHIVE" ]]; then

echo "Archive unpacked"

return

fi

if [ -z "$etag" ]; then

echo "$response"

return 1

fi

if [ "$MD5CHECK" == "1" ]; then

if [ "z$etag" != "z$filehash" ]; then

echo "$response"

return 6

fi

echo "ETag: $etag"

return

fi

# -- handler error responses

echo "$response"

if [ "$resp_status" == "403 Forbidden" ]; then

return 2

fi

if [ "$resp_status" == "401 Unauthorized" ]; then

return 2

fi

if [ "$resp_status" == "400 Bad Request" ]; then

return 7

fi

return 1

}

## Upload file (with attempt again if failed)

#

# params:

# * $1 - destination path in stotage

# * $2 - local file path

# ret_codes:

# * 0 - successfully

# * 1 - fail

upload() {

local count

local src

local dst

local need_reauth

local out

dst="$1"

src="$2"

need_reauth="0"

count=0

while [ 1 ]; do

((++count))

if [ $count -gt 5 ]; then

msg 1 "[!] Failed upload $src after $((count - 1)) attempts."

return 1

fi

if [ "x$need_reauth" == "x1" ]; then

auth "${AUTH_URL}" "${USER}" "${KEY}"

rc=$?

if [ $rc -eq 0 ]; then

need_reauth="0"

else

sleep "$count"

continue

fi

msg 0 "[.] Uploading $src..."

out=$(_upload "$dst" "$src")

rc=$?

if [ $rc -eq 0 ]; then

msg 0 "[*] Uploaded OK! $out"

return

fi

if [ $rc -eq 1 ]; then

msg 1 "[!] Attempt failed, try uploading again"

sleep "$count"

continue

fi

if [ $rc -eq 2 ]; then

msg 1 "[!] Access denied, try reauth and uploading again"

sleep "$count"

need_reauth="1"

continue

fi

if [ $rc -eq 3 ]; then

msg 1 "[!] Source file $src doesn't exist!"

return 1

fi

if [ $rc -eq 4 ]; then

msg 1 "[!] Error with calculate file hash, skip uploading $src"

return 1

fi

if [ $rc -eq 5 ]; then

msg 0 "[.] File already uploaded"

return

fi

if [ $rc -eq 6 ]; then

msg 1 "[!] Hash doesn't match after uploading"

msg 2 "$out"

sleep "$count"

continue

fi

if [ $rc -eq 7 ]; then

msg 1 "[!] Something is wrong with the upload request:"

msg 2 "$out"

return 1

fi

msg 1 "[!] Unknown error, failed upload $src"

msg 2 "$out"

return 1.

done

}

## Main

main() {

local rc

local exc_opts

auth "${AUTH_URL}" "${USER}" "${KEY}"

rc=$?

if [ $rc -ne 0 ]; then

exit 1

fi

if [ "`check_container "${DEST_DIR}"`" != "ok" ]; then

echo "[!] Container not exist"

exit 1

fi

## Single file uploading

if [ "z${RECURSIVEMODE}" != "z1" ]; then

upload "${DEST_DIR}" "${SRC_PATH}"

rc=$?

exit $rc

fi

## Recursive uploading

if [ ! -d "${SRC_PATH}" ]; then

echo "[!] ${SRC_PATH} is not dir"

exit 1

fi

for i in "${EXCLUDE_LIST[@]}"; do

exc_opts="$exc_opts -not -wholename $SRC_PATH/$i"

done

opts=""

if [[ -n "$MTIME" ]]; then

opts="-mtime ${MTIME}"

fi

find "${SRC_PATH}" $opts -type f $exc_opts -print0 | while read -d $'\0' f

do

src=$f

a="${f#$SRC_PATH}"

a="${a%/*}"

dest="${DEST_DIR}${a#/}"

dest="${dest%%/}/"

upload "$dest" "$src"

done

rc=$?

if [ $rc -eq 0 ]; then

echo "[*] All files uploaded"

exit 0

fi

}

main

Tagged With file:///offsite/?token=434-512 , help option bash

Adding Help & Option in Bash Script

Adding Help & Option in Bash Script : Basic Theory

Adding Help & Option in Bash Script : Add Option

Adding Help & Option in Bash Script : Add Help

About Abhishek Ghosh

Here’s what we’ve got for you which might like :

Take The Conversation Further ...

Popular Articles

Adding Help & Option in Bash Script : Basic Theory

Adding Help & Option in Bash Script : Add Option

Adding Help & Option in Bash Script : Add Help

About Abhishek Ghosh

Here’s what we’ve got for you which might like :

Articles Related to Adding Help & Option in Bash Script

Take The Conversation Further ...