UNIX shell

Index

General
  • Linux shell
  • explainshell
  • 10 More Funny and/or Useless Linux Commands
  • Comparison of command shells (wp)
    • bash
    • python
    • ...
  • Bash scripting FAQ
  • BASH Programming HOW-TO
    • functions
    • Exemples / Examples
      • my_func () {
            ...
        }
      • function my_func {
            ...
        }
      • function my_func () {
            ...
        }
      • # call to function
        my_func
      • a=$(my_func)
  • Entorn / Environment
    • set:
      • export TOTO="abc"
    • unset:
      • declare +x TOTO
    • list:
      • export
    • stored into files:
      • /etc/bashrc
      • /etc/profile
      • /etc/profile.d/toto.sh
        • export TOTO="toto"
      • ~/.bashrc
  • Prompt
  • Shebang
    • #!/bin/bash
    • exit 1 if some command inside the script does not return 0
      • any place in the script
        • #!/bin/bash -e # all commands executed anywhere in the script that return a value
          # different from 0 will make the whole script stop and return 1
      • in some parts of the script
        • #!/bin/bash

          # begin of critical section:
          set -e
          # all commands executed inside this section that return a value
          # different from 0 will make the whole script stop and return 1
          ...
          # end of critical section:
          set +e

          # non-critical calls: commands or functions executed from this
          # point will let the script continue to next commands, despite of
          # the returned value
          ...
  • Debug:
    • from command line:
      • bash -v toto.sh
      • bash -x toto.sh
    • inside the script:
      • #!/bin/bash -x
      • #!/bin/bash
        ...
        # start debug
        set -x
        ...
        # end debug
        set +x
  • $?
    		 returned value (get the exit code of each command in a pipe)
    $@
    		 all command options: $1 $2... (can be programatically set with set -- param1 param2 ...)
    $# number of parameters
    $0
    		 command
    $1
    		 first parameter
    $2
    		 second parameter
    ...
    ${@:2}
    		 from second parameter until the last one
    $$
    		 own pid
    $BASHPID
    		 own pid (like $$, but updated for subshells)
    $!
    		 pid of the latest comand
  • Get the returned value of the latest command:
  • Procés / Process
  • Seguretat / Security

Tests unitaris / Unit tests
  • Bats
    • TAP-compliant (Test Anything Protocol)
    • Info
    • Source
    • Instal·lació / Installation
      • Welcome to bats-core’s documentation!
      • using git submodules in myproject
        • dir structure:
          • my_project/
            • src/
              • myscript.sh
            • test/
              • bats/
              • test_helper/
                • bats-support/
                • bats-assert/
                • common-setup.bash
                  • #!/usr/bin/env bash

                    _common_setup() {
                        # when bats-assert is installed using nvm and npm: npm install -g bats-assert
                        load ${NVM_BIN}/../lib/node_modules/bats-support/load.bash
                        load ${NVM_BIN}/../lib/node_modules/bats-assert/load.bash
                        # get the containing directory of this file
                        # use $BATS_TEST_FILENAME instead of ${BASH_SOURCE[0]} or $0,
                        # as those will point to the bats executable's location or the preprocessed file respectively
                        PROJECT_ROOT="$( cd "$( dirname "$BATS_TEST_FILENAME" )/.." >/dev/null 2>&1 && pwd )"
                        # make executables in src/ visible to PATH
                        PATH="$PROJECT_ROOT/src:$PATH"
                    }
              • mytest.bats
                • setup() {
                      load 'test_helper/common-setup'
                      _common_setup
                  }

                  @test "My first test" {
                      run myscript.sh
                      assert_output 'my output from script'
                      # multiline
                      #assert_output --partial $'first line\nsecond line'
                      # do not forget double quotes, to print the line breaks
                      echo "${output}" >&3
                      echo "# my text!" >&3
                  }
        • my_project must be a git-controlled dir
        • then download git submodules
          • cd myproject
          • git submodule add https://github.com/bats-core/bats-core.git test/bats
          • git submodule add https://github.com/bats-core/bats-support.git test/test_helper/bats-support
          • git submodule add https://github.com/bats-core/bats-assert.git test/test_helper/bats-assert
        • ...
      • using npm
        • global-npm-bats-install-including-bats-assert.md
        • global
          • npm install -g bats
          • npm install -g bats-assert
          • npm install -g bats-file
          • # npm install -g bats-mock # from jasonkarns
          • ...
        • in project (as one of devDependencies in your package.json)
          • cd myproject
          • mkdir node_modules
          • npm install --save-dev bats
          • npm install --save-dev bats-assert
          • npm install --save-dev bats-file
          • # npm install --save-dev bats-mock # from jasonkarns
          • ...
      • Mageia
        • sudo dnf install bats
      • Alma / CentOS
        • sudo dnf install ...
    • Emacs
    • Exemples / Examples
      • setup block
        • #!/usr/bin/env bats

          setup_file() {
              # run once for all tests

              # get the containing directory of this file
              # use $BATS_TEST_FILENAME instead of ${BASH_SOURCE[0]} or $0,
              # as those will point to the bats executable's location or the preprocessed file respectively
              DIR="$( cd "$( dirname "$BATS_TEST_FILENAME" )" >/dev/null 2>&1 && pwd )"
              # make executables in ../ visible to PATH
              export PATH="$DIR/..:$PATH"
          }

          setup() {
              # run before every test

              # https://bats-core.readthedocs.io/en/stable/
              
              # modules installed with nvm and global npm:
              # npm install -g bats
              # npm install -g bats-support
              # npm install -g bats-assert
              load ${NVM_BIN}/../lib/node_modules/bats-support/load.bash
              load ${NVM_BIN}/../lib/node_modules/bats-assert/load.bash

          }

          teardown_file() {
              # run once after all tests
              :
          }

          teardown() {
              # run after every test
              :
          }
      • Test a script
        • @test "Test myscript.sh" {
              run myscript.sh ${parameter1}
          }
      • Test a script with debug
        • @test "Test myscript.sh" {
              run bash -x myscript.sh ${parameter1}
          }
      • Test a function in a library
        • @test "Test mylib.myfunction" {
              source mylib.sh
              run myfunction
          }
      • Test ...
    • Ús / Usage
      • debug
        • display output from script
          • echo "${output}" >&3
        • debug script
          • run bash -x toto.sh
            echo "${output}" >&3
      • run a single bats file
        • bats mytest.bats
      • run all bats files in a dir
        • bats test
      • print raw TAP format
        • bats -t mytest.bats
      • ...

Procés / Process
  • Get the process id of the latest command:
  • & (ampersand)
    • quan es crea un procés fill amb aquesta opció, s'inicia el fill, però torna de seguida el control al pare
    • a la jerarquia de processos, el fill continua penjant del pare
  • wait
    • espera que tots els processos fills hagin acabat (o els hagin matat)
    • si els fills s'engeguen amb & i no es fa un wait, quan el propi procés s'acabi, els fills passaran a ser orfes (penjaran del procés 1)
  • Senyals / Signals (POSIX.1-1990)
    • code
      		 name
      		 description
      		 generated by
      		 captured by "trap ... EXIT"



      • kill -<code>
      • kill -<name>
      • pkill -<name>
      		 others
      0
      		 EXIT

      • exit (in a script)
      • other signals (except KILL)

      1
      		 HUP
      		 Penjat

      		x
      2
      		 INT
      		 ^C
      		CTRL-C x
      3
      		 QUIT


      		-
      4
      		 ILL
      		 La instrucció no és permesa
      		 x
      		x
      5
      		 TRAP
      		 Trampa de traçat/punt d’aturada
      		 x
      		x
      6
      		 IOT
      		 Avortat
      		 x
      		x
      7
      		 BUS
      		 Error de bus
      		 x
      		x
      8
      		 FPE
      		 Excepció de coma flotant
      		 x
      		x
      9
      		 KILL
      		 Matat
      		 x
      		-
      10
      		 USR1
      		 Senyal 1 definit per l’usuari
      		 x
      		x
      11
      		 SEGV
      		 Violació de segment
      		 x
      		x
      12
      		 USR2
      		 Senyal 2 definit per l’usuari
      		 x
      		x
      13
      		 PIPE
      		 -
      		 x
      		x
      14
      		 ALRM
      		 Temporitzador
      		 x
      		x
      15
      		 TERM
      		 Terminat
      		 x
      		x
      16
      		 STKFLT
      		 Fallada de pila
      		 x
      		-
      17
      		 CHLD


      		-
      18
      		 CONT
      		 Continua un procés aturat amb STOP

      		-
      19
      		 STOP
      		 Aturat ()
      		 x
      		 CTRL-Z
      		 -
      20
      		 TSTP
      		 Aturat x
      		x
      21
      		 TTIN
      		 Aturat x
      		x
      22
      		 TTOU
      		 Aturat x
      		x
      23
      		 URG


      		-
      24
      		 GXCPU
      		 S’ha excedit el temps límit de CPU
      		 x
      		x
      25
      		 GXFSZ
      		 S’ha excedit la mida màxima de fitxer
      		 x
      		x
      26
      		 VTALRM
      		 Ha expirat el temporitzador virtual
      		 x
      		x
      27
      		 PROF
      		 El temps de perfilat ha expirat
      		 x
      		x
      28
      		 WINCH


      		-
      29
      		 IO
      		 L’operació d’E/S és possible
      		 x
      		x
      30
      		 PWR
      		 Fallada d’alimentació
      		 x
      		x
      31
      		 SYS
      		 La crida al sistema no és vàlida
      		 x
      		x
    • trap
    • kill
      • quan es mata un procés, els fills (orfes) passen a penjar del procés 1 (/sbin/init)
    • pkill
  • top
  • htop
  • Reinicia un procés si es mor / Restart a process if it dies (respawn)
  • Communication between processes
  • Exemples de processos jeràrquics:
    • ...

Opcions i arguments / Options and arguments

  • Altres llenguatges / Other languages
  • Check the number of arguments (*)
    • #!/bin/bash

      EXPECTED_ARGS=4
      if (( $# != $EXPECTED_ARGS ))
      then
          cat <<EOF
      Usage: `basename $0` arg1 arg2 arg3 arg4
      EOF
          exit 1
      fi
    • #!/bin/bash

      MIN_ARGS=2
      MAX_ARGS=5
      if (( $# < $MIN_ARGS )) || (( $# > $MAX_ARGS ))
      then    
      cat <<EOF
      Usage: `basename $0` arg1 arg2 [arg3] [arg4] [arg5]
      EOF
                exit 1
      fi
    • #!/bin/bash

      # this script does not allow any argument
      if (( $# != 0 ))
      then
          cat <<EOF
      Usage: `basename $0`
      EOF
          exit 1
      fi
    • #!/bin/bash
      MIN_ARGS=2
      MAX_ARGS=8
      #if [ $# -lt $MIN_ARGS ] || [ $# -gt $MAX_ARGS ]
      if (( $# < $MIN_ARGS )) || (( $# > $MAX_ARGS ))
      then
      cat <<EOF
      Usage: `basename $0` [-a|--option_a] [-b|--option_b option_value_b] [--option_c] [--option_d option_value_d] non_option_parameter_1 non_option_parameter_2 Options:
        -a, --option_a                   explanation for option_a
        -b, --option_b option_value_b    explanation for option_b
            --option_c                   explanation for option_c
            --option_d option_value_d    explanation for option_d Parameters:
        non_option_parameter_1           explanation for non_option_parameter_1
        non_option_parameter_2           explanation for non_option_parameter_2 EOF
          exit 1
      fi
    • #!/bin/bash

      # defaults
      verbose=0
      flag_a=0
      option_value_b="value_b"
      flag_c=0
      option_value_d="value_d"

      function print_help_and_exit {
          cat <<EOF
      Usage: `basename $0` [options] non_option_parameter_1 non_option_parameter_2
        - non_option_parameter_1        ...
        - non_option_parameter_1        ...

      Options:
        -h, --help                       print this help
        -v, --verbose                    verbose output (default: ${verbose})
        -a | --option_a                  ... (default: ${flag_a})
        -b | --option_b option_value_b   ... (default: ${option_value_b})
        --option_c                       ... (default: ${flag_c})
        --option_d option_value_d        ... (default: ${option_value_d})

      Examples:
      - `basename $0` -h
      - `basename $0` -a -b 3 --option_c --option_d value_d 4 5
      - `basename $0` -a -b 3 --option_c --option_d value_d -- 4 -5

      EOF
          exit 1
      }

      MIN_ARGS=2
      MAX_ARGS=9
      if (( $# < $MIN_ARGS )) || (( $# > $MAX_ARGS ))
      then
          print_help_and_exit
      fi


      # options
      if ! params=$(getopt -o hvab: --long help,verbose,option_a,option_b:,option_c,option_d: -n $0 -- "$@")
      then
          # invalid option
          print_help_and_exit
      fi
      eval set -- ${params}

      while true
      do
          case "$1" in
              -h | --help ) print_help_and_exit;;
              -v | --verbose ) verbose=1; shift ;;
              -a | --option_a ) flag_a=1; shift ;;
              -b | --option_b ) option_value_b="$2"; shift 2 ;;
              --option_c ) flag_c=1; shift ;;
              --option_d ) option_value_d="$2"; shift 2 ;;
              -- ) shift; break ;;
              * ) break ;;
          esac
      done

      # parameters
      non_option_parameter_1=$1
      non_option_parameter_2=$2

      # summary
      echo "Options:"
      echo "  verbose:        ${verbose}"
      echo "  flag_a:         ${flag_a}"
      echo "  option_value_b: ${option_value_b}"
      echo "  flag_c:         ${flag_c}"
      echo "  option_value_d: ${option_value_d}"
      echo "Parameters:"
      echo "  ${non_option_parameter_1}"
      echo "  ${non_option_parameter_2}"

      # do things
      (( verbose )) && echo "$(date '+%Y-%m-%dT%H:%M:%S.%06NZ' --utc) [$(basename $0)]..."
      # ...

      exit 0
    • shift
  • Opcions / Options
    • options and parameters can also be forcely separated by '--' (see rm, getopt)
    • Linux tip: Bash parameters and parameter expansions
    • Using getopts in bash shell script to get long and short command line options
    • tools
      • getopts
        • built-in
        • only single character options: -p -m m_value -d non_option_parameter_1 non_option_parameter_2
        • while getopts ":pm:d" optname
            do
              case "$optname" in
                "p")
                  echo "Option $optname is specified"
                  ;;
                "m")
                  echo "Option $optname has value $OPTARG"
                  ;;
                "d")
                  echo "Option $optname is specified"
                  ;;
                "?")
                  echo "Unknown option $OPTARG"
                  ;;
                ":")
                  echo "No argument value for option $OPTARG"
                  ;;
                *)
                # Should not occur
                  echo "Unknown error while processing options"
                  ;;
              esac
              echo "OPTIND is now $OPTIND"

              next_arguments=${@:$OPTIND}
            done
      • getopt
        • not built-in (provided by e.g. GNU)
          • OSX
            • Install brew
            • brew install gnu-getopt
            • brew link --force gnu-getopt
        • also accepts long option with double-dash prefix: -a|--option_a -b|--option_b option_value_b --option_c --option_d option_value_d non_option_parameter_1 non_option_parameter_2
        • options
          • -o shortops
            • when no short options are available, you must specify an empty set:
              • -o ''
          • -l, --longoptions longopts
            • : -> required argument
            • :: -> optional argument
        • # options
          eval set -- $(getopt -o ab --long option_a,option_b:,option_c,option_d: -n $0 -- "$@")

          while true
          do
            case "$1" in
              -a | --option_a ) flag_a=1; shift ;;
              -b | --option_b ) option_value_b="$2"; shift 2;;               --option_c ) flag_c=1; shift ;;
              --option_d ) option_value_d="$2"; shift 2 ;;
              -- ) shift; break ;;
              * ) break ;;
            esac
          done

          # parameters
          non_option_parameter_1=$1           non_option_parameter_2=$2
        • multiple values
        • if you need to specify newlines as one of the options (e.g.: --user-data-plain '#!/bin/bash -xe\necho "hello world"')
          • # options
            eval set -- $(getopt -o '' --long image-id:,user-data-plain: -n $0 -- "$@")

            while true
            do
              case "$1" in
                  --image-id ) image_id="$2"; shift 2 ;;
                  --user-data-plain ) user_data_plain=$(echo "$2" | sed 's/\\n/\n/g'); shift 2 ;;
                  --do-not-stop ) do_not_stop=1; shift ;;
                  -- ) shift; break ;;
                  * ) break ;;
              esac
            done

Sintaxi / Syntax
  • 3.5 Shell expansions
    • rangs / ranges
      • caràcter únic / single character
        • ls -l name_[0-9].txt
      • 3.5.1 Brace expansion
        • find
        • strings
          • ls -l {a-,b-}name.txt
            • a-name.txt b-name.txt
          • ls -l {a-,b-,}name.txt
            • a-name.txt b-name.txt name.txt
        • integers
          • ls -l name_{00..30}.txt
          • cp -p DSC{00627..00717}.JPG /tmp
        • integers with delta
          • ls -l name_{00..30..2}.txt
    • 3.5.2 Tilde expansion
      • ~
      • ~/foo
      • ...
    • 3.5.3 Shell parameter expansion
    • 3.5.4 Command substitution
      • $(command)
      • `command`
    • 3.5.5 Arithmetic expansion
    • 3.5.6 Process substitution
      • <(list)
      • >(list)
    • 3.5.7 Word splitting
    • 3.5.8 Filename expansion
      • glob
        • shopt -s extglob
          (extglob is on in terminals, but off in scripts)
      • 3.5.8.1Pattern matching

        		• pattern
          		 description

          		*
          		 any string
          ?
          		 single character
          [...]
          		 any one of the enclosed characters
          must be activated with:
          shopt -s extglob
          (extglob is on in terminals,
          but off in scripts)
          		 ?(pattern-list)
          		 zero or one
          *(pattern-list) zero or more
          +(pattern-list) one or more
          @(pattern-list) one
          !(pattern-list) anything except one
  • Conditional constructs
    • List constructs (or, and chains)
    • if
      • Bash conditional expressions (6.4)
        • -a file
          		 True if file exists.
          -b file True if file exists and is a block special file.
          -c file True if file exists and is a character special file.
          -d file True if file exists and is a directory.
          -e file True if file exists.
          -f file True if file exists and is a regular file.
          -g file True if file exists and its set-group-id bit is set.
          -h file True if file exists and is a symbolic link.
          -k file True if file exists and its "sticky" bit is set.
          -p file True if file exists and is a named pipe (FIFO).
          -r file True if file exists and is readable.
          -s file True if file exists and has a size greater than zero.
          ...
        • Simple logical operators in BASH (so)
        • if my_function parameter
          then
            echo "call to my_function returned 0 (ok)"
          else
            echo "call to my_function did not return 0 (not ok)"
          fi
        • # if call to my_function does not return 0, exit with 1 (compact version)
          my_function || exit 1
        • # if call to my_function does not return 0, exit with 1 (long version)
          if ! my_function
          then
              exit 1
          fi
        • # if conversion is ok, remove the input file
          ffmpeg -y -i input.mp4 toto.mp4 1>/dev/null 2>&1
          result=$?
          echo "result: ${result}"
          if ! (( result ))
          then
              # result is ok
              touch ok.txt
              rm input.mp4
          else
              # result is not ok
              touch not_ok.txt
          fi
        • if [ condition ]
          then
            ...
          else
            ...
          fi
        • if [ condition ]
          then
            # do nothing
            :
          else
            ...
          fi
        • if [ ! condition1 ] || [ condition2 ]
          then
            ...
          else
            ...
          fi
        • if [ ! $var_a -eq 0 ] && [ $var_b -ne 0 ]
          then
            ...
          else
            ...
          fi
        • verbose=0
          # check if variable verbose is set to some value
          if [[ "$verbose" ]]
          then
              echo "variable verbose exists and has value $verbose"
          fi
        • verbose=0
          # check if variable verbose is set to some value
          if [ -v verbose ]
          then
              echo "variable verbose exists and has value $verbose"
          fi
        • verbose=0
          # check if content of variable verbose is equal to 1
          if (( $verbose ))
          then
              echo "variable verbose value is 1"
          fi
        • verbose=0
          # check if content of variable verbose is equal to 1
          if (( verbose ))
          then
              echo "variable verbose value is 1"
          fi
        • verbose=1
          (( $verbose )) && echo ...
        • línia única / single line
          • if [[ "Once upon a time..." =~ ^"Once upon" ]]; then echo "found"; else echo "not found"; fi
          • sudo sh -c 'if [[ "Once upon a time..." =~ ^"Once upon" ]]; then echo "found"; else echo "not found"; fi'
        • versió compacta / compact version
          • Replace if … then … fi with && and || in Bash
            • [ $(date +%w) -eq 6 ] && (echo "do something on Saturdays"; do_some_other_stuff; exit 0;) || echo "do different things on other days"
              • final exit 0 is important in order to not to execute the commands after ||
          • print verbose messages:
            • (( $verbose )) && echo ...
          • based on existence of a variable
          • generate options for a call:
            • #!/bin/bash

              do_opt1=0
              do_opt2=1

              function opt1 {
                  (( $do_opt1 )) && (echo "--opt1";exit 0;) || (echo "--no-opt1")
              }

              function opt2 {
                  (( $do_opt2 )) && (echo "--opt2";exit 0;)
              }

              command="my_command $(opt1) $(opt2) --opt3"
              echo $command
              eval $command

              exit 0
            • #!/bin/bash

              do_opt1=0
              do_opt2=1

              #command="my_command $( (( $do_opt1 )) && (echo "--opt1";exit 0;) || (echo "--no-opt1") ) $( (( $do_opt2 )) && (echo "--opt2";exit 0;) ) --opt3"
              echo $command
              eval $command

              exit 0
        • check a flag (e.g. given using getopt)
          • if [ $flag_a -eq 1 ]
        • check if the variable is not empty:
          • if [ "$var" ]
          • if [ -n "$var" ]
        • check if the variable is empty
          • if [ -z "$var" ]
        • check if file exists:
          • if [ -f filename ]
        • Regular expressions
          • Bash Regular Expressions (Linux Journal)
          • Bash in-process regular expressions
          • use regular expression in if-condition in bash
          • regex='(.*)-'
            if [[ $nom_fitxer =~ $regex ]]
            then
                echo $BASH_REMATCH és el que volia
                echo but just ${BASH_REMATCH[1]}
            fi
          • Nota: com posar directament l'expressió regular sense la variable auxiliar $regex?
          • check if a variable content is a member of a list
            • uploadable_extensions="mp4 m3u8 ts"
              item_extension=${item_basename##*.}
                     
              if [[ $uploadable_extensions =~ (^| )$item_extension($| ) ]]
              ...
        • check if a user exists:
          • # create user celery if it does not exist
            if ! id -u celery > /dev/null 2>&1; then
                useradd celery
                echo "Created user celery"
            else
                echo "User celery already exists"
            fi
        • check if user running the script is root:
        • check if the content of a variable matches an element of a given list
        • cadenes / strings
    • case
      • detection of Linux distribution:
        • #!/bin/bash

          distribucio=$(grep '^ID=' /etc/os-release | cut -f2 -d= | tr 'A-Z' 'a-z' | tr -d '"')
          # alternative: distribucio=$(lsb_release -i | awk -F: '{print $2}' | tr -d '\t')
          case $distribucio in
              "mageia")
                  # urpmi ...
                  echo "Detected distribution: $distribucio"
                  ;;               centos)
                  # yum install ...
                  echo "Detected distribution: $distribucio"
                  ;;
              debian | ubuntu)
                  echo "Detected distribution: $distribucio"
                  #apt-get install ...
                  ;;

              *)
                  echo "Unknown distribution"
                  ;;
          esac
    • test
      • test expr
      • [ expr ]
      • AND
        • [ expr1 -a expr2 ]
      • OR
        • [ expr1 -o expr2 ]
  • Looping constructs
    • Bash for loop examples
    • Loops for, while and until
    • for
      • #!/bin/bash
        for freq in 514 522 554 570 578 586 618 658 666 682 794 818 842 850 858;
        do
            /usr/bin/dvbstream -f ${freq}000 -qam 64 -gi 4 -cr 2_3 -bw 8 -tm 8 -n 60 8192 -o > /tmp/${freq}_`date +"%Y%m%d-%H%M"`.ts
        done
      • #!/bin/bash
        for f in *;
        do ... ;
        done
      • #!/bin/bash
        for f in *;
        do
        ...
        done
      • #!/bin/bash
        i=0
        while [ $i -lt 5 ];
        do
          echo $i
          let i=i+1
        done
      • #!/bin/bash
        for i in {1..100};
        do
            echo "-------$i--------------"
            ffmpeg -re -i toto.mp4 -c copy -f flv rtmp://localhost/toto/stream01
        done
      • #!/bin/bash
        END=100
        for i in $(seq 1 $END);
        do
            echo $i
        done
      • #!/bin/bash
        max=10
        for (( i=1; i<=$max; i++ ))
        do
            echo $i
        done
    • while
      • While with read (when for has problems with spaces)
      • Bucle sobre json amb jq / Loop over json from jq
      • Wait with a timeout:
        • #!/bin/bash

          timeout=20
          increment=5
          t=0

          while (( t < timeout ))
          do
            echo $t

            if ...
            then
                break
            fi

            (( t+=increment ))
            sleep $increment
          done
      • wait until a file is found:
        • #!/bin/bash

          # seconds
          timeout=20
          increment=5
          filename=/tmp/toto.txt
          t=0

          while [ $t -lt $timeout ]
          do
            echo $t
            # check if the file exists
            if [ -f $filename ]
            then
                echo "detected file"
                break
            fi
            let t=t+$increment
            sleep $increment
          done
      • compact version
        • t=0; timeout=200; increment=2; while (( t < timeout )); do echo $t; if mount -t nfs4 192.168.1.100:/ /mnt/nfs; then break; fi; (( t+=increment )); sleep $increment; done;
    • until ...
      • Respawn
        • # respawn process only if it was terminated with code different than 0
          until myserver
          do
            echo "Server 'myserver' crashed with exit code $?. Respawning.." >&2
            sleep 1
          done

Variables
  • Variables
    • Shell parameter expansion (3.5.3) (Expansions)



    		• example
      conditional print
      		 ${parameter:-word} if parameter exists
        parameter
      else
        word
      ${parameter:?word}
      		 if parameter exists
        parameter
      else
        error: word
      ${parameter:+word} if parameter exists
        word
      else
        (nothing)
      conditional assignation
      		 ${parameter:=word} if parameter does not exist
        parameter=word
      substring
      ${parameter:offset}
      ${parameter:offset:length}
      		 substring of optional given length
      (negative value, preceded by a space, means from the end)
      		 ${parameter:3:2} ${parameter: -3:2}
      # remove last character:
      ${parameter: : -1}
      # get last character:
      ${parameter...}
      names
      ${!prefix*}
      ${!prefix@}
      		 names of variables with names starting with prefix

      ${!name[@]}
      ${!name[*]}

      length
      ${#parameter}
      		 number of characters
      removal
      		 ${parameter/pattern} remove pattern
      ${parameter#word}
      		 remove the shortest matching pattern (expansion of word), applied from the beginning of parameter
      get from the most-left specified pattern (pattern not included) ${parameter#*abc}
      ${parameter##word} remove the longest matching pattern (expansion of word), applied from the beginning of parameter
      get from the most-right specified pattern (pattern not included) ${parameter##*abc}
      ${parameter%word}
      		 remove the shortest matching pattern (expansion of word), applied from the end of parameter
      get until the most-right specified pattern (pattern not included) ${parameter%abc*}
      ${parameter%%word} remove the longest matching pattern (expansion of word), applied from the end of parameter
      get until the most-left specified pattern (pattern not included) ${parameter%%abc*}
      replacement
      ${parameter/pattern/string}
      		 replacement (only first match)
      ${parameter/pattern} remove pattern (all matches)
      ${parameter//pattern/string} replacement (all matches)
      ${parameter//pattern} remove pattern (all matches)
      ${parameter/#pattern/string} replacement: pattern at the beginning
      ${parameter/%pattern/string} replacement: pattern at the end
      case
      		 ${parameter^pattern} lowercase to uppercase (only first character)
      ${parameter^^pattern} lowercase to uppercase (all characters)
      ${parameter,pattern} uppercase to lowercase (only first character)
      ${parameter,,pattern}
      		 uppercase to lowercase (all characters)
  • Variable com si fos un fitxer / Variable as it was a file
  • Conversió de base numèrica / Numerical base conversion
  • Integer arithmetics (observe the absence of spaces)

    		• using let
      		 using ((...))
      increment
      		 let var1=$var1+1 (( var1++ ))
      addition
      		 let c=$a+$b c=$((a+b))
      subtraction
      		 let c=$a-$b c=$((a-b))
      ceil(divide/by) let result=($divide+$by-1)/$by; result=$(((divide+by-1)/by))
      random (from 20 to 30)
      		a=$(( ( RANDOM % 10 )  + 20 ))
    • comparison of integers:
      • equal:
        • if [ $variable -eq 2 ]
        • if (( variable == 2 ))
      • not equal:
        • if [ $variable -ne 2 ]
        • if (( variable != 2 ))
      • greater than:
        • if [ $variable -gt 2 ]
        • if (( variable > 2 ))
      • lesser than:
        • if [ $variable -lt 2 ]
        • if (( variable < 2 ))
  • Non-integer arithmetics:
    • Math commands
    • Bash, non-integers and arithmetic
    • total=`echo $var1/$var2 | bc -l`
    • var1=`bc << EOF
      $var1 / $var2
      EOF
      `
    • var1=$(bc << EOF
      $var1 / $var2
      EOF
      )
    • Comparison of non-integers (Shell scripting):
      • if [ `echo "$a < $b" | bc -l` = 1 ] then ...
    • Comprovació que el valor de variable sigui 1.3:
      • if [ `echo "$variable == 1.3" | bc -l` = 1 ]
        then
            echo "la variable si que val 1.3"
        else
            echo "la variable no val 1.3"
        fi  
    • nombre de decimals definit i printf amb punt decimal en lloc de coma (si teniu un locale amb coma decimal) / defined number of decimals and printf with point instead of comma (if you have a locale with decimal comma):
      • export LC_NUMERIC="C"
        a=1.234
        printf "%.2f" ${a}
        # result will be 1.23
      • export LC_NUMERIC="C"
        a=1
        b=3
        echo "$a/$b" | bc -l | xargs printf "%.2f"
        # result will be: 0.33
    • arrodoniment / round
      • export LC_NUMERIC="C"
        a=1.39
        printf "%.1f" ${a}
        # result will be: 1.4
      • echo "23.51" | awk '{printf("%d\n",$1 + 0.5)}'
    • sostre / ceil

Cadenes / Strings

Ordres / Commands

  • Bash reference manual (GNU)
  • Lists of commands (wp: List of UNIX commands):
    • asyncronous: &
    • sequential: ;
    • AND: &&
    • OR: ||
  • Functions
  • Job control:
    • jobs
    • bg
    • fg
    • ...
  • Arrays
    • 6.7 Arrays
    • type
      		 declaration
      		 creation of elements
      		 remove all elements
      		 remove single element
      		 length
      		 print keys
      		 print all values
      		 print keys and values
      		 print single value
      		 print range of values
      indexed
      • declare -a my_array
      • local -a my_array
      • readonly -a my_array
      • my_array=()
      • my_array[0]=tata
        my_array[1]=tete
      • my_array=(tata tete)
      • my_array+=(titi)
      		unset my_array
      		 unset my_array[1]
      • ${#my_array[1]}
      • ${#my_array[@]}
      • ${#my_array[*]}
      • ${!my_array[@]}
      • ${!my_array[*]}
      • ${my_array[@]}
      • ${my_array[*]}
      • printf "%s\n" "${my_array[@]}"
      • for value in "${my_array[@]}"
        do
            echo $value
        done
      • ${my_array}
      • ${my_array[0]}
      • ${my_array[1]}
      • first element:
        • ${my_array[0]}
      • last element:
        • ${my_array[-1]}
      • from second element to the end:
        • ${my_array[@]:2}
      • 3 elements, starting from the second:
        • ${my_array[@]:2:3}
      • last 2 elements
        • ${my_array[@]: -2}
      • all elements but the last 2:
        • ...
      associative
      • declare -A my_array
      • local -A my_array
      • readonly -a my_array
      • my_array[primer]=titi
      • my_array[segon]=toto


      		for key in "${!my_array[@]}"
      do
          echo "${key}: ${my_array[$key]}"
      done
      • ${my_array[primer]}
      • ${my_array[segon]}
    • Append
      • my_array=()
        my_array+=('tata')
        my_array+=('tete')
    • Remove
    • Sorting
    • Split string:
    • Join array with dots:
      • echo $(IFS='.';echo "${my_array[*]}";IFS=$'')
    • Intersection
    • Unique
  • CPU/memory activity
    • ps
      • fields:
        • VSZ: virtual_size = physical_memory + swap_memory
        • RSS: physical_memory
      • ps --help
      • ps -edalf
      • by command (http), and sort by RSS
        • ps -yl -C httpd --sort:rss
      • sum the memory ()
        • ps aux | awk '{sum+=$4}; END {print sum}'
    • pgrep
      • Instal·lació / Installation
        • CentOS
          • sudo yum install procps
        • Mageia
          • ...
      • look for a regular expression
        • pgrep -f "my_process.sh"
      • list full command line
        • pgrep -a ...
      • processes with specified parent:
        • pgrep -P <parent_pid>
      • kill
        • pgrep "my_process.sh" | xargs kill -9
        • pgrep -f "some expression" | xargs kill -9
        • processes=$(pgrep -f ...)
          ps ${processes}
          kill -15 ${processes}
    • pstree
      • show also pid:
        • pstree -p <pid>
      • show ancestors
        • pstree -s ...
      • highlight ancestors
        • pstree -h ...
      • ...
    • top
    • atop
      • One-stop performance analysis using atop
      • logs of usage (historic)
        • /etc/sysconfig/atop
          • # interval (default 10 minutes)
            INTERVAL=600
        • /usr/share/atop/atop.daily
          • # delete logfiles older than four weeks
            # start a child shell that activates another child shell in
            # the background to avoid a zombie
            #
            ( (sleep 3; find $LOGPATH -name 'atop_*' -mtime +28 -exec rm {} \;)& )
        • sudo systemctl enable atop
        • sudo systemctl start atop
        • atopsar
          • CPU total:
            • atopsar -c | grep all
        • atop -r /var/log/atop/atop_yyyymmdd
          • t
          • T
          • b: hh:mm
        • search for peaks
          • of memory (minimum free memory):
            • for f in /var/log/atop/atop_*; do atopsar -r $f -m -R 1 | sort -b -h -k 3 | head; done
          • of cpu (maximum all cpu usage):
            • for f in /var/log/atop/atop_*; do echo $f; atopsar -r $f -c -R 1 | grep all | sort -r -b -h -k 3 | head; done
        • prepend every line with analysis date
        • graphs with gnuplot
          • cpu usage
            • atopsar -r atop_20210914 -c | grep all >cpu.txt
            • cpu.plot
              • set timefmt "%H:%M:%S"
                set xtics rotate
                plot "cpu.txt" using 1:3 with lines
          • free memory
            • single day:
              • atopsar -r atop_20210914 -m | awk 'BEGIN {date_stamp=""; } /analysis date: /{date_stamp=$4;} /^[0-9]/ {print date_stamp, $0;}' >free_memory.txt
            • several days:
              • for f in atop_*; do atopsar -r $f -m | awk 'BEGIN {date_stamp=""; } /analysis date: /{date_stamp=$4;} /^[0-9]/ {print date_stamp, $0;}' >>free_memory.txt; done
            • free_memory.plot
              • set timefmt "%Y/%m/%d %H:%M:%S"
                set xtics rotate
                plot "free_memory.txt" using 1:4 with lines
        • ...
    • htop
    • vmstat 1
    • System Activity (KDE)
      • Definit a Arranjament del sistema -> Dreceres i gestos -> Dreceres de teclat globals -> Interfície d'execució d'ordres -> Mostra l'activitat del sistema
        • CTRL+ESC
  • Activitat de fitxers / File activity
  • Activitat de logs / Logs activity
    • logtop
  • Xarxa / Network
    • Network activity
    • ipcalc
      • get CIDR from a given expression
        • eval $(ipcalc -np 172.31.37.147/20); echo $NETWORK/$PREFIX
      • get local CIDR from ip command (only first occurence)
        • cidr=$(ip -o address | awk '$2 !~ /lo/ && $3 ~ /^inet$/ {print $4; exit;}')
          eval $(ipcalc -np $cidr); echo $NETWORK/$PREFIX
      • get ipv4 first address
        • ipv4_address=$(ip -o -f inet address | awk '$2 !~ /lo/ {gsub(/\/24/,"",$4);print $4; exit;}')
    • nc
    • conversion int/ipv4
      • function ipv4_to_int () {
            # convert 1.2.3.4 to 1*256^3 + 2*256^2 + 3*256^1 + 4*256^0
            local ipv4=$1
            ipv4_array=(${ipv4//./ })
            ipv4_hex_array=()
            ipv4_without_dots=''
            for value in ${ipv4_array[@]}
            do
                hex_value=$(printf '%02x' ${value})
                ipv4_without_dots="${ipv4_without_dots}${hex_value}"
                ipv4_hex_array+=(${hex_value})
            done

            # convert base 16 to base 10
            ipv4_10=$((16#$ipv4_without_dots))

            echo "$ipv4_10"

            return 0
        }

        function int_to_ipv4 () {
            # convert 1*256^3 + 2*256^2 + 3*256^1 + 4*256^0 to 1.2.3.4
            value_10=$1

            # convert base 10 to base 16
            value_16=$(printf '%08x' ${value_10})
            length=${#value_16}

            if (( length > 8))
            then
                echo "ERROR: provided value ($value_10, 0x$value_16) is too large to be converted to ipv4"
                exit 1
            fi

            local offset=2
            local ipv4=""
            while (( offset < length+2 ))
            do
                substring_16=${value_16: -${offset}:2}
                substring_10=$((16#$substring_16))

                if (( ${#ipv4} > 0 ))
                then
                    ipv4="${substring_10}.${ipv4}"
                else
                    ipv4=${substring_10}
                fi

                (( offset+=2 ))
            done

            echo "$ipv4"

            return 0
        }
        address_start=224.0.0.1
        address_start_int=$(ipv4_to_int ${address_start})
        echo "${address_start} ${address_start_int}"
        address_stop_int=$(( address_start_int + 2 ))
        address_stop=$(int_to_ipv4 ${address_stop_int})
        echo "${address_stop} ${address_stop_int}"
  • Utilització de disc / Disk usage
  • Substitució / Replacement
  • Elimina el CR
    • sed -e "s!\\r!!g" toto.txt
  • Errors in English:
    • LC_ALL=C <command>
  • File descriptors: stdin, stdout, stderr, ...
  • Power Up Linux GUI Apps (Linux Magazine)
  • Camins i noms de fitxers / Paths and file names
    • URL parsing
    • Python paths
    • Names:
      • my_path=./dir1/dir2/toto.mp4      # ./dir1/dir2/toto.mp4
      • my_dirname=$(dirname $my_path)    # ./dir1/dir2
      • my_rel_dirname=${my_dirname#*\./} # dir1/dir2
      • my_basename=$(basename $my_path)  # toto.mp4
      • my_name=${my_basename%.*}         # toto
      • my_extension=${my_basename##*.}   # mp4
      • my_rel_path=${my_path#*\./}       # dir1/dir2/toto.mp4
      • cd /tmp; mkdir -p dir1/dir2; touch dir1/dir2/toto.mp4
        my_abs_path=$(realpath $my_path)  # /tmp/dir1/dir2/toto.mp4
        my_abs_path=$(eval realpath $my_path) # (use eval when variable contains quotes)
    • Get the present directory:
      • $PWD
    • Get the directory from a filename:
      • dirname $filename
    • Get the abolute path of a file:
      • realpath $filename
    • Get the name of the file (without directory):
      • basename $filename
    • Remove file extension:
      • file_wo_ext=${file%.*}
    • Remove file extension and add a new one ".txt": (Shell parameter expansion)(FAQ)
      • txtfile=${file%.*}.txt
    • Get the file extension: (Shell parameter expansion)(FAQ)
      • extension=${file##*.}
      • to get an empty extension if the file has none:
        • extension_with_point=$([[ "$file" = *.* ]] && echo ".${file##*.}" || echo '')
    • Recursively create the needed directories:
      • mkdir -p ...
  • fitxers ini / ini files
    • Nginx config
    • augtool (Augeas)
      • No tracta qualsevol fitxer: només gestiona els fitxers inclosos per les lenses, sempre i quan siguin correctes
      • Instal·lació / Installation
        • CentOS
          • sudo yum install augeas
        • Mageia
          • urpmi augeas
      • Lenses (modules)
      • Ús / Usage
        • export AUGEAS_ROOT=/tmp/augeas-sandbox
        • augtool -b
          • print /augeas/root
      • Errors
        • Tracking down errors
        • augtool errors (only from version ...?)
        • augtool -b
          • match /augeas//error
          • ls /augeas/files/etc/nginx/nginx.conf/error
          • ...
    • crudini
      • crudini --get toto.in my_section my_key
      • crudini --set toto.in my_section my_key my_value
      • Problems
        • Whitespace around = #33
          • Workaround
            • put your_key=foo. E.g.:
              • sed -i -e '/^#host-name/ s/^#//' avahi-daemon.conf
            • crudini will preserve no spaces around '=':
              • crudini --set avahi-daemon.conf server host-name my_hostname
  • Esborrar línies en blanc / Removing blank lines (*):
    • sed '/^$/d' input.txt > output.txt
    • grep -v '^$' input.txt > output.txt
  • Gotchas
  • Bash completion
    • urpmi bash-completion
    • files:
      • rules:
        • /etc/bash_completion.d/
      • /etc/sysconfig/bash-completion
      • ~/.bash_completion (copied from /etc/skel/.bash_completion)
  • Història / History
  • Comunicació sèrie / Serial communication
    • Linux Serial Console HOWTO
    • 5 Linux / Unix Commands For Connecting To The Serial Console
      • discover

        		• lsusb
          		 dmesg | egrep --color 'serial|tty'
          SparkFun FTDI Basic Breakout - 5V Bus 001 Device 005: ID 0403:6001 Future Technology Devices International, Ltd FT232 Serial (UART) IC usb 1-1.2: FTDI USB Serial Device converter now attached to ttyUSB0
          ...

      • configure
        • print configuration
          • setserial -a /dev/ttyUSB0
          • setserial -G /dev/ttyUSB0
      • communicate
        • command usage
          screen screen /dev/ttyUSB0 115200
          cu cu -l /dev/ttyUSB0 -s 115200
          minicom
          putty
          tip tip -115200 /dev/ttyUSB0
  • aleatori / random
  • atop
  • awk
    • The GNU Awk User’s Guide
    • awk [options] 'pattern1 [&& pattern1b] {action1a;action1b}[;] pattern2 {action2}[;] pattern_start,pattern_stop {action3} ...'
    • Gawk: Effective AWK Programming ()
    • Awk - A tutorial and introduction (UNIX Grymoire)
    • awk (Eric Pement)
    • Add a column and match two files
    • Reimplementation of cut in AWK
    • Exemples / Examples
      • sdp_split.sh
      • nginx logs
      • AWS ALB logs
      • Exemples amb dvbsnoop / dvbsnoop examples
      • Exemples amb OpenSSL / OpenSSL examples
      • els espais dins de les cometes dobles no delimiten camps:
        • gawk ... FPAT ...
        • exemples
          • AWS S3 logs
          • first="first value" second="second value"
            • 'BEGIN {FPAT="([^=]+=\"[^\"]+\")"}; {print $2}'
          • first,"se cond",third
            • 'BEGIN {FPAT="([^,]+)|(\"[^\"]+\")"}; {print $2}'
          • first "se cond" third
            • 'BEGIN {FPAT="([^ ]+)|(\"[^\"]+\")"}; {print $2}'
          • [fi rst] "se cond" third
            • 'BEGIN {FPAT="([^ ]+)|(\"[^\"]+\")|(\\[[^\\[\\]]+\\])"}; {print $2}'
      • Escriu les línies numerades del primer fitxer (NR==FNR) i després les del segon:
        • awk 'NR==FNR{print "first "FNR": " $0;next}{print "second "FNR": " $0}' file1 file2
      • Obté l'enèsima línia (p.ex. 15a) d'un fitxer / Get the nth (e.g. 15th) line of a file
        • awk 'NR==15' toto.txt
        • sed -n '15p' toto.txt
      • Escriu els valors de la segona columna del fitxer, on les columnes estan separades pel caràcter '=':
        • awk -F= '{print $2}' toto.txt
      • Mostra el segon camp ($2) ("xx") de la línia on el primer camp ($1) contingui (~) l'expressió regular "ID_LENGTH" del fitxer .toto.txt (ID_LENGTH=xx)
        • awk -F= '$1 ~ /ID_LENGTH/ {print $2}' toto.txt
      • ...
        • awk -F ":" '{print $1 | "sort" }' /etc/passwd
      • Fes un grep en un altre fitxer de cadascuna de les cadenes trobades:
        • awk -F, '{print $2}' toto.csv | xargs -n 1 -I % grep % tata.txt
      • De la línia que conté la paraula "aquesta", mostra la tercera columna, però sense cap parèntesis / Replace parenthesis to nothing:
        • awk '/aquesta/{gsub(/[()]/,"",$3);print $3}}' toto.txt
      • De la línia (on els camps estan separats pel caràcter =) que conté la paraula aws_access_key_id (aws_secret_access_key), elimina els espais del segon camp i el mostra:
        • awk -F= '/aws_access_key_id/{gsub(/ /,"",$2);print $2}' /etc/aws/credentials
        • awk -F= '/aws_secret_access_key/{gsub(/ /,"",$2);print $2}' /etc/aws/credentials
      • Elimina les cometes simples / Remove sinlge quotes:
        • awk '/.../{gsub(/['\'']/,"",$2);print $2}'
        • awk '/.../{gsub(/[\x27]/,"",$2);print $2}'
      • Elimina les cometes dobles / Remove double quotes:
        • {gsub(/"/, "", $2)}
      • Elimina la primera i segona columnes / Remove first and second columns:
        • awk '!($1="") !($2="")' toto.txt
        • awk '!($1="") !($2=""){print}' toto.txt
      • Elimina les línies que comencen per "/dev"
        • awk '!/^\/dev/' toto_fstab.txt
      • Blocs separats per una línia en concret
      • Línies al voltant:
        • Print current,next,previous line using awk,sed - BASH
        • Print next few lines after pattern - awk
        • Print the line immediately before a regex, but not the line containing the regex:
          • awk '/regex/{print x};{x=$0}'
        • Print the line immediately after a regex, but not the line containing the regex (only the first occurrence):
          • awk '/regex/{f=1;next}f{print;exit}' toto.txt
        • Print the line immediately after a regex, but not the line containing the regex (all the occurrences):
          • awk '/regex/{f=1;next}f{print;f=0}' toto.txt
      • Awk: print all other columns but not $1, $2 and $3
      • Print directory of files with name ending with index.m3u8 or mp4 (split acts like basename; $(NF-2) takes the third last column)
        • fragment_list=$(awk '/(index.m3u8|mp4)$/ {n=split($(NF-2),a,"/"); print a[n-1]}' $log_path)
      • Esborra línies duplicades / Remove duplicated lines
        • awk '!a[$0]++' toto.txt
      • Obté la suma de la segona columna:
        • awk '{sum+=$2}; END {print sum}'
      • Find the funcion in fabfile.py that contains a put of file mcd.service:
        • awk -v pat="put.*mcd.service" '/^def/ {function_name = $0;} $0 ~ pat {print function_name $0}' fabfile.py
      • HLS
        • mostra les durades dels segments:
          • awk -F: '/^#EXTINF/ {gsub(/,/,"",$2); print $2}' monovariant.m3u8
        • suma les durades de tots els segments:
          • awk -F: '/^#EXTINF/ {gsub(/,/,"",$2); sum += $2} END {print sum}' monovariant.m3u8
        • mostra la suma parcial i el nom de cada segment:
          • awk -F: '/^#EXTINF/ {gsub(/,/,"",$2); sum += $2; print $2 " " sum} /.ts/ {print $1} END {print sum}' monovariant.m3u8
        • troba el segment que conté el punt temporal especificat:
          • awk -F: -v limit=3600 '/^#EXTINF/ {gsub(/,/,"",$2); sum += $2; if (sum>limit) f=1;next} f {print $0; exit}' monovariant.m3u8
      • Get listening port for process snowmix
        • netstat -pnl --ip | awk '$7 ~ /snowmix$/ {gsub(/*.:/,"",$4);print $4}'
      • get disk and mountpoint with a usage greater than a threshold (adding 0 to $2 is necessary to convert it to a number, to make sure that 17>2):
        • usage_threshold=12
          df --local --output=source,ipcent,target | tail -n +2 | awk -v usage_threshold=${usage_threshold} '{gsub(/%/,"",$2)} $2+0 >= usage_threshold {print $1 " " $3}'
  • base64
  • bc
    • ...
  • case
  • cat / EOF
    • to standard output:
      • cat <<EOF
        ...
        EOF
    • to a variable
      • my_var=$(cat <<EOF
        ...
        EOF
        )
    • to a file
      • cat >file_name <<EOF
        ...
        EOF
  • cd
    • va al directori anterior ($OLDPWD) / go to previous directory:
      • cd -
  • chmod
    • chmod -R +X dir_name (-R: recursive; +X: executable/browseable only for directories)
    • change mod of directories:
      • find /var/lib/mysql -type d | xargs chmod 700
    • change mod of files:
      • find /var/lib/mysql -type f | xargs chmod 660
    • recursive sticky group / setgid (only directories)
      • find /absolute/path/to/dir -type d -exec chmod 2770 {} +
  • comm
    • see also: uniq, join
    • print 3 columns
      • first column: only in first file
      • second column: only in second file
      • third column: in both files
    • specified files must be sorted
    • comm f1_sorted,txt f2_sorted.txt
    • comm <(sort f1_unsorted.txt) <(sort f2_unsorted.txt)
    • only entries in f1 (remove columns 2 and 3):
      • comm -23 <(sort f1_unsorted.txt) <(sort f2_unsorted.txt)
    • only entries in f2 (remove columns 1 and 3):
      • comm -13 <(sort f1_unsorted.txt) <(sort f2_unsorted.txt)
    • only common entries (remove columns 1 and 2):
      • comm -12 <(sort f1_unsorted.txt) <(sort f2_unsorted.txt)
    • from variables instead of files:
      • ...
  • cp
    • fes una còpia de seguretat / backup:
      • cp -b
    • filename expansion
    • còpia amb excepcions / copy with exceptions:
      • rsync -a --exclude='dir_to_exclude' source_path destination_path
  • crontab
  • cu
  • curl
  • cut (esborra parts d'una línia / remove parts of a line)
    • get the second field when delimiter character is '_':
      • echo 'someletters_12345_moreleters.ext' | cut -d'_' -f 2
        12345
  • date
    • command
      		 output
      		 notes

      		LANG=en_GB
      		 LANG=en_US
      		 LANG=ca_ES.UTF-8
      date
      		 Mon 20 Feb 12:07:54 CET 2017
      		 Mon Feb 20 12:08:14 CET 2017
      		 dl feb 20 12:08:27 CET 2017
      		 see languages ()
      date '+%c'
      		 Mon 20 Feb 2017 12:06:06 CET
      		 Mon 20 Feb 2017 12:06:18 PM CET
      		 dl 20 feb 2017 12:00:12 CET see:
      date '+%x'
      		 20/02/17
      		 02/20/2017
      		 20/02/17
      date -Iseconds -u
      date --iso-8601=seconds --utc
      date '+%Y-%m-%dT%H:%M:%S%z' --utc       		2014-10-31T11:03:02+0000
      date -Ins --utc
      		 2016-06-30T11:51:20,083994321+0000
      date '+%Y-%m-%dT%H:%M:%S.%N%z' --utc 2016-06-30T11:51:20.083994321+0000 NOT valid for default date format in DRF
      date '+%Y-%m-%dT%H:%M:%S.%06NZ' --utc
      		 2016-06-30T13:57:26.562720Z
      		 valid for default date format in DRF
      date '+%Y-%m-%dT%H:%M:%SZ' --utc
      		 2015-11-10T16:28:56Z
      date '+%Y-%m-%dT%H%M%SZ' --utc
      		 2015-11-10T162945Z
      date '+%Y%m%d_%H%M' 20141031_1203
      date +%s 1414753430
      		 seconds
      date +%s%03N 1414753430123 milliseconds
      date +%s%06N 1414753430123456 microseonds
      date +%s%N 1414753430123456789 nanoseconds
      date -d '2012-11-20 19:19:00' +%s
      		1353435540
      		 seconds
      date --date=@1530780235

      		dj jul  5 10:43:55 CEST 2018
      		 seconds
      date --rfc-2822
      date -R
      		 Fri, 16 Sep 2016 09:34:17 +0200
      		 used by curl -z ... / -H 'If-Modified-Since: ...'
    • present date in ISO 8601 format (-I or --iso-8601), up to seconds precision (-Iseconds), UTC (-u)
      • date -Iseconds -u
    • present date in ISO 8601 format (-I or --iso-8601), up to nanoseconds precision (-Ins), UTC (-u)
      • date -Ins -u
    • present date in a given format:
      • date '+%Y%m%d_%H%M'
    • seconds between 1970-01-01 00:00:00 UTC and now (%s)
      • date +%s
    • milliseconds between 1970-01-01 00:00:00 UTC and now
      • date +%s%03N
    • seconds between a given date and 1970-01-01 00:00:00 UTC:
      • date -d '2012-11-20 19:19:00' +%s
    • Re: find out UTC in seconds since Jan 01 1900
      • date -d '2012-11-20 19:19:00' +2208988800+%s | bc
    • dateutils
      • ...
    • timedate addition
      • now=$(date -Iseconds -u)
        duration_seconds=10
        end_date=$(date '+%Y-%m-%dT%H:%M:%SZ' --date="$now + $duration_seconds seconds" -u)
      • now=$(date -Ins -u)
        duration_seconds=10.5
        end_date=$(date '+%Y-%m-%dT%H:%M:%S.%06NZ' --date="$now + $duration_seconds seconds" -u)
      • now_ms=$(date +%s%03N -u)
        duration_ms=300
        end_date_ms=$(( now_ms + duration_ms )        )
    • time difference
      • max_time="09:22:01"
        min_time="08:44:55"
        date -u -d "0 $(date -u -d "$max_time" +"%s") sec - $(date -u -d "$min_time" +"%s") sec" +"%H:%M:%S"
      • microseconds precision:
        • max_time="09:22:01.123456"
          min_time="08:44:55.654321"
          date -u -d "0 $(date -u -d "$max_time" +"%s.%06N") sec - $(date -u -d "$min_time" +"%s.%06N") sec" +"%H:%M:%S.%06N"
      • date -u -d @$(( $(date -u -d "$max_time" +"%s") - $(date -u -d "$min_time" +"%s") )) +"%H:%M:%S"
    • datetime difference
      • How to calculate time difference in bash script?
        • $SECONDS
      • Find difference between two dates in bash
      • formatted output (integer precision)
        • min_date="2016-03-03T22:00:00Z"
          max_date="2016-03-04T22:30:00Z"
          date -d @$(( $(date -d "$max_date" +%s) - $(date -d "$min_date" +%s) )) -u +'%H:%M:%S'
      • output in seconds (integer precision)
        • min_date="2016-03-03T22:00:00Z"
          max_date="2016-03-04T22:30:00Z"
          #date -d @$(( $(date -d "$max_date" +%s) - $(date -d "$min_date" +%s) )) -u +'%s'
          delta=          $(( $(date -d "$max_date" +%s) - $(date -d "$min_date" +%s) ))
          echo $delta
      • output in seconds (nanosecond precision)
        • min_date="2016-03-03T22:00:00.123456789Z"
          max_date="2016-03-04T22:30:00.987654321Z"
          delta=$( echo $(date -d "$max_date" +%s.%N) - $(date -d "$min_date" +%s.%N) | bc -l )
          echo $delta
    • date comparison
      • date_1="2017-11-01T13:58:08.200Z"
        date_2="2017-11-01T14:58:08.000Z"

        if [[ $date_1 < $date_2 ]]
        then
            echo "$date_1 < $date_2"
        else
            echo "$date_1 >= $date_2"
        fi
    • convert seconds to time
      • time_s=10.5
        time=$(date -d@${time_s} -u +%H:%M:%S.%06N)
    • convert time (hh:mm:ss.dd) to seconds:
      • time="00:01:02.30"
        IFS=: read -r h m s <<<"$time"
        time_s=$( echo "($h * 60 + $m) * 60 + $s" | bc -l)
  • dd
    • remove the first 35 bytes from a file:
      • dd bs=35 skip=1 if=full_file.bin of=stripped_file.bin
    • from a raw file (e.g. yuv) with lines of 1440 bytes, take the 100th and 400th lines:
      • dd bs=1440 skip=100 count=1 if=test.uyvy | od -v -t u1
      • dd bs=1440 skip=400 count=1 if=test.uyvy | od -v -t u1
    • image from an sd card to a file (e.g. from a Raspberry Pi)
      • dd bs=4M if=/dev/mmcblk0 | gzip > /tmp/raspbian.bin.gz
    • create a file of 1MB containing zeros:
      • dd if=/dev/zero of=1M_zeros.dat bs=1M count=1
    • create a file of 1MB containing random values:
      • dd if=/dev/urandom of=1M_random.dat bs=1M count=1
  • dig
  • dmesg
  • dmidecode
    • Informació sobre el maquinari / Hardware information
  • du
    • Utilització de disc / Disk usage
      • du -cs * | sort -h
  • dump
    • od (octal dump)
    • xxd (hexadecimal dump)
  • echo
    • print variable content with a final \n:
      • echo $var
    • print variable content (preserve inner \n), with a final \n:
      • echo "$var"
    • do not print final \n
      • echo -n ...
    • eval escaped chars
      • echo -e ...
      • print character specified by its ascii value specified in octal:
        • old bash (<3.2):
          • echo -e \nnn...
        • new bash (>=3.2)
          • echo -e \0nnn...
  • eval
  • exec
  • expect
  • file
    • more info
      • file -b toto.file
    • mime type
      • file -b --mime-type toto.file
    • text coding
      • file -bi toto.tex
  • find
    • find
    • Find (UNIX Grymoire)
    • sintaxi / syntax:
      • opcions / options
      • proves / tests
        • -mtime +7 (més d'una setmana de temps de modificació / more than a week of modification time)
        • -newerXY
        • -type f (només fitxers / only files)
        • -perm +6000
        • -name ...
        • combinacions
          • -a -and
          • -o -or
          • !
      • accions / actions
    • -name
      • globular expressions (wildcard)
      • to use regular expressions instead of globular expressions, see regextype
      • regular expression
      • get all txt files, but exclude *bak*.txt
        • find . -name "*.txt" ! -name "*bak*"
      • remove all ts files or jpeg files older than 7 days
        • find . -type f \( -name "*.ts" -or -name "*.jpeg" \) -mtime +7 -delete
    • -regextype posix-extended -regex ...
      • How to use regex in file find
      • find all directories with name a-toto_..., b-toto_..., toto_... (but not c-toto_...)
        • find . -maxdepth 1 -type d -regextype posix-extended -regex '\./(a-|b-|)toto_.*'
    • -type
      • find regular files
        • find . -type f
      • find symbolic links (-type l)
        • find . -type l
      • find broken (-L) symbolic links (-type l)
        • find -L . -type l
      • find subdirectories
        • find . -maxdepth 1 -type d
    • -delete
      • delete files older than 5 days (print the list first):
        • find . -type f -mtime +5 -print0 | xargs -0 ls -ltr
        • find . -type f -mtime +5 -delete
      • delete all *.ts files (not beginning with a dot), without entering into the subdirectories (maxdepth 1), with less than 1k:
        • find . -maxdepth 1 -size -1k -name "[^\.]*.ts" -delete
      • recursively delete all symbolic links
        • find . -type l -delete
      • delete broken (-L) symbolic links (-type l)
        • find -L . -type l -maxdepth 1 -delete
        • verbose (-print)
          • find -L . -type l -maxdepth 1 -print -delete
      • recursively delete directories older than 150 days:
        • find . -maxdepth 1 -type d -mtime +150 -exec rm -rf {} \;
    • -exec executes command for each output item; a pipe to xargs takes the output as a single string
      • list (detailed) all files with extension m2v:
        • find . -name "*.m2v" -exec ls -l '{}' \;
        • find . -name "*.m2v" -ls
      • Copy all the files (-type f) in /var/export/backup_zeus3/ modified in the latest 24h (-mtime -1) to /tmp:
        • find /var/export/backup_zeus3/ -mtime -1 -type f -exec cp -pf '{}' /tmp \;
      • Recursively delete all directories named ".svn":
        • find . -name ".svn" -type d -exec rm -rf {} \;
      • Add ".bak" suffix to all txt files:
      • find . -name '*.txt' -exec mv {} {}.bak \;
    • | xargs
      • xargs by examples
      • list of found files (one line per file):
        • find . -name "*uyvy" -exec echo '{}' \;
      • list of found files (one line with all the files):
        • find . -name "*uyvy" -exec echo '{}' +
      • Calculates the total size of files modified today (-exec produces another behaviour):
        • find . -mtime -1 | xargs du -c
      • Fa un ls de cadascun dels fitxers (el nom pot incloure espais) / ls of each of the files (the file name can contain spaces):
        • find $PWD -name "*.html" -print0 | xargs -0 ls -l
      • Fa un zip dels fitxers */$trimestre/*.pdf
        • find . -name "*.pdf" -print0 | grep -FzZ ${trimestre} | xargs -0 zip ${trimestre}.zip
    • amb grep / with grep
      • grep toto $(find . -type f -newermt 2022-09-28)
      • find . -type f -newermt 2022-09-28 -exec grep toto '{}' \;
      • find . -type f -newermt 2022-09-28 | xargs grep tot
    • -print0
      • la separació dels resultats es fa amb '\0' en lloc d'espais / the separation of the results is made with '\0' instead of spaces
      • si va seguit per xargs, cal posar l'opció '-0' / if followed by xargs, xargs must have the option '-0'
    • -printf (%p: path and file name; %h: only path; %f: only file name)
      • find . -type f -name "*txt" -printf "fitxer: %h %f\n"
      • find . -type f -name "*txt" -printf "fitxer: %p\n"
    • How can I escape white space in a bash loop list?
    • Esborra amb ex...
    • Ignora alguns directoris / Ignore certain dirs
      • find . -name toto -not -path './not_here*'
    • dates
      • rsync with date filter
      • Unix/Linux find and sort by date modified
      • fitxers del 31 de març de 2013 / files modified on 31st March 2013:
        • find . -newermt 2013-03-31 ! -newermt 2013-04-1
      • fitxers entre el 9 d'agost de 2013 i l'11 d'agost de 2013 (ambdós inclosos), ordenats per temps:
        • find . -newermt 2013-08-9 ! -newermt 2013-08-12 -exec ls -ltr '{}' +;
        • find . -newermt 2013-08-9 ! -newermt 2013-08-12 -printf "%T@ %Tc %p\n" | sort -n
        • find . -newermt 2013-08-9 ! -newermt 2013-08-12 -printf "%T@ %p\n" | sort -n | awk '{print $2}'
        • find . -newermt 2013-08-9 ! -newermt 2013-08-12 -printf "%T@ %p\n" | sort -n | awk '{print $2}' | paste -s -d ' '
  • for
  • getent
    • getent [ahosts ahostsv4 ahostsv6 aliases ethers group gshadow hosts netgroup networks passwd protocols rpc services shadow]
  • grep
  • groups
  • head
    • see also tail
    • print first 2 lines
      • head -2
  • htop
  • iconv
  • id
  • if
  • ip
    • list all
      • ip a
      • ip addr show
    • get ip address
      • Consider: hostname --ip-address
    • route
      • ip route list
      • ip route replace default via 192.168.43.1 dev wlan0  metric 101
      • ip route del default via 192.168.43.1 dev wlan0  metric 303
      • ...
  • ipcalc
  • join
  • journalctl
  • kill
  • konsole
  • locate
  • ln
  • ls (*) (*)
    • ls -ltr (reverse time sorted)
    • ls -l 0225[1-5,7-9].MTS ("[]" means one character)
    • precise local time: (see also rsync)
      • ls --full-time
      • files
        • /etc/localtime
        • /usr/share/zoneinfo/Europe/Andorra
        • /usr/share/zoneinfo/Etc/UTC
    • precise UTC:
      • TZ="UTC" ls --full-time
    • one file per line (note that argument is number one)
      • ls -1
    • horizontal
      • ls -x
    • only dirs
      • ls -d */ | cut -f1 -d'/'
      • for i in $(ls -d ${root_dir}/*/); do echo $(basename $i); done
    • filename expansion
    • sort numbers as humans (1, 2, 10, 20):
      • ls -v
    • count number of files
    • ...
  • lsblk
  • minicom
  • mkdir
  • mktemp
    • create a temporary file:
      • file_name=$(mktemp)
    • create a temporary dir
      • dir_name=$(mktemp -d)
  • mount
    • mount NFS
    • mount some partition from an image file (e.g. an image file from an entire SD card)
      • Mounting a hard disk image including partitions using Linux
      • Mounting a raw partition file made with dd or dd_rescue in Linux
      • fdisk -l raspbian_20150731.bin
        • Disk raspbian_20150731.bin: 14,9 GiB, 15931539456 bytes, 31116288 sectors
          Units: sectors of 1 * 512 = 512 bytes
          Sector size (logical/physical): 512 bytes / 512 bytes
          I/O size (minimum/optimal): 512 bytes / 512 bytes
          Disklabel type: dos
          Disk identifier: 0x307439bc

          Device                 Boot    Start      End  Sectors   Size Id Type
          raspbian_20150731.bin1          2048  1683593  1681546 821,1M  e W95 FAT16 (LBA)
          raspbian_20150731.bin2       1687552 31050751 29363200    14G 85 Linux extended
          raspbian_20150731.bin3      31050752 31116287    65536    32M 83 Linux
          raspbian_20150731.bin5       1695744  1818623   122880    60M  c W95 FAT32 (LBA)
          raspbian_20150731.bin6       1826816 31049727 29222912    14G 83 Linux
      • parted raspbian_20150731.bin
        • GNU Parted 3.2
          Utilitzant /disc/rpi/raspbian_20150731.bin
          Welcome to GNU Parted! Type 'help' to view a list of commands.
          (parted) unit                                                            
          Unit?  [compact]? B                                                      
          (parted) print                                                           
          Model:  (file)
          Disk /disc/rpi/raspbian_20150731.bin: 15931539456B
          Sector size (logical/physical): 512B/512B
          Partition Table: msdos
          Disk Flags:

          Number  Start         End           Size          Type      File system  Flags
           1      1048576B      862000127B    860951552B    primary   fat32        lba
           2      864026624B    15897985023B  15033958400B  extended
           5      868220928B    931135487B    62914560B     logical   fat16        lba
           6      935329792B    15897460735B  14962130944B  logical   ext4
           3      15897985024B  15931539455B  33554432B     primary   ext4
      • Mount first ext4 partition (6)
        • mkdir /mnt/toto
        • mount -o loop,ro,offset=935329792 raspbian_20150731.bin /mnt/toto
      • resize partition in an image file
        • How To : Resize Partitions in an Image File
        • resize2fs
        • create a device (/dev/loop0) for all partitions:
          • losetup -f --show raspbian_20150731.bin
          • parted /dev/loop0
            • unit
              • B
            • print
              • Number  Start         End           Size          Type      File system  Flags
                 1      1048576B      862000127B    860951552B    primary   fat32        lba
                 2      864026624B    15897985023B  15033958400B  extended
                 5      868220928B    931135487B    62914560B     logical   fat16        lba
                 6      935329792B    15897460735B  14962130944B  logical   ext4
                 3      15897985024B  15931539455B  33554432B     primary   ext4
        • create a device (/dev/loop1) for partition 6:
          • losetup -f --show -o 935329792 raspbian_20150731.bin
          • parted /dev/loop1
            • unit
              • B
            • print
              • Number  Start  End           Size          File system  Flags
                 1      0B     14996209663B  14996209664B  ext4
        • check partition 6
          • e2fsck -f /dev/loop1
        • get the minimum size of partition 6
          • resize2fs -P /dev/loop1
            • resize2fs 1.42.12 (29-Aug-2014)
              Estimated minimum size of the filesystem: 1712943
            • as units are 4k, this means: 1712943*4096 = 7016214528 (~7GB)
        • resize partition 6
          • resize2fs /dev/loop1 8G
        • remove loops
          • losetup -d /dev/loop0 /dev/loop1
    • Problemes / Problems
      • SELinux
      • /var/log/messages: systemd: Unit mnt-vol1.mount is bound to inactive unit dev-xvdh.device. Stopping, too.
        • Solution:
          • systemctl daemon-reload
  • nc / ncat / netcat
  • netstat
  • nmap
  • nslookup
  • paste (posa línies en una sola / put lines into one single line)
    • put all ts files into one single line, where filenames are separated by a '|' (useful when concatenating files):
      • find . -name "*.ts" | paste -s -d '|'
    • put three consecutive lines into three columns:
      • paste -s -d '\t\t\n' input.dat > output.dat
    • other examples:
  • patch
  • pgrep
  • pkill
    • kill (SIGKILL) all child processes
      • pkill -KILL -P $$
  • printf
    • The printf command
    • integer format with zeros:
      • printf "%06d" $my_var
    • quote special characters
      • printf "%q" $my_var
    • Colors / Colours
      • Standard ECMA-48: Control Functions for Coded Character Sets
      • C
      • start colouring
        • "\e[<attributes>m"
      • attributes are ";" separated
        • 0
          		 all attributes off
          1
          		 bold on
          4
          		 undescore
          5
          		 blink on
          7
          		 reverse video on
          8
          		 concealed on
          foreground colour
          		 background
          30
          		 black 40
          31
          		 red 41
          32
          		 green 42
          33
          		 yellow 43
          34
          		 blue 44
          35
          		 magenta 45
          36
          		 cyan 46
          37
          		 white 47
      • end colouring
        • "\e[0m"
      • Print "Hello", in blue foreground (34), yellow background (43), blinking (5)
        • printf "\e[5;34;43mHello\e[0m\n"
      • emacs
  • ps
  • pstree
    • CentOS
      • sudo yum install psmisc
  • pv
  • read
    • read one line from stdin and populate specified variables
    • read <options> <list of vars>
    • options
      • option
        		 description
        		 examples
        -r
        		 The backslash is considered to be part of the line, and not interpreted as an escape character
        -s
        		 silent
        -t <secs>
        		 timeout



    • used variables
      • variable
        		 description
        		 examples
        		 info
        IFS
        		 Internal Field Separator
        		 IFS= (or IFS='')  prevents leading/trailing whitespace from being trimmed
    • line by line from a file
    • read from a file, containing three fields (separated by space) per line
      • while read -r field1 field2 field3
        do
            echo $field1 $field2 $field3
        done < $filename
    • Bucle sobre json amb jq / Loop over json from jq
    • iterate over lines that can contain spaces (a simple for would only work if there were no spaces in lines) from a variable:
      • #! /bin/bash

        fitxers_html=$(find $PWD/ -name "*.html")

        while IFS= read -r linia
        do
            echo "${linia}"
            sed -i 's#src="http://www.w3.org/Icons/valid-html401"#src="//www.w3.org/Icons/valid-html401"#g' "${linia}"
        done < <(echo "$fitxers_html")

        exit 0
      • #!/bin/bash

        tags=$(cat <<EOF
        TAG:major_brand=mp42
        TAG:minor_version=0
        TAG:compatible_brands=isommp42
        TAG:creation_time=2016-10-27 14:41:05
        EOF
        )
        echo "tags: $tags"

        while IFS= read -r
        do
            tag=$REPLY
            echo "tag: $tag"
            pair=${tag#TAG:}
            echo "pair: $pair"
        done < <(echo "$tags")
  • recode
  • rm
    • esborra el fitxer "-nom_fitxer":
      • rm -- -nom_fitxer
    • esborra recursivament amb excepcions / recursively remove with exceptions:
      • find dir -maxdepth 1 -mindepth 1 ! -name "name_to_avoid" -exec rm -rf {} \;
    • esborra els fitxers el nom dels quals està especificat en un fitxer:
      • ls *.mp4 -1 > mp4_files_to_be_deleted.txt
      • xargs rm < mp4_files_to_be_deleted.txt
    • esborra els fitxers apuntats per enllaços simbòlics
  • rsync
  • screen
    • serial
    • Screen Command: Set Baud Rate [ Terminal Communication ]
    • Commands
      • Common screen commands
      • screen /dev/ttySX baud_rate,cs8|cs7,ixon|-ixon,ixoff|-ixoff,istrip|-istrip
        • baud_rate
        • cs7|cs8: 7|8 bits per byte
        • ixon|-ixon: enable|disable software flow-control (CTRL-S/CTRL-Q) for sending data
        • ixoff|-ixoff: enable|disable flow-control (CTRL-S/CTRL-Q) for receiving data
        • istrip|-istrip: clear|keep the eight bit in each received byte
      • CTRL+a ?
        		 help
        CTRL+a i
        		 info
        CTRL+a K
        		 kill session
        ...
  • sed
    • Sed - An Introduction and Tutorial by Bruce Barnett
    • Unix Sed Tutorial: Append, Insert, Replace, and Count File Lines
    • sed [-n] [-e] '[/begin_pattern/[,/end_pattern/]] [!]command' [-e 'command'] ... input_filename
    • sed [-n] [-e] '[/begin_pattern/[,/end_pattern/]] [!]{command}' [-e 'command'] ... input_filename
    • Command line options
      • -f filename
        		 execute script in a file (several commands can be specified, each in one line)
        -e script
        		 execute script (no need to specify -e for the first script)
        -n only print the matching pattern
        -i replace input file
        ...
    • Commands
    • Inserció de línies / Line insertion
      • before
        • sed '/This is an existing line/ i This is a new line' toto.txt
        • spaces has to be escaped:
          • sed -i '/<link rel="stylesheet" href="css\/fpm.css"/ i\  <meta name="viewport" content="width=device-width">' toto.html
      • after
        • sed '/This is an existing line/ a This is a new line\n and a second one' toto.txt
    • Substitució / Replacement
      • Per a substituir totes (g) les cadenes inici (pot ser una expressió regular) en el fitxer fitxer_original per final, en fitxer_final:
        • sed 's/inici/final/g' fitxer_original > fitxer_final
        • per a sobreescriure el fitxer inicial: "-i" / to overwrite original file
          • sed -i 's/inici/final/g' fitxer_original
          • create a backup (fitxer_original.bak)
            • sed -i.bak 's/inici/final/g' fitxer_original
        • el separador no cal que sigui '/'. Pot ser qualsevol altre caràcter.
        • si a dins de l'ordre volem fer servir variables, cal posar-la entre cometes dobles:
          • baseurl=$1 sed -i "s#<BaseURL>.*</BaseURL>#<BaseURL>${baseurl}</BaseURL>#g" toto.html
      • newline
        • sed: How can I replace a newline (\n)?
        • remove line feed (^M) (see also dos2unix):
          • sed -i 's/\r//g' toto.txt
        • replace all "/n" (2 chars) by real newlines in a variable:
          • echo "$toto" | sed 's/\\n/\n/g'
      • Replace ' -> '\'' (e.g. for curl POST with a specified json)
        • echo "${input}" | sed "s/'/'\\\''/g"
      • Estableix BaseURL en un fitxer mpd (MPEG DASH):
        • #! /bin/bash
          baseurl=$1
          fitxers_mpd=$(find $PWD -name "*.mpd")
          for fitxer in $fitxers_mpd; do
              echo "${fitxer}: setting BaseURL to ${baseurl}"
              sed -i "s#<BaseURL>.*</BaseURL>#<BaseURL>${baseurl}</BaseURL>#g" $fitxer
          done
      • En tots els fitxers del directori public_html:
        • #! /bin/bash
          for f in public_html/*
          do
              echo $f;
              sed -i.bak 's*<P ALIGN=center>*<P STYLE="text-align:center">*g' $f;
          done
        • #! /bin/bash
          for f in public_html/*;
          do
              echo $f;
              sed 's*<P ALIGN=center>*<P STYLE="text-align:center">*g' $f > ${f}#;
              rm -f $f;
              mv ${f}# $f;
          done
      • Canvia la mida de la lletra dels textos en Android / Change the font size in Android texts:
        • find . -name *.xml -exec sed -i.bak 's/android:textSize="10sp"/android:textSize="12sp"/g' {} \;
      • Afegeix un meta refresh a tots els fitxers html:
        • #! /bin/bash
          fitxers_html=$(find $PWD -name "*.html")
          for fitxer in $fitxers_html;
          do
              echo "${fitxer}"
              fitxer_nom=$(basename ${fitxer})
              inicial="<head>"
              final="<head>\n  <meta http-equiv=\"refresh\" content=\"5; url=http://www.francescpinyol.cat/${fitxer_nom}\">"
              sed -i "s#${inicial}#${final}#g" ${fitxer}
          done
      • Comenta totes les línies que comencin per "net.ipv4.ip_forward":
      • Uncomment all lines that have an "=":
        • sed -e '/=/ s/^#//' -i manila.rpm.uncommented.conf
      • Uncomment lines that start with #submission and subsequent lines with options, unil the next line that is not an option (in the example, line starting with smtps):
        • /etc/postfix/master.cf
          • ...
            #submission inet n       -       n       -       -       smtpd
            #  -o syslog_name=postfix/submission
            #  -o smtpd_tls_security_level=encrypt
            #  -o smtpd_sasl_auth_enable=yes
            #  -o smtpd_reject_unlisted_recipient=no
            #  -o smtpd_client_restrictions=$mua_client_restrictions
            #  -o smtpd_helo_restrictions=$mua_helo_restrictions
            #  -o smtpd_sender_restrictions=$mua_sender_restrictions
            #  -o smtpd_recipient_restrictions=permit_sasl_authenticated,reject
            #  -o milter_macro_daemon_name=ORIGINATING
            #smtps     inet  n       -       n       -       -       smtpd
            ...
        • activate_submission.sed
          • /^#submission/,/^#\w/ {
                /^#submission/ s/^#//
                /^#  / s/^#//
            }
        • sed -f activate_submission.sed master.cf
          • ...
            submission inet n       -       n       -       -       smtpd
              -o syslog_name=postfix/submission
              -o smtpd_tls_security_level=encrypt
              -o smtpd_sasl_auth_enable=yes
              -o smtpd_reject_unlisted_recipient=no
              -o smtpd_client_restrictions=$mua_client_restrictions
              -o smtpd_helo_restrictions=$mua_helo_restrictions
              -o smtpd_sender_restrictions=$mua_sender_restrictions
              -o smtpd_recipient_restrictions=permit_sasl_authenticated,reject
              -o milter_macro_daemon_name=ORIGINATING
            #smtps     inet  n       -       n       -       -       smtpd
            ...
    • Substitueix tota la línia / Replace all line:
      • sed -e "/^.\?admin_token/ c\admin_token=$ADMIN_TOKEN" -i /etc/keystone/keystone.conf
      • sed -e "/^host    all             all             127.0.0.1/ c\host    all             all             127.0.0.1/32            md5" /var/lib/pgsql/data/pg_hba.conf
      • canvia el valor de la variable toto_var:
        • sed '/var toto_var*/ c\var toto_var = "nou_valor"' toto.js
    • grep-like
      • sed -n /pattern/p file.txt
        • print only lines starting with "admin_token" or one single character followed by "admin_token":
          • sed -n '/^.\?admin_token/p' /etc/keystone/keystone.conf
        • print only line with "duration=" in it, and replace it by nothing (i.e. get the value) (e.g. to parse result given by ffprobe):
          • sed -n 's/duration=//p'
      • printing all lines except (!) those that match the pattern (grep -v):
        • sed -n -e '/pattern/ !p' file.txt
    • Ranges by patterns
    • elimina totes les línies que comencin per «abc»:
  • seq
    • similar to {10..20}, but with seq one of the limits can be a variable
    • END=20
      for i in $(seq 10 $END)
      do
          echo $i
      done
  • set
    • activate debug
      • set -x
    • set positional parameters
      • set -- param1 param2
        • Exemple / Example
          • #!/bin/bash

            echo "supplied $# parameters: $@"
            # add a parameter
            set -- "new_param1" "new_param2"
            echo "new $# parameters: $@"

            exit 0
  • shopt
  • sleep
  • ssh
  • su / sudo
    • RootSudo (Ubuntu)
    • sudoers file
      • su root
      • adduser mynewuser
      • passwd mynewuser
      • [su]
      • option 1: add files to /etc/sudoers.d (because they are included by the last line in /etc/sudoers file)
        • su
        • echo "mynewuser ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/mynewuser
      • option 2: directly edit sudoers file
        • visudo (don't use regular editor, as you can end up with a non-working system)
          • got to after line with "root ALL=(ALL) ALL"
          • i
          • mynewuser ALL=(ALL) ALL
            • to avoid password request every time you execute sudo, append the following line at the end of the file (/etc/sudoers):
              • mynewuser ALL=(ALL) NOPASSWD: ALL
          • ESC
          • :wq
      • Problemes / Problems
    • How to Add a User and Grant Root Privileges on CentOS 7
    • give a user admin privileges:
      • sudo adduser <username> admin
    • redirect
      • sudo sh -c 'echo "nameserver 192.168.0.1" >> /etc/resolv.conf'
    • execute as another user (which has home)
      • su my_user -c "mkdir /mnt/nfs/my_dir"
      • sudo su - my_user -c "mkdir /mnt/nfs/my_dir"
    • execute as another user (which has no home; created as --no-create-home)
      • su nginx -s /bin/bash -c "mkdir /mnt/nfs/my_dir"
      • sudo -u nginx mkdir /mnt/nfs/my_dir
      • sudo su - nginx -s /bin/bash -c "export"
    • mode interactiu / interactive mode
      • sudo -i
    • mode gràfic / graphical mode
      • Gnome
        • gksudo gedit /etc/fstab
      • KDE
        • kdesudo kate /etc/X11/xorg.conf
      • LXDE
        • gksudo leafpad /etc/fstab (text editor)
        • gksudo pcmanfm /etc (file browser)
    • sudo in ssh, crontab
      • to avoid having to specify -t option (to avoid error: sudo: sorry, you must have a tty to run sudo):
        • /etc/sudoers.d/80-my_user
          • myusername ALL=(ALL) NOPASSWD: ALL
            # no need to specify "-t" when doing sudo via ssh for this user
            Defaults:myusername !requiretty
        • for all users: /etc/sudoers
          • Defaults !requiretty
    • while in ssh
      • ssh -q remote_user@remote_server -t "sudo bash -c \"while ! [ -e /dev/xvdf ]; do sleep 1; done\""
  • sysctl
  • tail
    • see also: head
    • get latest 5 lines
      • tail -n5
  • tar
    • root tarball needs to be extracted with --numeric-owner
    • ignore git dirs:
      • tar --exclude=.git --exclude=.gitignore -cvzf toto.tgz my_dir
    • follow symbolic links (-h)
      • tar chvf documents.tgz Documents
    • uncompress
      • extension
        		 tar
        .tgz, .tar.gz
        		 -xzf
        .bz2
        		 -xjf
        .xz
        		 -xJf
    • partially remove path when creating (/mnt/vol1 will not appear in paths for files inside the tar file):
      • tar cvzf toto.tgz --directory /mnt/vol1 letsencrypt
  • tee
  • telnet
  • test
  • timeout
  • tip
  • top
  • tr
    • substitució de caràcters individuals / replacement of individual characters:
      • tr '<input_characters>' '<output_characters>'
    • elimina les comes i cometes / delete comma and quotes:
      • tr -d '",'
    • elimina les cometes simples / delete single quotes
      • tr -d "'"
    • remove line feed (^M):
      • tr -d '\r'
    • majúscules i minúscules / upper case and lower case
      • ...
    • url safe
  • trap
    • indica una funció que cal executar quan es rep el senyal, excepte KILL (9), en lloc del comportament habitual
    • How "Exit Traps" Can Make Your Bash Scripts Way More Robust And Reliable
      • Capping Expensive Resources: AWS AMI after updating an instance
    • Sending and Trapping Signals
    • Identifying received signal name in Bash
    • Examples
      • trap <code or function> <list of signals>
      • #!/bin/bash

        trap "rm $TEMP_FILE; exit" SIGHUP SIGINT SIGTERM
        ...
      • #!/bin/bash

        function on_exit {   
            # variable to know whether we arrive here from an "exit 0" or "exit 1"
            rv=$?

            # do something before exiting (e.g. remove temporary files)
            ...
        }

        # when EXIT signal is sent to this script (either from an exit command or from a CTRL-C), go to finish function
        trap on_exit EXIT

        ...

        if ...
        then
           exit 1
        fi

        exit 0
      • #!/bin/bash

        function on_exit {   
            # variable to know whether we arrive here from an "exit 0" or "exit 1"
            rv=$?

            # do something before exiting (e.g. kill all children processes)
            own_pid=$$
            echo "sending TERM to all children processes (whose parent id is ${own_pid}):"
            # only informative:
            # install pstree with: sudo yum install psmisc
            command -v pstree >/dev/null 2>&1 && pstree -p ${own_pid}
            # install pgrep with: sudo yum install procps
            command -v pgrep >/dev/null 2>&1 && pgrep -a -P ${own_pid}

            pkill -TERM -P $$
        }

        # when EXIT signal is sent to this script (either from an exit command or from a CTRL-C), go to finish function
        trap on_exit EXIT

        ...

        if ...
        then
           exit 1
        fi

        exit 0
  • tree
    • add colours:
      • tree -C .
  • umask
  • useradd
    • with specified password:
      • useradd --password $(openssl passwd -1 ${new_password}) ${new_user}
  • usermod
  • wait
  • which
  • while
  • xargs
    • takes input and passes it as arguments to the specified command. Input can be separated by whitespaces (default) or by '\0' (option: -0).
    • find and xargs
    • remove files specified in a file as one file per line
    • list files specified in a file as one file per line
      • xargs ls -l < list_files.txt
      • cat list_files.txt | grep something | xargs ls -s
  • zip / unzip
    • compress
      • recursive
        • zip toto.zip -r my_dir
      • encrypt with a password (prompt):
        • zip -e toto.zip -r my_dir
      • with find
        • #!/bin/bash

          EXPECTED_ARGS=1
          if [ $# -ne $EXPECTED_ARGS ]
          then
              cat <<EOF
          Usage: `basename $0` yyyy_t
          EOF
              exit 1
          fi

          PREFIX=my_prefix
          trimestre=$1
          find . -name "*.pdf" -print0 | grep -FzZ ${trimestre} | xargs -0 zip ${PREFIX}_${trimestre}.zip
    • uncompress
      • unzip -d /destination/dir toto.zip
    • password protection

Usuaris i grups / Users and groups
  • Usuaris / Users:
    • A Complete Guide to Usage of ‘usermod’ command – 15 Practical Examples with Screenshots
    • Llista d'usuaris / List of users
    • Crea un usuari / Create a user:
      • useradd user_name -u user_id -g main_group -G additional_group_1,additional_group_2
      • sense casa / with no home:
        • adduser --system --no-create-home --user-group -s /sbin/nologin nginx
        • make a dir as user with no login
        • add path to PATH variable for all users (including those without home)
          • /etc/profile.d/local_path.sh
            • # add /usr/local/bin path for all users
              pathmunge /usr/local/bin
    • Afegeix contrasenya a un usuari / Add password to a user:
    • Afegeix grups addicionals a un usuari / Add additional groups to a user:
      • usermod -a -G additional_group_1,additional_group_2 user_name
        • Nota: "-a" afegeix els grups secundaris citats. Si no, els substitueix per la nova llista
      • usermod -a -G dialout my_user
    • Elimina tots els grups secundaris d'un usuari:
      • usermod -G "" user_name
    • Llista les propietats i grups d'un usuari:
      • id nom_usuari
    • ID d'usuari de l'usuari actual:
      • id -u
    • ID de grup de l'usuari actual:
      • id -g
    • Llista de groups dels quals l'usuari actual és membre:
      • groups
    • Canvia l'uid de l'usuari (fa els canvis pertinents als fitxers del directori arrel de l'usuari) / Change user id:
      • # killall -u nom_usuari
      • # usermod -u nou_uid nom_usuari
    • Actualització dels grups per a l'usuari actual, sense haver de reiniciar la sessió (the group has already been assigned to the user, and it is shown with id user_name):
      • newgrp additional_group_1
    • Esborra un usuari / Delete a user
      • userdel user_name
    • Sticky group / setgid (els nous fitxers i directoris a dins es crearan amb el grup del directori): 
    • umask
      • umask (wp)
      • mostra la màscara actual / show present mask
        • umask [-S]
      • estableix la màscara / set the mask:
        • umask txyz
        • permanentment / permanently
          • ~/.bashrc
            • umask 022
          • /etc/bashrc
            • umask 022
          • /etc/login.defs
            • UMASK 022
          • /etc/csh.login
            • umask 022
      • usage in:
    • Problemes / Problems
      • pam_succeed_if(su:auth): requirement "uid >= 1000" not met by user ...
    • add user to Samba
  • Grups / Groups
    • Howto: Linux Add User To Group
    • Llista de grups / List of groups
    • Crea un grup / Create a group:
      • groupadd [-r] -g group_id group_name
    • Llista els usuaris d'un grup / List the members of a group:
      • grep ^group_name /etc/group
    • Llista de grups d'un usuari
      • groups nom_usuari
    • Afegeix usuaris a un grup
    • Esborra usuaris d'un grup
    • Canvia el gid d'un grup / Change the gid of an existing group:
      • groupmod --gid <new_gid> <group_name>
    • chgrp additional_group_1 -R dir_name_1
    • chmod +s dir_name_1

Exemples / Examples
  • Detecció de fitxers duplicats, encara que tinguin noms diferents:
    • #!/bin/bash
      if [ $# -ne 2 ]
      then
          echo "Usage: " $0 dir1 dir2
      else
          dir1=$1
          dir2=$2
          echo "dir1: " $dir1
          echo "dir2: " $dir2

          for f1 in $dir1/*
          do
          if [ -f $f1 ]
          then
              echo "-> " $f1
              for f2 in $dir2/*
              do
              if [ -f $f2 ]
              then
                  if cmp $f1 $f2 > /dev/null 2>&1
                  then
                  echo "    " $f2 "iguals"
                  fi
              fi
              done
          fi
          done

      fi
  • Comprova si totes les imatges de la targeta SD han estat passades a un altre directori / Check if all the pictures in SD card have been copied to a local directory (comprova_fotos.sh):
    • #!/bin/bash

      EXPECTED_ARGS=2
      if [ $# -ne $EXPECTED_ARGS ]
      then
          cat <<EOF
      Usage: `basename $0` dir1 dir2

      E.g.: `basename $0` /run/media/cesc/41AD-7342/DCIM/101CANON/ ~/Imatges/Canon/
      EOF
          exit 1
      fi

      dir1=$1
      dir2=$2

      for fitxer_dir1 in ${dir1}/*
      do   
          if [ -f $fitxer_dir1 ]
          then
            fitxer1=$(basename $fitxer_dir1)
            trobat=$(find $dir2 -name $fitxer1)
            if [ -n "$trobat" ]
            then
              # 32: verd
              printf "\33[32m"
              printf "$fitxer1 -> $trobat"
            else
              # 31: vermell
              printf "\33[31m"
              printf "$fitxer1"
            fi
            # sense colors
            printf "\33[0m"
            printf "\n"
          fi

      done
  • Check mime-type (or use mimetype command instead of file command)
    • if [ `file -b --mime-type $1` = "text/plain" ]
      then
          echo "plain text"
      else
          echo "no plain text"
      fi
    • # any image mime-type
      if [[ $(file -b --mime-type $filename) =~ ^"image" ]]
      ...
  • Reading files
  • Subsample image file names (e.g. to make a faster timelapse from images)
    • #!/bin/bash

      EXPECTED_ARGS=1
      if [ $# -ne $EXPECTED_ARGS ]
      then
          cat <<EOF
      Usage: `basename $0` delta
      EOF
          exit 1
      fi

      delta=$1

      # remove all symbolic links
      find . -type l -delete

      counter=0
      for filename in *
      do
          # only image files
          if [[ $(file -b --mime-type $filename) =~ ^"image" ]]
          then
              echo $filename
              # file extension
              file_ext=${filename##*.}
              # only digits
              file_index=${filename//[^0-9]/}
              # remove leading zeros by converting to int in base 10
              file_index_wo_zeros=$((10#$file_index))
              # calculate modulo delta
              sub=$(( file_index_wo_zeros % delta ))
             
              if (( sub == 0 ))
              then
                  new_filename=$(printf "im_%05d.$file_ext" $counter)
                  echo "$filename -> $new_filename"
                  ln -s $filename $new_filename
                  (( counter++ ))
              fi
          fi
      done

      echo "You can play images with, e.g.: ffplay -i im_%05d.$file_ext"

http://www.francescpinyol.cat/shell.html
Primera versió: / First version: 4.III.2016
Darrera modificació: 9 de febrer de 2023 / Last update: 9th February 2023

Valid HTML 4.01!

Cap a casa / Back home