Contenu

• chat
• tête
• queue
• toilettes
• grep
• Couper
• pâte
• Trier
• uniq
• awk
• tr
• plier
• sed
• De vrais exemples
• Approfondissement: boucles for et xargs
• Sommaire
• Et après

Traitement de texte Shell

Faire en sorte que chaque programme fasse bien une chose - Les fondements de la philosophie Unix

Un exemple de fichier CSV est disponible en ligne . Vous pouvez le télécharger pour vérifier le matériel.

chat

$ cat Documents/readme
Thanks again for reading this book!
I hope you're following so far!

$ cat Documents/computers
Computers are not intelligent
They're just fast at making dumb things.

$ cat Documents/readme Documents/computers
Thanks again for reading this book!
I hope you are following so far!

Computers are not intelligent
They're just fast at making dumb things.

tête

$ head -n 2 metadata.csv
metric_name,metric_type,interval,unit_name,per_unit_name,description,orientation,integration,short_name
mysql.galera.wsrep_cluster_size,gauge,,node,,The current number of nodes in the Galera cluster.,0,mysql,galera cluster size

queue

$ tail -n 1 metadata.csv
mysql.performance.queries,gauge,,query,second,The rate of queries.,0,mysql,queries

$ tail -n +42 metadata.csv
mysql.replication.slaves_connected,gauge,,,,Number of slaves connected to a replication master.,0,mysql,slaves connected
mysql.performance.queries,gauge,,query,second,The rate of queries.,0,mysql,queries

toilettes

$ wc -l metadata.csv
43  metadata.csv
$ wc -w metadata.csv
405 metadata.csv
$ wc -c metadata.csv
5094 metadata.csv

$ wc metadata.csv
43     405    5094 metadata.csv

$ cat metadata.csv | wc
43     405    5094
$ cat metadata.csv | wc -l
43
$ wc -w < metadata.csv
405

grep

$ grep mutex metadata.csv
mysql.innodb.mutex_os_waits,gauge,,event,second,The rate of mutex OS waits.,0,mysql,mutex os waits
mysql.innodb.mutex_spin_rounds,gauge,,event,second,The rate of mutex spin rounds.,0,mysql,mutex spin rounds
mysql.innodb.mutex_spin_waits,gauge,,event,second,The rate of mutex spin waits.,0,mysql,mutex spin waits

$ grep mutex metadata.csv | grep OS
mysql.innodb.mutex_os_waits,gauge,,event,second,The rate of mutex OS waits.,0,mysql,mutex os waits

$ grep -v gauge metadata.csv
metric_name,metric_type,interval,unit_name,per_unit_name,description,orientation,integration,short_name

$ grep -i os metadata.csv
mysql.innodb.mutex_os_waits,gauge,,event,second,The rate of mutex OS waits.,0,mysql,mutex os waits
mysql.innodb.os_log_fsyncs,gauge,,write,second,The rate of fsync writes to the log file.,0,mysql,log fsyncs

$ grep -l mysql metadata.csv
metadata.csv

$ grep -c select metadata.csv
3

$ grep -r are ~/Documents
/home/br/Documents/computers:Computers are not intelligent
/home/br/Documents/readme:I hope you are following so far!

$ grep follow ~/Documents/readme
I hope you are following so far!
$ grep -w follow ~/Documents/readme
$

Couper

$ tail -n 5 metadata.csv | cut -d , -f 1
mysql.performance.user_time
mysql.replication.seconds_behind_master
mysql.replication.slave_running
mysql.replication.slaves_connected
mysql.performance.queries

$ tail -n 5 metadata.csv | cut -d , -f 1,2
mysql.performance.user_time,gauge
mysql.replication.seconds_behind_master,gauge
mysql.replication.slave_running,gauge
mysql.replication.slaves_connected,gauge
mysql.performance.queries,gauge

pâte

$ cat ingredients
eggs
milk
butter
tomatoes
$ cat prices
1$
1.99$
1.50$
2$/kg
$ paste ingredients prices
eggs    1$
milk    1.99$
butter  1.50$
tomatoes    2$/kg

$ paste ingredients prices -d:
eggs:1$
milk:1.99$
butter:1.50$
tomatoes:2$/kg

$ paste -s -d, ingredients
eggs,milk,butter,tomatoes

$ cat ingredients | paste -s -d, -
eggs,milk,butter,tomatoes

Trier

$ cat ingredients
eggs
milk
butter
tomatoes
salt
$ sort ingredients
butter
eggs
milk
salt
tomatoes

$ sort -r ingredients
tomatoes
salt
milk
eggs
butter

$ cat numbers
0
2
1
10
3
$ sort numbers
0
1
10
2
3
$ sort -n numbers
0
1
2
3
10

uniq

$ cat duplicates
and one
and one
and two
and one
and two
and one, two, three
$ uniq duplicates
and one
and two
and one
and two
and one, two, three

$ sort duplicates | uniq
and one
and one, two, three
and two

$ sort duplicates | uniq -c
   3 and one
   1 and one, two, three
   2 and two

$ sort duplicates | uniq -u
and one, two, three

Remarque. uniqIl est particulièrement utile en combinaison avec le tri, car le pipeline | sort | uniqvous permet de supprimer toutes les lignes en double dans un fichier ou un flux.

awk

$ cat -t multi-columns
John Smith    Doctor^ITardis
Sarah-James Smith^I    Companion^ILondon
Rose Tyler   Companion^ILondon

Remarque. cat -taffiche les onglets comme ^I.

$ cat multi-columns | awk '{ print $1 }'
John
Sarah-James
Rose
$ cat multi-columns | awk '{ print $3 }'
Doctor
Companion
Companion
$ cat multi-columns | awk '{ print $1,$2 }'
John Smith
Sarah-James Smith
Rose Tyler

Remarque. { print $NF }affiche la dernière colonne d'une ligne.

tr

$ echo "Computers are fast" | tr a A
computers Are fAst

$ echo "computers are fast" | tr '[:space:]' ','
computers,are,fast,%

$ echo "computers are fast" | tr '[:lower:]' '[:upper:]'
COMPUTERS ARE FAST
$ echo "COMPUTERS ARE FAST" | tr '[:upper:]' '[:lower:]'
computers are fast

$ echo "computers are fast" | tr -c '[aeiouy]' ' '
 o  u e   a e  a

$ echo "Computers Are Fast" | tr -d '[:lower:]'
C A F

$ echo "computers are fast" | tr 'a-e' 'x'
xomputxrs xrx fxst
$ echo "5uch l337 5p34k" | tr '1-4' 'x'
5uch lxx7 5pxxk

$ echo "Computers         are       fast" | tr -s ' '
Computers are fast

plier

$ cat ~/Documents/readme | fold -w 16
Thanks again for
 reading this bo
ok!
I hope you're fo
llowing so far!

Thanks again
for reading
this book!
I hope you're
following so
far!

sed

[address[,address]]function[arguments]

Remplacement de texte

s/PATTERN/REPLACEMENT/[options]

$ cat hello
hello hello
hello world!
hi
$ cat hello | sed 's/hello/Hey I just met you/'
Hey I just met you hello
Hey I just met you world
hi

$ cat hello | sed 's/hello/Hey I just met you/g'
Hey I just met you Hey I just met you
Hey I just met you world
hi

$ cat hello | sed 's@hello@Hey I just met you@g'
Hey I just met you Hey I just met you
Hey I just met you world
hi

$ cat hello | sed '1s/hello/Hey I just met you/g'
Hey I just met you hello
hello world
hi
$ cat hello | sed '2s/hello/Hey I just met you/g'
hello hello
Hey I just met you  world
hi

$ cat hello | sed '1,2s/hello/Hey I just met you/g'
Hey I just met you Hey I just met you
Hey I just met you world
hi
$ cat hello | sed '2,3s/hello/Hey I just met you/g'
hello hello
Hey I just met you world
hi
$ cat hello | sed '2,$s/hello/Hey I just met you/g'
hello hello
Hey I just met you world
hi

$ sed -i '' 's/hello/Bonjour/' sed-data
$ cat sed-data
Bonjour hello
Bonjour world
hi

Remarque. Sous Linux, juste assez -i. Mais sur macOS, le comportement de la commande est légèrement différent, -ivous devez donc l'ajouter juste après ''.

De vrais exemples

Filtrage CSV avec grep et awk

$ grep -w gauge metadata.csv | awk -F, '{ if ($4 == "query") { print $1, "per", $5 } }'
mysql.performance.com_delete per second
mysql.performance.com_delete_multi per second
mysql.performance.com_insert per second
mysql.performance.com_insert_select per second
mysql.performance.com_replace_select per second
mysql.performance.com_select per second
mysql.performance.com_update per second
mysql.performance.com_update_multi per second
mysql.performance.questions per second
mysql.performance.slow_queries per second
mysql.performance.queries per second

Affiche l'adresse IPv4 associée à l'interface réseau

$ ifconfig en0 | grep inet | grep -v inet6 | awk '{ print $2 }'
192.168.0.38

en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
    ether 19:64:92:de:20:ba
    inet6 fe80::8a3:a1cb:56ae:7c7c%en0 prefixlen 64 secured scopeid 0x7
    inet 192.168.0.38 netmask 0xffffff00 broadcast 192.168.0.255
    nd6 options=201<PERFORMNUD,DAD>
    media: autoselect
    status: active

$ ifconfig en0 | grep inet
    inet6 fe80::8a3:a1cb:56ae:7c7c%en0 prefixlen 64 secured scopeid 0x7
    inet 192.168.0.38 netmask 0xffffff00 broadcast 192.168.0.255

$ ifconfig en0 | grep inet | grep -v inet6
inet 192.168.0.38 netmask 0xffffff00 broadcast 192.168.0.255

$ ifconfig en0 | grep inet | grep -v inet6 | awk '{ print $2 }'
192.168.0.38

Remarque. On m'a proposé de remplacer par grep inet | grep -v inet6une équipe aussi fiable awk:

$ ifconfig en0 | awk ' $1 == "inet" { print $2 }'
192.168.0.38

Il est plus court et spécifiquement ciblé sur IPv4 avec la condition $1 == "inet".

Récupération d'une valeur à partir d'un fichier de configuration

$ grep 'editor =' ~/.gitconfig  | cut -d = -f2 | sed 's/ //g'
/usr/bin/vim

$ grep 'editor =' ~/.gitconfig
     editor = /usr/bin/vim
$ grep 'editor =' ~/.gitconfig  | cut -d'=' -f2
 /usr/bin/vim
$ grep 'editor =' ~/.gitconfig  | cut -d'=' -f2 | sed 's/ //'
/usr/bin/vim

Extraire les adresses IP d'un fichier journal

$ grep 'Too many connections from' db.log | \
  awk '{ print $12 }' | \
  sed 's@/@@' | \
  sort | \
  uniq -c | \
  sort -rn | \
  head -n 10 | \
  awk '{ print $2 }'
   10.11.112.108
   10.11.111.70
   10.11.97.57
   10.11.109.72
   10.11.116.156
   10.11.100.221
   10.11.96.242
   10.11.81.68
   10.11.99.112
   10.11.107.120

$ grep "Too many connections from" db.log | head -n 1
2020-01-01 08:02:37,617 [myid:1] - WARN  [NIOServerCxn.Factory:1.2.3.4/1.2.3.4:2181:NIOServerCnxnFactory@193] - Too many connections from /10.11.112.108 - max is 60

$ grep "Too many connections from" db.log | awk '{ print $12 }'
/10.11.112.108
...

$ grep "Too many connections from" db.log | awk '{ print $12 }' | sed 's@/@@'
10.11.112.108
...

Remarque. Comme nous l'avons vu précédemment, sedvous pouvez utiliser n'importe quel séparateur dans. Bien qu'il soit généralement utilisé comme séparateur /, nous remplaçons ici ce caractère particulier, ce qui nuira légèrement à la lisibilité de l'expression de substitution.

sed 's/\///'

$ grep 'Too many connections from' db.log | \
  awk '{ print $12 }' | \
  sed 's@/@@' | \
  sort | \
  uniq -c
   1379 10.11.100.221
   1213 10.11.103.168
   1138 10.11.105.177
    946 10.11.106.213
   1211 10.11.106.4
   1326 10.11.107.120
   ...

$ grep 'Too many connections from' db.log | \
  awk '{ print $12 }' | \
  sed 's@/@@' | \
  sort | \
  uniq -c | \
  sort -rn | \
  head -n 10 | \
  awk '{ print $2 }'
  10.11.112.108
  10.11.111.70
  10.11.97.57
  10.11.109.72
  10.11.116.156
  10.11.100.221
  10.11.96.242
  10.11.81.68
  10.11.99.112
  10.11.107.120

Renommer une fonction dans le fichier source

$ cat izk/utils.py
def bool_from_str(s):
    if s.isdigit():
        return int(s) == 1
    return s.lower() in ['yes', 'true', 'y']

$ sed -i 's/def bool_from_str/def is_affirmative/' izk/utils.py
$ cat izk/utils.py
def is_affirmative(s):
    if s.isdigit():
        return int(s) == 1
    return s.lower() in ['yes', 'true', 'y']

Remarque. Sur macOS, sed -iutilisez plutôt sed -i ''.

Approfondissement: cycles foretxargs

$ grep -r bool_from_str .
./tests/test_utils.py:from izk.utils import bool_from_str
./tests/test_utils.py:def test_bool_from_str(s, expected):
./tests/test_utils.py:    assert bool_from_str(s) == expected
./izk/utils.py:def bool_from_str(s):
./izk/prompt.py:from .utils import bool_from_str
./izk/prompt.py:                    default = bool_from_str(os.environ[envvar])

-l, --files-with-matches
        Only the names of files containing selected lines are written to standard out-
        put.  grep will only search a file until a match has been found, making
        searches potentially less expensive.  Pathnames are listed once per file
        searched.  If the standard input is searched, the string ``(standard input)''
        is written.

$ grep -r --files-with-matches bool_from_str .
./tests/test_utils.py
./izk/utils.py
./izk/prompt.py

$ grep -r --files-with-matches bool_from_str . | \
  xargs -n 1 sed -i 's/bool_from_str/is_affirmative/'

$ sed -i 's/bool_from_str/is_affirmative/' ./tests/test_utils.py
$ sed -i 's/bool_from_str/is_affirmative/' ./izk/utils.py
$ sed -i 's/bool_from_str/is_affirmative/' ./izk/prompt.py

grep -r --files-with-matches bool_from_str . | xargs sed -i 's/bool_from_str/is_affirmative/'

$ sed -i 's/bool_from_str/is_affirmative/' ./tests/test_utils.py ./izk/utils.py ./izk/prompt.py

Remarque. Le synopsis sedde la page de manuel montre que l'équipe peut prendre plusieurs arguments.

SYNOPSIS
     sed [-Ealn] command [file ...]
     sed [-Ealn] [-e command] [-f command_file] [-i extension] [file ...]

En effet, comme nous l'avons vu dans le chapitre précédent, cela file ...signifie que plusieurs arguments sont acceptés, qui sont des noms de fichiers.

$ grep -r is_affirmative .
./tests/test_utils.py:from izk.utils import is_affirmative
./tests/test_utils.py:def test_is_affirmative(s, expected):
./tests/test_utils.py:    assert is_affirmative(s) == expected
./izk/utils.py:def is_affirmative(s):
./izk/prompt.py:from .utils import is_affirmative
./izk/prompt.py:                    default = is_affirmative(os.environ[envvar])

for item in list; do
    command $item
done

$ for file in $(grep -r --files-with-matches bool_from_str .); do
  sed -i 's/bool_from_str/is_affirmative/' $file
done

$ sed -i 's/bool_from_str/is_affirmative/' ./tests/test_utils.py
$ sed -i 's/bool_from_str/is_affirmative/' ./izk/utils.py
$ sed -i 's/bool_from_str/is_affirmative/' ./izk/prompt.py

Sommaire

• grep chaînes qui correspondent au modèle de chaînes avec des adresses IP
  
• recherchez la colonne avec l'adresse IP, extrayez-la avec awk
  
• trier la liste des adresses IP à l'aide sort
  
• éliminer les doublons adjacents avec uniq
  
• compter le nombre de lignes (c'est-à-dire les adresses IP uniques) en utilisant wc -l

Et après

1. Comptez le nombre de fichiers et de répertoires situés dans votre répertoire personnel.
   
2. .
   
3. , .
   
4. . .