My hobby is to open McIlroy's “UNIX Philosophy” on one monitor while reading mana on another.

The first of McIlroy’s principles is often rephrased as “Do one thing, but do well.” This is an abbreviation for his words, “Create programs that do one thing well.” For new work, create new programs, rather than complicate the old ones by adding new "functions". ”

McIlroy gives an example:

It seems surprising to outsiders that UNIX compilers do not issue listings: printing is better done and more flexibly configured using a separate program.

If you open the help for ls, then it starts with

ls [-ABCFGHLOPRSTUW@abcdefghiklmnopqrstuwx1] [file ...]

That is, the one-letter flags for lsinclude all lowercase letters, except for {jvyz}14 uppercase letters, @and 1. This is 22 + 14 + 2 = 38 only single-character options.

In Ubuntu 17, the help for lswill not show a normal summary, but you will see that lsthere are 58 options (including --helpand --version).

Let's see if this is a unique situation or a normal state of affairs. Let's make a list of some common commands, sorted by frequency of use.

The table shows the number of command line options for various v7 Unix (1979), slackware 3.1 (1996), ubuntu 12 (2015), and ubuntu 17 (2017) commands. The more parameters, the darker the cells (on a logarithmic scale).

We see that over the years the number of options increases dramatically: as a rule, records darken from left to right (more options), and there are no cases when records become lighter (fewer options).

McIlroy has long condemned the increase in the number of options, size and overall functionality of teams ¹ :

, , Linux … []. , , . , , … Unix : « ? ?» , - , . , , , . … , .

Ironically, one of the reasons for the growing number of command-line options is McIlroy’s other saying: “Write programs for processing text streams, because this is a universal interface” (see lsas an example).

If structured data or objects were transmitted, formatting can be left to the final stage. But in the case of plain text, formatting and content are mixed; since formatting can only be done by parsing content, commands usually add formatting options for convenience. In addition, the format can be performed when a user applies his knowledge of the data structure, and "encodes" the knowledge of the arguments for cut, awk,sedetc. (the user also uses his knowledge of how these programs work with formatting, because it is different for different programs, so the user must know, for example, how it cut -f4 differs from awk '{ print $4 }² ) This is a lot more hassle than passing one or two arguments to the next command in a sequence, and this transfers the complexity of the tool to the user.

People sometimes say that they don’t want to support structured data, because then in a universal tool one would have to support several formats. But they already have to support several formats to make a universal tool. Some standard commands cannot read the output from other commands because they use different formats. For example, wc -wUnicode does not correctly handle it, etc. To say that “text” is a universal format is the same as saying that “binary” is a universal format.

It is said that there is no alternative for such complication of command line tools. But people who say that have never tried an alternative, something like PowerShell. I have many complaints about PowerShell, but the transfer of structured data and the ability to easily work with structured data without having to keep metadata in my head so that I can transfer it to the right command-line tools in the right places on the pipeline is not one of my complaints ³ .

When you are told that programs should be simple and compatible while processing text, these people pretend that text data does not have a structure to parse ⁴ . In some cases, we can think of everything as one row, separated by spaces, or as a table with row and column separators ( with behavior that, of course, is not consistent with other tools ). This adds a bit of trouble. There are still cases where serializing data into a plain text format adds significant complexity, since due to the data structure, simple serialization to text subsequently requires significant parsing efforts to re-assimilate data in a meaningful way.

Another reason why teams now have more options is because people have added convenient flags for functionality that could be implemented by a pipeline of several teams. This practice has been going on since Unix v7, where inlsan option appeared to change the sort order (although this could be done by passing the output to tac).

Over time, purely for convenience, added additional parameters. For example, a command mvthat was originally without parameters can now move the file and simultaneously create a backup copy (three options; two different ways to specify a backup, one of which takes an argument and the other takes no arguments, reads an implicit argument from an environment variable VERSION_CONTROL; another option allows you to override the default backup suffix). Now mvthere are still options to never overwrite files or overwrite only newer files.

mkdiranother program that previously had no options. Today, all its flags, with the exception of the security options for SELinux or SMACK, as well as help and version numbers, are added for convenience only: setting permissions for the new directory and creating parent directories if they do not exist.

At the tailbeginning there was only one option -number, indicating the starting point for the work. Then we added both formatting and options for the convenience of formatting. The flag -zreplaces the line separator with null. Here are other examples of options added for convenience: -fto print when new changes appear, -sto set the timeout interval between checking changes / code> -f, and also -retryto retry accessing the file if it is not available.

McIlroy criticizes developers for adding all of these options, but I personally feel better. Although I have never used some of them, I rarely used others, but this is the beauty of the command line parameters - unlike the graphical interface, adding these parameters does not clutter the interface. Yes, mana and help swell, but in the era of Google and Stackoverflow, in any case, many just google any question, rather than read mana.

Of course, adding options increases the load on the maintainers. But this is a fair payment for the benefits they bring. Given the ratio of the number of maintainers and users, it is logical to place an additional burden on the former, and not on the latter. This is similar to Gary Bernhardt’s remark that it’s wise to rehearse a performance 50 times. If the audience is 300 people, then the ratio of time spent watching the performance to time spent on rehearsals will still be 6: 1. For popular command line tools, this ratio is even more extreme.

Some might argue that all of these additional options put an extra burden on users. This is not entirely wrong, but this burden of complexity will always exist. The question is where exactly. If you imagine that a set of command line tools together with a shell form a language in which everyone can write a new method, and in case of popularity, the method is effectively added to the standard library, and the standards are determined by axioms like “Write programs for processing text streams, because it’s universal interface ”, then the language will turn into incoherent write-only chaos, if you take it in its entirety. At least, thanks to tools with a wide range of options and functionality, Unix users can replace a giant set of wildly inconsistent tools with just a large set of tools that,although they are inconsistent with each other from the outside, they have some internal consistency.

McIlroy implies a lack of thoughtfulness in the toolbox. Like, the founding fathers of Unix should sit in the same room and think carefully until they came up with a set of sequential tools of "extraordinary simplicity." But it will not scale, the philosophy of Unix itself inevitably led to the mess we are in. It is not that someone did not think long or hard. The point is a philosophy that does not scale beyond a relatively small team with a common cultural understanding that can fit in one room.

If someone wants to write a tool based on the “Unix philosophy”, then different people will have different opinions about what “simplicity” means or the principle of “doing one thing” ⁵ how the tool should work correctly - and inconsistency will blossom in lush color, as a result of which you will get enormous complexity, similar to wildly inconsistent languages like PHP. People make fun of PHP and JavaScript for various oddities and inconsistencies, but like a language and a standard library, any popular shell with a collection of popular * nix tools, taken together, is much worse and contains much more random complexity due to inconsistencies even within the same Linux distribution . It cannot be otherwise. If you compare the distributions of Linux, BSD, Solaris, AIX, etc., the amount of random complexity that users must keep in mind when switching systems, overshadows the inconsistency of PHP or JavaScript.The most widely ridiculed programming languages are real examples of great design compared to them.

For clarity, I am not saying that I myself or someone else could better cope with the development in the 70s, taking into account the knowledge available then, and create a system that would be both useful at that time and elegant today. Of course, it's easy to look back and find problems in retrospect. I simply disagree with the comments of some Unix connoisseurs, like McIlroy, who hint that we have forgotten or do not understand the value of simplicity. Or Ken Thompson, who says that C is as safe a language as any other, and if we don’t want errors to appear, we just need to write code without errors. Comments of this kind imply that little has changed over the years. Allegedly, in the 70s we built systems in the same way as today, and for five decades of collective experience, tens of millions of person-years did not teach us anything. And if we turn to the origins, to the creators of Unix, then everything will be fine. With all due respect, I do not agree.

Application: memory

Although McIlroy’s complaints about bloating binaries are a bit outside the scope of this article, I’ll note that in 2017 I bought a Chromebook with 16 GB of RAM for $ 300. The 1 megabyte binary could be a serious problem in 1979, when the standard Apple II was equipped with 4 kilobytes of memory. Apple II cost $ 1,298 in 1979, or $ 4,612 in 2020. Today you can buy an inexpensive Chromebook that costs less than 1/15 of this price, while it has four million times more memory. Complaints that memory usage has grown a thousand times seem a bit ridiculous when (portable!) The machine costs an order of magnitude cheaper and has four million times more memory.

I like the optimization, so I narrowed my homepage to two packages (there would be one if the CDN supported brotli high-level), but this is a purely aesthetic requirement, I do it for fun. The bottleneck of the command-line tools is not using memory, and the time to optimize the tool's memory with a size of one megabyte is like reducing a home page to one package. Perhaps a fun hobby, but nothing more.

Table compilation methodology

The frequency of use of commands is obtained from the public files of the history of commands on github, it does not necessarily correspond to your personal experience. Only "simple" commands were counted, excluding instances like curl, git, gcc (the latter has more than 1000 options) and wget. The concept of simplicity is relative. Built-in shell commands , such as cd, were also not taken into account.

Repeating flags was not considered a separate option. For example, u git blame -C, git blame -C -Cand git blame -C -C -Cdifferent behavior, but they will all be considered as one argument, although -C -Cthey -C are actually different arguments.

Sub-options in the table are counted as one option. For example, it lssupports the following:

--format=WORD across -x, commas -m, horizontal -x, long -l, single-column -1, verbose -l, vertical -C

Despite seven options format, this counts as one option.

Options that are explicitly indicated as useless are still considered options, for example ls -g, which is ignored is also considered.

Multiple versions of the same option are considered one option. For example, -Aand --almost-allfor ls.

If the certificate says that the option exists, but in reality it does not exist, then it is not taken into account. For example, the help for v7 mv says:

BAGS

If file1 and file2 are in different file systems, then mv should copy the file and delete the original. In this case, the name of the owner becomes the name of the copy process, and any connection with other files is lost.

mv must accept the -f flag as rm in order to suppress a message about the existence of a target file that is not writable.

But -fit is not considered a flag in the table, because the option does not actually exist.

The table ends in 2017 because the first draft of this article was then written. Only now did they get to read it.

On this topic

Mjd's discussion of Unix philosophy, with thoughts of the / usr / bin / time mess versus built-in time
The mjd joke about distributing command options in 1991
With HN:
p1mrx:

It is strange that ls has grown to 58 options, but still cannot display file names ending with \ 0.

As an exercise, try sorting the directory by size or date and transfer the result to xargs, while supporting all valid file names. In the end, I just gave up and put in the script ignore any files with \ n in the name.

whelming_wave:

, (whitespace-in-filenames-safe). printf (od -> sed) , , , , , , . , sort --zero-terminated — GNU , macOS, , .
```
      printf '%b' $(
        find . -maxdepth 1 -exec sh -c '
          printf '\''%s %s\0'\'' "$(stat -f '\''%m'\'' "$1")" "$1"
        ' sh {} \; | \
        sort --zero-terminated | \
        od -v -b | \
        sed 's/^[^ ]*//
      s/ *$//
      s/  */ \\/g
      s/\\000/\\012/g')
```
zsh, , 'command', printf, zsh, escape- . , .

POSIX, , sort.
Unix
?

_{, , -, //.}

1. This quote is slightly different from the common version because I watched the original video . As far as I can tell, all copies of this quote on the Internet (Bing, DuckDuckGo and Google indexes) are taken from the same transcription of one person. There is a certain ambiguity, because the sound is of poor quality, and I hear words that are slightly different from what that person heard. [to return]

2. Another example of how complexity is passed on to the user, because different teams handle formatting differently, is time formatting . The built-in shell time is time, of course, incompatible with /usr/bin/time. The user must be aware of this fact and know how to handle it. [to return]

3. For example, for any object you can use ConvertTo-Jsonor ConvertTo-CSV. Or “cmdlets” to change the display of object properties . You can write formatting configuration files that define your preferred formatting methods.

Another way to look at this is through the prism of Conway's law . If we have a set of command-line tools created by different people, often from different organizations, these tools will be wildly inconsistent if someone cannot determine the standard and force people to accept it. This actually works relatively well on Windows, not just PowerShell.

A common complaint against Microsoft is the massive turnover of the API, often for non-technical organizational reasons (for example, see Stephen Sinofsky’s actions as described in responses to a remote tweet ). It's true. However, from the point of view of a naive user, standard Windows software, as a rule, transmits non-textual data much better than * nix. The coverage of non-textual data in Windows dates back to at least COM in 1999 (and possibly OLE and DDE, released in 1990 and 1987, respectively).

For example, if you copy from the Foo, which supports the binary format Aand Bin the Bar, which supports formats Band Cthen copy from Bar to Baz, which supports CandD, everything will work fine, even if Foo and Baz do not have common supported formats.

When you cut or copy something, the application basically “tells” the clipboard in which formats it can provide data. When pasted into an application, the final application may request data in any of the available formats. If the data is already on the clipboard, Windows provides it. If this is not the case, Windows receives data from the source application, and then transfers it to the target application, and a copy is stored for some time. If you “cut” from Excel, he will say “you” that he has data available in many dozens of formats. Such a system is pretty good for compatibility, although it certainly cannot be called simple or minimalistic.

In addition to the good support of many formats, it is long enough that many programs begin to deal well with these features; in Windows, out of the box there is usually good clipboard support.

Suppose you copy and paste a small amount of text. In most cases, no surprises will occur on either Windows or Linux. But now suppose you copied some text, closed the program from which you copied, and then pasted it. Many users tend to think that when copying data is stored in the clipboard, and not in the program from which it is copied. On Windows, software is usually written according to this expectation (although technically users of the clipboard API should not do this). This is less common on Linux with X, where the correct mental model for most programs is that copying saves a pointer to the data that still belongs to the program from which it is copied. That is, the insert will not work if the program is closed.When I (informally) interviewed programmers, they were usually surprised at this, unless they really worked with the copy + paste function for their application. When I interviewed non-programmers, they usually found this behavior not only surprising, but also confusing.

The drawback of transferring the clipboard to the OS is that copying large amounts of data is expensive. Suppose you copy a really large amount of text, a lot of gigabytes or some kind of complex object, and then do not paste it. In fact, you do not want to copy this data from your program to the OS so that it is stored there and available. Windows does this wisely: applications can only provide data on demand , if deemed beneficial. In our case, when the user closes the program, it can determine whether to put data on the clipboard or delete it. In this case, many programs (for example, Excel) will offer to “save” data to the clipboard or delete them, which is quite reasonable.

Some of these features can be implemented on Linux. For instance,The ClipboardManager specification describes the save mechanism, and GNOME applications usually support it (albeit with some bugs ), but the situation on * nix really differs from the ubiquitous support for Windows applications, where a competent clipboard is usually implemented. [to return]

4. Another example is tools on top of modern compilers. Let's go back and look at the canonical Macilroy example, where the right Unix compilers are so specialized that listing is done by a separate tool. But today this has changed, although a separate listing tool has remained. Some popular Linux compilers have literally thousands of options - and they are extremely feature rich. For example, one of the many functions of the modern clang is static analysis. At the time of this writing, there are 79 routine tests of static analysis and 44 experimental tests.. If these were separate commands, they would still rely on the same basic compiler infrastructure and impose the same maintenance burden - it is actually unreasonable for these static analysis tools to work with plain text and redefine the entire compiler tool chain needed for getting the point where they can perform static analysis. They can be separate commands instead of being combined into clang, but they will still depend on the same mechanism and either impose the burden of maintenance and complexity on the compiler (which should support stable interfaces for tools that work on top of it), or they will be constant break.

To do everything in the text for simplicity sounds beautiful, but in fact the textual representation of the data is often not what you need if you want to do a really useful job.

The same clang is simply more functional than any compiler that existed in 1979, or even all the compilers that existed in 1979 combined, regardless of whether it was executed by a monolithic command or thousands of smaller instructions. It is easy to say that in 1979 everything was simpler and that we, modern programmers, went astray. But in reality it is difficult to offer a design that is much simpler and will be truly accepted by everyone. It is impossible that such a design could preserve all existing functionality and configurability and be as simple as something from 1979. [to return]

5. Since its inception, curl has moved from supporting three protocols to 40. Does this mean that it “does 40 things”, and does Unix philosophy require it to be divided into 40 separate commands? Depends on who to ask. If each protocol were its own team, created and supported by another person, we would have the same mess as with the teams. Inconsistent command line parameters, inconsistent output formats, despite the fact that all these are text streams, etc. Does this bring us closer to the simplicity that McIlroy advocates? Depends on who to ask. [to return]

Complication of console commands, 1979−2020

Application: memory

Table compilation methodology

On this topic

More articles: