> The core Unix philosophy is about enabling workflows that combine a set of independent tools in novel ways.
You mean like
ls | rm
deletes the files in a directory?
Uh wait, that doesn't actually work.
Because the Unix tools actually don't implement the Unix philosophy at all.
Because as it turns out, text streams are not a universal interface after all.
Nushell succeeds where the coreutils have failed for decades precisely because its commands are designed to work together. They pass structured data around, which means that pipelines can actually work without requiring weird hacks to make one underspecified text format conform to the other. That is the Unix philosophy: Tools that work together.
But don't worry, the old Unix tools can be used from Nushell – and they work together just as poorly in Nushell as in every other shell, not one bit worse.
This is a strawman. It's trivially easy to do what you're describing.
ls | xargs rm
And in fact, a better implementation is
rm *
The Unix philosophy doesn't say "Every program interfaces intuitively and correctly with every other program". It says that programs should be able to work together. There are arguments that coreutils such as ls aren't following the Unix philosophy, but they lie much more in that ls has dozens of arguments than they do in your contrived example not working.
The fact that the first example has a bug (it doesn't handle file names with spaces correctly) sort of proves the point of the commenter you responded to.
I personally don't use nushell (or another alternative, oil) because bash for scripting and zsh for interactive shell is good enough for me, and compatibility with other people is valuable. However, whenever you start to write slightly complex scripts in bash, you have to think carefully about spaces in file names and other similar problems. Of course, you can argue you should just write those scripts in Python, but shell is a lot quicker to write and would be more elegant if it had fewer footguns and less esoteric syntax.
It's a huge leap to assert that Unix tools don't implement the Unix philosophy because a specific example doesn't work in the exact way you expect it to. You could just as well implement a version of `rm` that parses stdin in a specific way by default, but that would likely make it work for that specific use case, and not in the generic and composable way the tools are meant to be used.
The whole point of Unix is for tools to be independent of each other _and_ of the environment they run in, while giving the user the freedom to compose them in any way they need. What Nushell has built instead is a closed ecosystem of tools that interoperate well with each other, but not with the outside environment. This means that a) the ecosystem is not extensible unless one contributes to Nushell, or directly implements Nushell features, and b) external tools will always be second-class citizens, that might even be incompatible with Nushell.
To give you an example, how would I use GNU ls within Nushell? Suddenly, I can't use any of the `where` fanciness, and my entire pipeline breaks. I would have to resort to another Nushell-specific helper to integrate the command, which needs to be written in a very generic way to support a wide range of use cases, or switch my entire pipeline to use external commands only, which defeats the purpose of using Nushell to begin with.
This is a contrived example, but if you compound this with the amount of CLI tools that exist and have yet to be written, it's a usability and maintenance headache.
So I'm glad that Nushell exists, as it challenges existing notions of what a shell can be, but let's not disparage existing shells or Unix to prove a point. The Unix ecosystem is so widespread and successful today _because_ of these early decisions, and will more than likely continue to exist because of them as well. That doesn't mean that we can't do better, but I'd argue that a monolithic shell with a strict contract between commands is not the way to build a sustainable and future-proof ecosystem.
> That doesn't mean that we can't do better, but I'd argue that a monolithic shell with a strict contract between commands is not the way to build a sustainable and future-proof ecosystem.
I don't think it is far fetched to imagine that most popular command line tools will have support for JSON output within let's say 5 years. With that, I think nushell's value proposition becomes a whole lot stronger. Granted, there will never be a time when all tools integrate well, but I can see a critical mass evolve.
Sure, but why JSON? :) What happens when a new format comes along that is better, more efficient, or whatever? Would all tools then need to be updated to support it? What if a change in the format is introduced? Would all tools need to be updated, and have to maintain backwards compatibility indefinitely? This would be an even bigger UX problem, while also adding much more maintenance work for tool developers.
It might seem like unstructured text as an exchange format is a usability nuisance, but it is what makes independent and disparate tools work well together. Choosing a strict contract is only sustainable if a single project maintains all tools, and even then it's a huge maintenance effort.
> Would all tools then need to be updated to support it?
The effort to translate JSON to another structured data format is orders of magnitude easier than converting the unstructured mess we have now into JSON.
So, it does not matter if we choose JSON, XML, TOML, etc. Once we get structured data, then things progress quickly.
The nice thing about JSON at this moment is that it is a very common and easy to use format.
JSON seems to have established itself for this to a large extent already. I can imagine a binary format like MsgPack to work well for this too but I think JSON is good enough and popular enough that it could actually be embraced widely.
> Choosing a strict contract is only sustainable if a single project maintains all tools
You're obviously not working in the web space, where literally hundreds of thousands of sites have somehow successfully standardized on interoperating via JSON input/output for their HTTP API endpoints despite using many, many different technology stacks, over the last 10 years or so.
It isn't perfect, but it's easy to use and common and the tooling is mature.
I absolutely think that treating data piped from or to another process as JSON by default, and treating data going back to the terminal as text (by transforming the JSON to tabular data etc.) is the way to go (also because it's backwards-compatible). To the point that I wanted to write some wrappers for the standard commands that automatically did all this via `jq`. I know commands can detect whether they're being piped to another command or to a terminal, since certain commands will automatically skip ANSI coloring when being piped elsewhere...
> I wanted to write some wrappers for the standard commands that automatically did all this via `jq`.
If you're not already aware of it, you may wish to check out `jc`[0] which describes itself as a "CLI tool and python library that converts the output of popular command-line tools, file-types, and common strings to JSON, YAML, or Dictionaries. This allows piping of output to tools like jq..."
The `jc` documentation[1] & parser[2] for `ls` also demonstrates that reliable & cross-platform parsing of even "basic" commands can be non-trivial.
This is interesting, but I like neither the TUI nor the fact that it's written in Python, lol.
My idea was to have namespaced wrappers for the commands which were designed to generate output as JSON and/or accept input as JSON. So for example "ls" would have a wrapper "qls" (queryable ls) or maybe "jsls" (json LS), etc. But in thinking about it, many questions remain- Would some of the structs or struct elements have types, like "file_path" or "file_name" for example?
I think standardized JSON output makes the value of nushell weaker, not stronger: more structure on stdin and stdout (e.g. switching them to unix sockets instead of fifos and passing FDs around) means the shell composing commands can be less sophisticated. The only way a monolith like nushell adds value is if you compare it to the current standard of unstructured bytes everywhere.
> What Nushell has built instead is a closed ecosystem of tools that interoperate well with each other, but not with the outside environment.
IMO "closed ecosystem" feels like a mischaracterization of the project--there's multiple features that are designed to support interoperability outside the nushell environment.
> how would I use GNU ls within Nushell?
To specifically answer this question there's at least 3 possible approaches (including basic parsing of output based on "Parse external ls command and combine columns for datetime" example[0])):
# Option 1 (use `^ls`)
$ nu --commands '^ls -lb | detect columns --no-headers --skip 1 --combine-columns 5..7 | select column4 column8 | rename size name | update size {|it| $it.size | into filesize} | last 5 | where size > 10KiB'
# Option 2 (use `run-external`)
$ nu --commands 'run-external --redirect-stdout "ls" "-lb" | detect columns --no-headers --skip 1 --combine-columns 5..7 | select column4 column8 | rename size name | update size {|it| $it.size | into filesize} | last 5 | where size > 10KiB'
# Option 3 (use `--stdin`)
$ ls -lb | nu --stdin --commands 'detect columns --no-headers --skip 1 --combine-columns 5..7 | select column4 column8 | rename size name | update size {|it| $it.size | into filesize} | last 5 | to nuon' | nu --stdin --commands 'from nuon | where size > 10KiB'
```
You could also use `jc` to parse the ls output to JSON & pipe that into nushell.
> I would have to resort to another Nushell-specific helper to integrate the command, which needs to be written in a very generic way to support a wide range of use cases
Well, one person would need to write that, once (or just use `jc`).
As a matter of interest, how would you do this example task ("display name & size for which of the last 5 files in a directory list are larger than 10KiB", I think) with bash/coreutils command pipeline? (Presumably some combo of `ls`, `cut`, `tail` and others?)
You're right, I'm being a bit harsh towards Nushell, partly because I don't see the problem these tools are trying to fix as particularly significant, and because the legacy Unix design decisions are what helped make it successful today.
My point is that an open ecosystem can't prosper if all tools are part of a single project, and they depend on each other to work. There's a good reason why even GNU tools, and even those part of coreutils, are not built with an assumption that they will be part of the same pipeline. All of them are independent, and "do one thing well", and how they integrate into a pipeline is entirely up to the user. All external tools that follow this same principle can generally be composed in the same way. The benefit of this is that the user doesn't depend on a single project, and each tool is easily replaceable by another.
Thanks for those Nushell examples. At first glance, they don't look readable or intuitive to me, but maybe it's because you're parsing the output of `ls`, which is generally a bad idea. I only mentioned `ls` as a contrived example, but imagine you have to integrate one or more external commands, and not just at the start of the pipeline, but somewhere in the middle. Doing those conversion steps from/to Nushell would become very cumbersome.
I wouldn't use `jc` or any structured exchange format for this task. `find` can do most of the legwork:
This doesn't show the results in a nice table as Nushell would probably do, but IMO it's much simpler, very clear what each tool does, and the pipeline is easily extensible at any point.
I think these are both orthogonal arguments. @p-e-w is arguing for tools that neatly and simply compose workflows, while you're arguing for independence of tools set.
Let's explore further.
> It's a huge leap to assert that Unix tools don't implement the Unix philosophy because a specific example doesn't work in the exact way you expect it to.
Fair statement, but it's never just one example. It's tonnes of examples that can be made. There are groups of tools whose sole function is just to help stitch other commands together (I.e. getting ls and rm into the same workflow). It works, but the point being made is that it could be better.
> You could just as well implement a version of `rm` that parses stdin in a specific way by default, but that would likely make it work for that specific use case, and not in the generic and composable way the tools are meant to be used.
I would argue this is _exactly_ what's happened - just that it isn't for only rm, but a whole common subset of tools to bootstrap a new way of working. Your exact argument works still for nushell, if someone wants a new tool for rm (or the legacy one) they could still plausibly do that - and your point above would still hold, it would completely defeat the purpose of nushell, as suddenly _where_ doesn't work. That same argument however applies for current shells... If I suddenly swap out rm, my scripts at some point are going to start failing. Unfortunately we're tightly bound to an environment and ecosystem based on legacy decisions. There are plenty of examples of how that's played out with technology in other ways (for good and bad) - the question should be, is this the right evolution?
> That doesn't mean that we can't do better, but I'd argue that a monolithic shell with a strict contract between commands is not the way to build a sustainable and future-proof ecosystem.
I would point out that no good reason has been given for no contract in existing shells? Or maybe we should observe that there is actually a contract, but it's just a lot less structured. To the point that the contract is just an agreed historical structure that popular individual programs are expected to use. Regardless, a contract isn't a bad thing, especially when it's done correctly to balance flexibility and structure.
I'm not convinced nushell is the way to go, but I would prefer the debate not to be muddled. Sure current shells have done fine. We expect future shells to do better. Will they imbue the Unix philosophy perfectly? Well like any good philosophy that will be up for debate.
> There are groups of tools whose sole function is just to help stitch other commands together
This is an example of the Unix philosophy working: replace smarts in the tools with smart compositions of tools; i.e. don’t make programs have to determine whether to read file names from arguments or stdin, use arguments by default and `xargs` to convert stdin into arguments.
Completely agree, but I think we get stuck in a loop at this point. Why don't we use the composition tools instead to take _structured data_ and make it print in a pretty fashion on screen, rather than pretty display data and have to compose tools to make it structured?
We've still custom created smarts in the tools to do something, it's just that those smarts are dealing with printing the data to console in a pretty fashion (which was the priority at the time) rather than the manipulation and handling of the core data they have been created for.
It's not necessarily a philosophy change, just a perspective/priority change. We less and less need to experiment and understand the systems, and more and more want to connect and streamline them.
In that sense, perhaps that facet of capabilities of Nushell would be more comparable to a Foreign-data Wrapper for SQL rather than a shell with pipes.
You mean like
deletes the files in a directory?Uh wait, that doesn't actually work.
Because the Unix tools actually don't implement the Unix philosophy at all.
Because as it turns out, text streams are not a universal interface after all.
Nushell succeeds where the coreutils have failed for decades precisely because its commands are designed to work together. They pass structured data around, which means that pipelines can actually work without requiring weird hacks to make one underspecified text format conform to the other. That is the Unix philosophy: Tools that work together.
But don't worry, the old Unix tools can be used from Nushell – and they work together just as poorly in Nushell as in every other shell, not one bit worse.