Diving into something nu

Calendar icon   2023-12-10   Scroll icon  3109
Tag icon  foss , opinion , review , software

Nushell is a cool new project that attempts to implement cool ideas. I was alerted to its existence by a colleague of mine, and I tried to daily drive it for a while. I have a lot of opinions about it so let's talk about it!   

First some context. Nushell is a new shell in the vein of bash and zsh written in Rust. To show my hand up front: I agree with the philosophy and design behind nushell, but the experience was just too painful for me to recommend it. I am going to say some negative things about the project, but I hope this will come across as constructive because that is how I mean it. It is a wonderful and ambitious project and I wish it the very best, and at the very least I'll be monitoring this. Once the project matures and the developers have worked out some of the kinks I will mention in this article, I'll be happy to give it another try.

Having got that out of the way, let's start with the stakes. Who cares if nu may or may not be a better shell? Well, approximately 80% of how I interact with my computer is through a terminal, which means going through a shell, so a shell colours how I interact with everything. So, the stakes are quite high as far as choosing software goes. If it's better, I stand to gain a lot of productivity, but if it isn't, well…. (this is what we in the biz call "foreshadowing")

The obligatory history lesson

But, before we discuss nu, we first have to cover POSIX compliance. I know it's a tedious detour, but stick with me because it's going to be important for the discussions. Besides, because it's so boring, few people (including me until I started writing this post) know what it means to be POSIX compliant. Because POSIX is much larger and more complicated than what we need here, I will not dive into POSIX itself, but I'm going to use an analogy. As always, purists can send their complaints to screaming@intothevoid.parody and my team and I will attempt to address your concern as soon as possible.

So here is my analogy: POSIX is similar to the USB standard. It's a specification that tells applications what they minimally need to be able to do to work with other USB applications. USB application that one might work with might very well be able to do more than what the specification prescribes, but you can't guarantee it. For POSIX, and specifically as it relates to shells, this includes, but is not limited to:

So it defines both what a shell must be able to do at a minimum, but also tells applications developed for POSIX systems what they can expect and how. This is to ensure compatibility between systems. POSIX is perhaps less relevant now than it was at its inception, but that was the basic idea. And just as you can plug any USB device into any USB port and it isn't guaranteed to work, but has a much better chance to, things that are developed for POSIX systems have a much higher chance of working on other systems. This is the reason things developed for, e.g. bash, might work on zsh if you're lucky.

For this reason I'm just going to talk about bash, zsh and other POSIX compliant shells interchangeably for the rest of this blog post. We greatly appreciate your understanding and flexibility on this issue.

I'm not like other shells

So why are we talking about this? To answer that I'm going to quote from the Nushell website:

Non-goals

I think the philosophy behind Nu is great, and the design is very well done. I also think that the decision to not be POSTIX compliant was one they took very deliberately, and I commend them for that bravery. However, I must also say that Nu pays a very hefty price for it.

As a prime example of this, it means that it cannot (or should not) be your login shell, as mentioned in their documentation (source):

Warning:

Nu is still in development and is not intended to be POSIX compliant. Be aware that some programs on your system might assume that your login shell is POSIX compatible. Breaking that assumption can lead to unexpected issues.

This means that Bash has to remain your login shell and always call nushell as a form of bootstrapping. This may seem harmless but had quite a nasty implication for me. Because bash and not nu is your login shell, it means that subprocesses get spawned into a Bash shell. The process may thus latch onto any settings they can find there first, before they get to nu. So now you have to do double bookkeeping to keep bash and nu in sync. I hope I don't have to explain how much of a drag that is.

Many applications that need to change your environment (which is a lot of them) will try to use your login to suggest the changes to make when you install them. This means that you then have to take all of those applications, which they developed for a POSIX environment, and figure out how to apply that to your new nushell environment. It's not a showstopper most times but was also just too much effort in some cases and caused me to abandon some tools during the time I was using it. It was just another paper cut. I recognise that this is one that will hopefully become less relevant as the project grows and gains wider support, but it's still important to mention. Because Nu is incompatible in some ways, I don't see any remedy to this except for all the relevant projects to add support for it themselves.

Luckily, a lot of rust projects that I already use as my daily driver already have support for nushell. This includes things like zoxide, starship, etc. etc. So those switch overs were relatively easy. This didn't apply to all of them, though. For example, navi, which is a nice cheat sheet utility, does not yet (at the time of writing this) have support for this. I have looked into adding this functionality myself, though I'll discuss my contributions to nushell later.

When I looked at it, there was a project underway to support the core util tools. However, that project had only just started during my time with it. Nu offers ways to deal with non-nu tools like the core utils (things like cat sed cp etc. etc.), but crossing the boundary is very awkward. And because of the unix philosophy (programmes should do one thing, one thing only, do it well, and be composable), you're gonna be crossing that boundary often.

For example, I went down several days' worth of rabbit holes to integrate the wc tool (a programme to determine word counts in a file) into nushell so I could use it. Now I enjoy contributing to projects, especially as a way of learning. I did not have to do it, so I don't hold that against it, however the first interaction was quite painful. In bash on the other hand, I'm likely to just slice out the only bit of output I need and just work further on that. However, because nushell relies on structured data, I found it more necessary to parse outputs in their entirety and using regex a lot more than I otherwise would be likely to do. This slowed me down a lot, and was not a great experience, if I'm honest.

Another problem is that random other scripts you might find on the internet to do quite important things for you, such as the somewhat famous sessionizer script written by The Primeagen no longer work. Sometimes this is not an enormous deal, but I consider it unfeasible to port something like the rustup install script to nushell.

This means that once you have to use one of these, you once again have to escape to bash. This is not only awkward but with installations, it is also a bit of a crap shoot whether nushell will find your installation once you go back to it. I've had more than a few problems with this while I was trying Nu. As with many a daring and uncharted greenfield project, you had better be ready to DIY a lot for what you need. I am fine with that, but I also cannot deny that it is a lot of work. So take this as a "Caveat emptor". 

Sorry, I no speak English

(just FYI, that title is supposed to be read in a very heavy Dutch accent. Thank you for your cooperation.)

One of Nu's major philosophies is that everything is data (think data frames and structs), instead of text, as UNIX conceptualises it. Just like Rust, it also puts a lot of emphasis on error handling. An attitude that I can only applaud.

I like the paradigm nushell goes for a lot better than that of bash. However, nushell suffers from the helix problem. It is better, but because it departs from a very dominant and ubiquitous paradigm, it will also be fighting an uphill battle. Programming in nushell is hard, especially if you have any amount of pre-existing bash muscle memory.

For example, one of the most ubiquitous workflows in shell scripting, executing a script that doesn't exist yet because you're going to download it as part of the setup, is impossible. To quote the documentation:

Thinking in Nushell: Nushell is designed to use a single compile step for all the source you send it, and this is separate from evaluation. This will allow for strong IDE support, accurate error messages, an easier language for third-party tools to work with…

As with many of the things Nu does, I love this in a vacuum, but it causes paper cuts in day to day use. For example, this also means it's impossible to do something like eval "$(zoxide init zsh)" to set up your environment. You have to put the output of that command into a file and then source that, so now you have an extra dotfile to manage. This on its own maybe isn't that big of a deal, but what is a big deal is that a lot of other applications assume you can, like the aforementioned rustup installation script.

Over the years I have amassed a massive number of dotfiles, aliases, functions and other semi-ad hoc utilities I use all day, every day. Having to port over these tools to nushell was quite the task. Nu is quite a different beast to any shell language I've used before. This is one reason that I've looked at nu shell multiple times but never took the plunge into driving it daily. It was a good way to get familiar with the Nu language, but it was quite difficult to wrap my head around.

Programming in shell scripting languages like Bash is a craft of its own, as we would say in Dutch. It is heavy on string manipulation to produce the correct commands stuck together with duct tape and wishful thinking. This can lead to systems and tools that can be very brittle. There is a reason some people consider bash to be a write only language. Its syntax is dense and, because of some very weird quirks, it has a lot of gotchas. However, because it is the Lingua franca of Linux systems, I have learnt to wield it efficiently. Not only do I lack the efficiency with the Nu language, but sometimes it hurts me to have the bash way of thinking. Another paper cut. I don't think there is any way to remedy this. It is simply a consequence of the current landscape we exist in and the design decisions the nu shell project made. It's brave and innovative, but it still means you have to deal with the consequences.

As a data scientist I was also extremely excited to learn that Nu has built-in support for polars data frames. However, I'm sad to say that the syntax is much too awkward to use. I had some hope that I could leave python at the door for small workloads and just use my shell. Not having to manage python installations and environments is an amazing prospect, but as it stands, that will not happen. The difference between what is an expression, what is a lazy or an eager frame and the necessity to prefix everything with dfr, among others, makes this a no go. For context, I have a simple script that does some mild transformation on the csv of transactions my bank exports so I can import it in my finance tracking software. Trying to port this simple script over to nu, even though they both use polars, was frustrating and confusing to where I just gave up. I love this idea and I hope the team continues to develop it, but as it stands, for a data scientist, it is not usable at all.

Read the friendly manual

During the porting of my current environment, I had to visit the documentation of nushell often. This is not a problem on its own. However, nushell's documentation was very immature when I used it. They generate the code in the same structure as the underlying code is structured, not according to how you might use it. This is not a cardinal sin or anything, but it is quite unintuitive if you're new to the project.

Besides that, the documentation also lacks some very familiar QoL features, such as breadcrumb navigation or the ability to go up a level in the hierarchy. This made the documentation confusing to navigate through any other means than through the search bar. Even when you found something, it wasn't clear in which section you found it, so you couldn't make a mental note of where it was for next time. This is easy enough to fix by simply adding these elements, but they make quite a big difference.

The documentation (as opposed to the cookbook) was also very, very example focused with very little context around it. This is usually not too bad, but here, in the new paradigm, I found it very difficult to find and understand what I needed. A bit more context would have been nice, you know?

There were also several parts of the language that were just not in the documentation at all. I only found out about them by talking to a maintainer on discord. Again, I don't mind this as it's something you get used to when using open source software for a while, but it's not very scalable if the project plans to keep growing. This is absolutely something that will mature along with the project, but if nu wants to improve on this, I would advise them to create a bit more flow in the documentation rather than letting it be just a collection of examples with no context.

Aside from the API documentation, there is also the cookbook, which is great for the bits that are there. However I found it is also missing a lot of material. Again, the documentation is quite terse there with little context or flow to it. So while I commend the work that has been done, I am afraid I cannot in good faith give nu's documentation a passing grade for the experience I had with it. 

Nushell also features a robust testing framework built in. This is one thing that I have found lacking in bash a few times, so I am thrilled to see this. However, because of the tricky nature of nu's import system and the ad hoc nature of using a shell language, I found this very difficult to use to its full potential. When you do shell programming, I rarely have a neat project structure for functions and scripts that I'm working on, which makes the testing framework somewhat cumbersome to deal with, especially since you also have to deal with setup and teardown.

This one I'm not entirely sure how to remedy. It's tricky, especially given the ad hoc nature of shell programming (at least for me), but I think it helps for tools you intend to have around for longer. Some more documentation around the import system might be nice too, as I remember being confused about that for some time (although I don't remember what I was actually confused about, sorry). The foundation of the testing system, however, is solid and I hope to see it develop further.

Conclusion

So, do I think nushell is better than bash or zsh? Yes. Will I be using it? Unfortunately, no. For me, nu is to bash as DVORAK is to QWERTY. I believe it is better, but because of the ubiquity of the latter I cannot in good conscience switch over to it. The muscle memory needed for either is incompatible with the other, and the former isn't widespread enough for me to work in that system most of the time. Especially given the DevOps like nature of my current job, this is just a no go for me.

I am very sad so say that I have switched back to zsh since trying nushell. The team behind nushell deserves praise for all the amazing work and innovation they have done and continue to do. I will be cheering them on from the sideline, and I will happily try again sometime down the line, but for now, it is just too immature for me to justify using it as a daily driver.