Rust can be difficult to learn and frustrating, but it's also the most exciting thing in software development in a long time

By Paul Dix / Use Cases, Developer
Oct 22, 2018

I recently decided to put serious effort into learning the Rust programming language. I saw it coming up frequently in interesting projects (e.g. ripgrep) and kept hearing good things about it. My hesitation to picking up Rust since its 1.0 release in 2015 came from two fronts. First, I’m completely invested in Go because of InfluxDB. Second, I heard that it was not the easiest thing to learn. While I don’t normally shy away from difficult tasks, I was hesitant because I believe that many developer tools that take off do so because they are easy to use or give developers significant productivity gains. More often than not, I want to invest my time in tools that I think have longevity that will get some critical mass in the market.

However, there were a few things about Rust that gave me the inkling that even if it might be difficult to learn, it also might be carving out a very important (and needed) niche in the programming language landscape. What follows are my reflections on what I think those strengths are, how I’m approaching learning the language, and why I’m very excited about Rust. But be warned, I haven’t yet run anything in production, I’ve only written around 2,500 lines of Rust code, and I have yet to do any multi-threaded or network programming, or benchmarking. This is all based on my early impressions.

So why Rust? What caused me to want to look into the language more deeply? I’d be lying if I didn’t say that its performance wasn’t one of the big factors. No garbage collection, but with primitives built into the language/compiler to make sure you don’t forget to free the mallocs or accidentally dereference invalid pointers? Sign me up. There were two other big features on the list that interested me. The ability to create libraries that you can link to in other languages (like Python, Ruby, Go, etc.) via FFI was something I’ve been thinking about for some future work at Influx. Also, low (or zero?) cost integration with C and C++ libraries was a big motivator. There are some big C++ projects that I’d like to integrate with, and Rust seemed like a nice way to do this.

As for Rust’s relevance to Influx, I have a dream of creating a Rust-based implementation of Flux (our new scripting & query language) that uses the C++ Apache Arrow libraries, that would be embeddable in other systems (like Spark, Kafka Streams or other places). Additionally, the new version of our cloud platform is a services-based design so Rust could find its way into our infrastructure through specific services that could benefit from its performance and guarantees in preventing data races (another big Rust advantage). So those are what motivated me to finally take the leap, but I soon found out that those wouldn’t be the only things that make Rust a compelling language to work with.

Learning Rust by Implementing a Lexer, Pratt Parser, and Tree Walking Interpreter

I decided that a small project was the best way to start picking up the language. I saw someone link to Thorsten Ball’s book on Writing an Interpreter in Go and thought that writing that in Rust would be a good place to start. After having read the book, I highly recommend it, but I’ll come back to that. I’ve written an interpreter before, I’m very familiar with Go, and the nice thing about this project is that it limits the scope of what I’d need to learn. It only uses standard library functionality and doesn’t require any multi-threading or networking. Thorsten’s book comes with all of the code and extensive tests so it’s easy to make sure things are correct. I didn’t have to think about the algorithmic how of what I was building, only how to do it in Rust specifically. But before I dove into implementation, I’d have to pick up the language basics.

I should probably give a little context for where I’m coming from as a developer, because I think people with different backgrounds might find it easier (or harder) to learn Rust. I’ve been a Go programmer pretty exclusively since mid-2013. Prior to that I worked on a single page Javascript (Backbone) application for the better part of a year and before that I had built a “time series database (API)” in Scala. Going back further I was writing Ruby and Rails applications with a bunch of Javascript thrown into the mix, which is what I focused on from about 2006 to 2010. Before that it was C# and before that Delphi (which isn’t one I see on many resumes). I’ve written C and C++ before, but not since 2009 and even then I’ve probably only written about 10k lines of code in those two. More to the point, I’ve spent almost my entire career working in garbage collected languages. I also haven’t been very low level and not a systems person (unless you count building a database).

I started out by reading The Rust Programming Language, which is part of the free online documentation. My learning process usually involves multiple passes over material that covers the subject I’m trying to learn. The first pass is just to get the higher level ideas and introduce the vocabulary. I refer to this as the point where I build an index in my head of concepts. So I read through the book fairly quickly without worrying about if I deeply understood each part since I’m intending on either re-reading it, or picking up another book to cover the same material in a slightly different way.

Reading through this particular book made me appreciate early one of the Rust community’s strengths: documentation is built into everything. Docs for the standard library are online, or you can bring them up locally with a single command (something that’s great when you’re learning on a plane). Documentation exists as comments in the code and this is standard for third-party libraries. Cargo, the package management system, is quite good at pulling this all together. If you have a library, you can bring up its docs and the docs for all of its dependencies with a single command. The other amazing thing about the documentation in comments is that the examples you put into your docs will actually get built during testing, so code in documentation never falls out of sync with the actual library definitions (example from my project). These little touches all combine to build what I think is a very solid foundation for future Rust library authors, and more importantly, users.

At this point I was ready to start actually writing code. Thorsten’s book comes in at only four chapters, but it’s quite a bit of material (just over 200 pages). The order you implement everything is this: first the lexer, then the parser, then the interpreter, and then go back to add features to the language in all three areas. Converting the lexer over to Rust was a fairly straightforward process and didn’t present too many challenges beyond initial struggles with the borrow checker and compiler. Since a big part of learning a new thing is rote memorization, the mechanical aspects of building a lexer and parser are actually a good repetitive task to start hammering the syntax and vocabulary into your brain. It’s also fun and satisfying to create a test, see it fail, then write a bit of code to make it pass. Thorsten’s writing style is great, and he makes the whole exercise quite fun.

When it came time to implement the parser, I hit a wall. Specifically, I had to figure out how to do recursion and a nested tree structure (the AST) without having the Rust borrow checker yell at me. I Googled around a bit and re-read some sections of the book, but I obviously needed a deeper understanding of the borrow checker. I also read other people’s accounts of “hitting a wall” in the process of learning Rust so I figured extra effort would lead me to scaling it.

In fact, there were multiple times during my learning process in which I was transported back in time to when I was initially trying to learn how to program. After learning Basic in elementary & middle school, I remember evenings in high school trying to get C programs to compile and just not figuring things out. It took me multiple attempts and approaches to eventually pick up programming in any meaningful way. Contrary to popular belief, I don’t think great programmers have some innate ability that makes learning to code easy. I think they just fight through the hurdles in search of those few moments when things “just work” and you feel a deep sense of satisfaction. My sense is that if you’re going to learn Rust, you need to prepare yourself for this level of effort and frustration, but if you do, I think the payoff is worth it.

I decided it was time to go back to some more structured learning so I picked up Programming Rust and read through the first ten chapters before coming back to code. This book is great and was exactly what I needed to start really getting some of Rust’s core concepts. This one covers some in-depth stuff about how memory is organized and frequently references C++ code in comparison, but knowledge of C++ isn’t a prerequisite for getting significant value out of it. I think reading both is a good approach and I’d probably repeat it in the same way: read one, write some code, then read the other.

After reading the first 10 chapters of Programming Rust, I was able to push through the rest of the implementation. I still have open questions, and I’m not sure if the structure I used makes the most sense. I tried to stick pretty close to Thorsten’s Go implementation, but I did bring in some Rust specific things. But overall I’m not sure if an experienced Rust programmer would approve of the style and structure. I’ll come back to what I plan on doing to address this gap in my knowledge.

I used enums pretty extensively, and I used Rust’s Result pattern for returning errors. Rust’s idioms on error handling are fantastic. Errors have to be handled (or explicitly ignored with extra code and keystrokes) and the ? operator is very nice in eliminating a bunch of boilerplate error handling code (which should be particularly compelling for anyone coming from Go). No more checking the error and returning if it’s there. Rust handles this for you. Take this function for parsing a hash literal from Thorsten’s Go implementation and my Rust implementation:

Go Implementation

func (p *Parser) parseHashLiteral() ast.Expression {
	hash := &ast.HashLiteral{Token: p.curToken}
	hash.Pairs = make(map[ast.Expression]ast.Expression)

	for !p.peekTokenIs(token.RBRACE) {
		p.nextToken()
		key := p.parseExpression(LOWEST)

		if !p.expectPeek(token.COLON) {
			return nil
		}

		p.nextToken()
		value := p.parseExpression(LOWEST)

		hash.Pairs[key] = value

		if !p.peekTokenIs(token.RBRACE) && !p.expectPeek(token.COMMA) {
			return nil
		}
	}

	if !p.expectPeek(token.RBRACE) {
		return nil
	}

	return hash
}

Rust Implementation

.   fn parse_hash_literal(parser: &mut Parser) -> ParseResult {
        let mut pairs: HashMap<Expression,Expression> = HashMap::new();

        while !parser.peek_token_is(&Token::Rbrace) {
            parser.next_token();
            let key = parser.parse_expression(Precedence::Lowest)?;

            parser.expect_peek(Token::Colon)?;
            parser.next_token();
            let value = parser.parse_expression(Precedence::Lowest)?;

            pairs.insert(key, value);

            if !parser.peek_token_is(&Token::Rbrace) {
                parser.expect_peek(Token::Comma)?;
            }
        }

        parser.expect_peek(Token::Rbrace)?;

        Ok(Expression::Hash(Box::new(HashLiteral{pairs})))
    }

As you can see, I kept a pretty faithful port with the function names and basic code structure being largely the same. The experienced Rust developers will probably notice my inconsistent use of moves or borrows when passing a token to a method (I should refactor). There are three spots in the Go implementation that check for some error condition and return a nil if found. Thorsten puts errors into the struct that’s doing the parsing, where I chose to propagate errors through the return values of functions so that I could use Rust’s ? operator. I think Rust has the most elegant patterns for error handling of any language I’ve worked with. I never liked exceptions and Go’s pattern is more about style, which can be ignored or abused. Rust enforces it in the compiler. I like this because I’m a flawed human being that works with other human beings, so if the compiler can force us onto the right path, I’m all for it.

Once I passed the hurdle of implementing the first pieces of the Parser, everything else was pretty straightforward. As I mentioned, it’s a pretty mechanical process, so I just used that time to work on memorizing syntax for everything and taking pleasure in the step by step process of test, fail, pass. I suppose I did take a little time to learn how to implement local scopes and closures in the language. I had to pick up Rc (a reference counted pointer) and RefCell (for dynamically mutating state in the Environment associated with a function).

Although this design led to memory leaks in my implementation, that’s because it creates a cycle of reference counted structs due to the Monkey language’s closures, so they don’t get freed up. I’m wondering if there’s a way to design and structure the code around this or if I need to implement a basic garbage collector. I’ve opened up an issue to discuss a GC for Monkey Rust if anyone is up for giving me a few pointers in the right direction.

Rust's Strengths and Sweet Spot

I’d estimate that I’m not quite halfway through my process of learning Rust, but I still have some thoughts about where I think Rust fits into the programming landscape. The easiest way to think about it is to talk about what language Rust might replace. I think that almost any project you’d consider doing in C or C++ is a candidate for doing in Rust. Lower level systems projects, things that require excellent performance, or projects that require more control over what happens with your memory. Load balancers, proxies, operating systems, databases, network queues, distributed systems, machine learning libraries, the underlying implementation for higher level languages, and probably countless other things that are not coming immediately to mind. I think all of these represent perfect candidates for Rust implementations.

Over the last five years, Go has been picking up a significant number of these kinds of projects. One of Go’s primary advantages is in how simple the language is and how quickly it can be learned. Contrast this to Rust, which has significantly more syntax, a model for working with memory management that few if any programmers are familiar with, and a compiler that can be more strict than even the worst disciplinarian. However, there are compelling advantages that Rust can boast, which I think warrant the initial learning curve.

I mentioned this earlier, but Rust’s model makes creating a data race impossible when developing safe Rust code. The concurrency model is checked by the compiler. We’ve had a number of production bugs in InfluxDB due to data races that were only caught under heavy load. While Go may have channels, it offers no compile time guarantees that you’re not creating data races.

Forced error checking by the compiler is another big win. Yes, your development process can force code reviews, which should catch any code not correctly handling errors, but that’s a fallible process. With Rust, the compiler forces you to “do the right thing”, which is great because then you won’t have to worry about it slipping past a code review.

No referencing nil pointers. Oh how many times over the last five years have I done this, seen this or been bitten by it? That all goes away with Rust. Again, the compiler will force you to do the right thing. Speaking of the compiler, the messages it gives are the best I’ve ever seen. Helpful and frequently tell you exactly what to do to fix your error.

In Rust there are entire classes of bugs that are simply impossible to create because of the compile time guarantees. Because it’s software, there will be bugs. Oh yes, there will be bugs. However, we can create all new bugs and not trip over these other ones that we’ve been creating for the last four-plus decades. This is the payoff, and despite the strictness of the compiler, Rust’s bet is that if you learn the way you can be as productive (or more) than with another language. In Programming Rust, Jim Blandy and Jason Orendorff refer to this as “Rust’s radical wager.”

Next Steps & Conclusion

I have more miles to travel on my Rust journey before I can validate Blandy and Orendorff’s proposition, but I intend to give it real effort. Here are the next steps on my journey to getting to some level of Rust expertise.

I intend to finish Programming Rust and read through The Rustinomicon, the book on advanced and unsafe Rust. After that I’ll pick up a small prototype project to create a web service that uses a C++ storage library. For that I’ll finally dig into network programming and multi-threading. I’ll also be picking up a little unsafe code (or using an existing Crate). Speaking of, how are Rustaceans writing gRCP services?

Another place I need to focus some time on is what idiomatic Rust looks like. For this, I’d like to spend time reading through Crates (Rust libraries) that other Rust developers think show good style and exercise all the various areas of the language. Any recommendations here would be greatly appreciated, just @pauldix on Twitter. Also, if there are any Rustaceans that have an hour to get on a Zoom call with me and pick apart my code for Monkey Rust, I’d be in your debt and I’d be happy to pay it forward to some other newcomers once I’m more up to speed. Or I’ve also left some questions on a pull request in the repo.

Finally, I picked up Thorsten’s sequel to Writing an Interpreter in Go, titled Writing a Compiler in Go, which I’m looking forward to reading through while adding to my implementation of Monkey Rust. If you have any interest in the subject, I highly recommend them.

After all of this, I’ll be in a good place to make real efforts on some of the projects I envision for Rust at Influx. I think an embeddable implementation of Flux would be a fantastic thing to create in Rust.

I haven’t yet figured out how productive I can be in Rust, but so far I’m enjoying learning the language and am very excited about what I might be able to build with it. I think there’s a very real possibility that mountains of C and C++ code gets rewritten over the next fifty years leading to more secure systems software built around Rust’s guarantees. Would I write a new web service in it? Maybe, although I’d probably still reach for Go, but depending on the requirements, Rust might be a first choice.

And if Rust’s radical wager turns out to be a winning bet, it’ll become my de facto first choice for servers and systems.